3D human pose optimization estimation method based on generative data augmentation

Abstract

A 3D human posture optimization estimation method based on generative data enhancement decouples the original data from different types of scenes and postures, migrates the target posture to the source scene through root node alignment and coordinate transformation, generates a cross-domain fused 3D posture sequence, generates a synthetic video containing diversified backgrounds, perspectives and actions by using a controllable video generation model, and then trains a 3D human posture estimation model by using the synthetic video and the original data. In the online stage, real-time posture estimation is performed by using the trained 3D human posture estimation model. The controllable video generation model synthesizes diversified video data fused across domains, and the explicit alignment of scenes and postures improves the adaptability of the model to unknown domains.

Description

Technical Field

[0001] This invention relates to a technique in the field of image processing, specifically a 3D human pose optimization estimation method based on generative data augmentation. Background Technology

[0002] 3D human pose estimation is a core technology in mixed reality, autonomous driving, and other fields, but its performance is limited by domain differences between training and testing data (such as scene, lighting, and camera viewpoint differences). Existing methods improve model robustness by directly augmenting pose data (e.g., rotation and translation keypoints), but these methods ignore the generation of video modalities and cannot effectively simulate the diversity of real-world scenes. Furthermore, existing datasets (such as Human3.6M and PMR) suffer from inconsistent distributions across domains due to differences in acquisition environments, further exacerbating the difficulty of model generalization. Traditional data augmentation methods generate samples with limited diversity and cannot combine scene and pose information from different domains (e.g., indoor / outdoor, real / virtual), leading to a significant performance degradation in cross-domain tests. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a 3D human pose optimization estimation method based on generative data augmentation. This method synthesizes diverse video data that is fused across domains through a controllable video generation model, and combines explicit alignment of scene and pose to improve the model's adaptability to unknown domains.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a 3D human pose optimization estimation method based on generative data augmentation. The method decouples the scene and pose from the raw data from different types of scenes. By aligning the root nodes and transforming the coordinates, the target pose is transferred to the source scene. After generating a cross-domain fused 3D pose sequence, a controllable video generation model is used to generate a synthetic video containing diverse backgrounds, perspectives and actions. The synthetic video and the raw data are then used to jointly train a 3D human pose estimation model. In the online stage, the trained 3D human pose estimation model is used for real-time pose estimation.

[0006] The raw data preferably uses publicly available 3D human pose datasets (such as Human3.6M, PMR), containing videos of human movements in specific scenes, i.e., RGB image sequences and their ground truth 3D keypoint positions in world coordinates. The RGB image sequences are... The 3D keypoint locations in world coordinates are , These are the number of samples, the length of each sample sequence, the number of image channels, the image height, the image width, and the number of human body key points, respectively.

[0007] The controllable video generation model, guided by an RGB reference image and a 2D keypoint representation of human pose, generates corresponding videos based on a stable diffusion algorithm, specifically as follows: ,in: Represents the first image in the image sequence. Location Sample No. Position frame, Represents the truth value of a 3D keypoint. The location of 2D key points after the location sample is projected onto the camera plane. Represents the generated first Video data of location samples.

[0008] The 3D human pose estimation model predicts the position of the corresponding 3D pose key points in camera space based on 2D human pose key points. The model adopts, but is not limited to, the techniques described by Jihua Peng in "KTPFormer: Kinematics and Trajectory Prior Knowledge-Enhanced Transformer for 3D Human Pose Estimation".

[0009] The training mentioned refers to: ,in: These are the predicted values ​​for 3D pose key points. To estimate the pose from a 2D model in video The estimated 2D keypoint locations obtained from the data are as follows: For the ground truth values ​​of 3D pose keypoints. The loss function of a 3D pose estimation model is typically measured by the MPJPE index between the predicted and true values ​​of 3D pose keypoints.

[0010] This invention relates to a system for implementing the above method, comprising: a cross-domain data fusion unit, a controllable video generation and 3D pose estimation training unit, wherein: the cross-domain data fusion unit aligns the root nodes of scene samples and pose samples in the original data and eliminates the differences between coordinate systems in the domains, and then transfers the target pose to the source scene through linear transformation to generate a cross-domain aligned 3D pose sequence; the controllable video generation and 3D pose estimation training unit projects the cross-domain aligned 3D pose sequence onto a 2D plane to generate a pose guidance sequence, and then combines it with the RGB scene input to generate continuous frame video data, and then jointly trains a 3D pose estimation model with the synthetic data and the original data for real-time prediction of 3D human pose.

[0011] Technical effect

[0012] This invention employs a cross-domain data fusion strategy to decouple and reassemble scene and pose information from different datasets. The reassembled information is aligned to the input of a controllable video generation model to obtain synthesized video modal data, which is then used in conjunction with the original data to train a downstream 3D pose estimation model. Compared to existing technologies, this invention enriches the diversity of human poses in the dataset at the video modal level by generating cross-domain fused video data, significantly improving the performance of the 3D human pose estimation model in unknown scenarios under practical conditions where only 2D keypoints extracted from video modalities are used as input data. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention;

[0014] Figure 2 This is a schematic diagram of a keypoint regressor.

[0015] Figure 3 This is a diagram illustrating the effect of cross-domain fusion data generation in an example.

[0016] Figure 4 This is a schematic diagram illustrating the performance improvement of 3D pose estimation in the example. Detailed Implementation

[0017] like Figure 1 As shown in the figure, this embodiment relates to a 3D human pose optimization estimation method based on generative data augmentation, including:

[0018] Step 1) Load the raw data and extract the scene image sequence. Camera parameters and 3D pose key point sequence Wherein: 3D pose keypoints are the coordinate positions in world space. These are the number of samples, the length of each sample sequence, the number of image channels, the image height, the image width, and the number of human body key points, respectively.

[0019] Step 2) Select two different samples from a cross-dataset or two different samples from a cross-scene dataset, respectively as the scene sample and the pose sample. Then, select the pose sample... World space 3D pose alignment to scene samples The world space 3D pose is used to obtain a fused 3D pose sequence. Specifically: Its meaning is: Pre-emptively eliminate its own root node offset and apply Root node offset Then, through linear transformation The pose data is further diversified, where: subscript 'a' represents a scene sample, subscript 'b' represents a pose sample, and superscripts 'm', 'n', and 'l' represent the position of the sample within its own dataset. Subscript 'o' represents the root node. Subscripts 'ab' and the ' symbol indicate that the data is generated data, with scene information derived from 'a' and pose information derived from 'b'.

[0020] The linear transformations mentioned include, but are not limited to, adjusting the aligned posture. Axis rotation angle.

[0021] Step 3) The fused 3D pose sequence obtained in Step 2 Through camera transformation Obtain the coordinates of the fused pose 3D key points in camera space. Then through projection transformation Obtain the true values ​​of 2D pose keypoints on the camera plane. Finally, the keypoint regressor generates the simulated fused pose 2D keypoint coordinates. This ensures that the number of key points is consistent with both the original and fused data.

[0022] The coordinates of the fused pose 3D key points in the camera space are further used as the ground truth of the 3D pose estimation model.

[0023] The camera transformation The external parameters, and the scene sample with subscript a in step 2. The corresponding camera extrinsic parameters remain consistent.

[0024] like Figure 2 As shown, the keypoint regressor will have a number of Key point format conversion to quantity To adapt to the input requirements of the video generation module, the keypoint regressor includes an encoder module, a regressor body, and a scorer module, all composed of fully connected neural networks. The encoder module encodes the torso keypoints as input and outputs them to the regressor body and scorer module. The regressor body iterates the encoding results multiple times: at iteration step it=0, it initializes the theta layer by copying the input torso keypoints and simultaneously initializing facial and hand keypoints from head and wrist keypoints, which are then output as the initial values ​​for the next iteration. After iteration, the theta layer outputs the predicted facial and hand keypoints. The scorer module decodes the encoder's encoding results and the predicted facial and hand keypoints output by the regressor body to obtain scores for each predicted keypoint.

[0025] The score of the key point, which is between 0 and 1, is used to measure the effectiveness of the key points it predicts; preferably, key points with scores below a specified threshold are discarded when there are too many key points with scores below a specified threshold, which is set to 0.3.

[0026] The coordinates of the additional 2D key points of the simulated fusion posture are not included in the subsequent evaluation because the additional key points are obtained from prior information stored inside the regressor, rather than ground truth data.

[0027] Step 4) From the scene sample described in Step 2 Extract RGB image frames Together with the simulated fused pose 2D key point coordinates obtained in step 3 Input the video generation module to obtain RGB video data corresponding to the fused pose. Using a 2D pose estimation model, the coordinates of 2D key points in the camera plane are estimated based on the generated RGB video data. .

[0028] The 2D key point coordinates Compared to directly using 2D keypoint truth values In contrast, the former takes into account the impact of factors such as video and scene on the accuracy of pose estimation.

[0029] Step 5) Mix the raw data obtained in Step 1 with the synthetic data obtained in Step 4, and input the mixture into the 3D pose estimation model for joint training. The optimization objective is to minimize the error between the predicted 3D pose and the true value. The test evaluation indicators are Where: N is the number of samples, T is the length of each sample sequence, and J is the number of key points on the human body.

[0030] The term "mixing" refers to combining the original data samples with the synthetic data samples and having them participate in the training of the 3D pose estimation model.

[0031] This embodiment uses PMR and H36M as raw data, and generates four types of augmented data through different cross-processing methods and Animate Anyone as the video generation module, which are used to train the 3D pose estimation model.

[0032] like Figure 3The table shows the generated video effect and the corresponding raw data, intermediate results, and ground truth. This embodiment uses PMR as pose samples and H36M as scene samples as the cross-validation method. The 2D keypoint coordinates are estimated using a 2D pose estimation model for both the raw and synthesized video data, and the error between the estimated and ground truth values ​​is calculated. Values ​​are calculated based on pixel distance. The experimental metrics in Table 1 can be considered as evaluation standards for the quality of the synthesized video. Experiments show that the generated video obtained using this method and embodiment is of similar quality to existing data, and therefore can be used for data augmentation and other applications.

[0033] Table 1

[0034]

[0035] Training and evaluation experiments on the downstream 3D pose estimation model show that training with the generated data constructed in this invention can significantly enhance the model's generalization performance across data domains. The MPJPE evaluation index (unit: mm) under different test data is obtained after training the 3D pose estimation model KTPFormer with the original data (i.e., the source dataset) and four generated datasets obtained through this method, combined with H36M. Both training and testing 2D data are derived from the 2D pose estimation model. Experimental metrics are shown in Table 2, and the results are illustrated in the figure below. Figure 4 As shown.

[0036] Table 2

[0037]

[0038] The original data (i.e., the source dataset) and the four generated datasets obtained through this method were merged with H36M to train the 3D pose estimation model KTPFormer. The MPJPE evaluation metrics (unit: mm) under different training and testing 2D keypoint input conditions are shown in Table 3. The test dataset is MPI-INF-3DHP.

[0039] Table 3

[0040]

[0041] Existing data augmentation techniques do not include video modalities, therefore the training 2D data obtained through these methods is ground truth (i.e., the upper part of Table 3). Although they achieve the best results when the test 2D data is also ground truth, their actual performance is worse in practice because only the test 2D estimates are valid. Conversely, this method, because it includes the generation of video modalities, can provide training 2D data as estimates (i.e., the lower part of Table 3), achieving relatively better results in practical applications where only test 2D estimates are available.

[0042] Compared with existing technologies, this method can achieve similar results by indirectly utilizing partial pose and scene information from H36M or PMR without directly introducing other existing datasets (i.e., PMR), thus enhancing generalization performance.

[0043] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A 3D human pose optimization estimation method based on generative data augmentation, characterized in that, Scene and pose decoupling is performed on raw data from different types of scenes. The target pose is transferred to the source scene through root node alignment and coordinate transformation. After generating a cross-domain fused 3D pose sequence, a controllable video generation model is used to generate a synthetic video containing diverse backgrounds, perspectives and actions. Then, the synthetic video and raw data are used to jointly train a 3D human pose estimation model. In the online stage, the trained 3D human pose estimation model is used for real-time pose estimation. The raw data used is a publicly available 3D human pose dataset, containing videos of human movements in specific scenes, i.e., RGB image sequences and their ground truth 3D keypoint positions in world coordinates, wherein: the RGB image sequences are... The 3D keypoint locations in world coordinates are , These are the number of samples, the length of each sample sequence, the number of image channels, the image height, the image width, and the number of human keypoints, respectively. The controllable video generation model, guided by an RGB reference image and a 2D keypoint representation of human pose, generates corresponding videos based on a stable diffusion algorithm, specifically as follows: ,in: Represents the first image in the image sequence. Location Sample No. Position frame, Represents the truth value of a 3D keypoint. The location of 2D key points after the location sample is projected onto the camera plane. Represents the generated first Video data of location samples; The 3D human pose estimation model predicts the position of the corresponding 3D pose key points in camera space based on the 2D human pose key points. The joint training mentioned above refers to: ,in: These are the predicted values ​​for 3D pose key points. These are the estimated 2D keypoint positions obtained from the 2D pose estimation model. For the ground truth values ​​of 3D pose keypoints. This represents the loss function of the 3D pose estimation model, measured by the MPJPE index between the predicted and ground truth values ​​of the 3D pose keypoints.

2. A 3D human pose optimization and estimation system based on generative data augmentation that implements the method of claim 1, characterized in that, include: The system comprises a cross-domain data fusion unit, a controllable video generation unit, and a 3D pose estimation training unit. The cross-domain data fusion unit aligns the root nodes of scene samples and pose samples in the original data and eliminates the differences between coordinate systems in different domains. Then, it transfers the target pose to the source scene through linear transformation to generate a cross-domain aligned 3D pose sequence. The controllable video generation unit projects the cross-domain aligned 3D pose sequence onto a 2D plane to generate a pose guidance sequence. This sequence is then combined with the RGB scene input to generate continuous frame video data. Finally, the synthesized data and the original data are jointly used to train a 3D pose estimation model for real-time prediction of 3D human pose.

Images