Three-dimensional attitude estimation method based on pre-training generative model
Through the three-dimensional pose estimation method based on pre-trained generative model, combined with the action generation model and diffusion model, the problem of insufficient accuracy of complex action prediction in the existing technology is solved, and high-accuracy and low-cost 3D pose estimation is achieved, which is suitable for Complex environment.
Patent Information
- Application Number
- CN202411265892.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-26
AI Technical Summary
The existing 3D pose estimation methods have insufficient prediction accuracy in complex actions and difficult environments, and have high demands on data dependence and computing resources, and sensor fusion increases system complexity and cost.
A three-dimensional pose estimation method based on pre-trained generative models is adopted, and features are extracted from 3D and 2D data through the action generation model and information is shared. The diffusion model is used for denoising, so as to achieve end-to-end fast prediction and high accuracy.
Significantly improve the accuracy of prediction of complex movements and the fit of real human movement laws, reduce calculation costs, maintain adaptability to high-difficulty movements and complex environments, and show lower average key point position errors and more High prediction accuracy.
Smart Images

Figure CN119028024A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer vision technology, and in particular to a three-dimensional posture estimation method based on a pre-trained generation model. Background Art
[0002] 3D pose estimation is a key technology in computer vision and is widely used in autonomous driving, robotics, virtual reality (VR), augmented reality (AR), and human motion analysis. Through 3D pose estimation, the system can infer the pose information of an object or a human body in three-dimensional space from a two-dimensional image or video. The core of this technology is to extract and reconstruct three-dimensional structures from two-dimensional data.
[0003] Current 3D pose estimation methods can be mainly divided into the following categories: 1) Deep learning-based methods: These methods rely on large-scale annotated datasets and deep neural networks to estimate 3D poses by extracting features from images. These methods have high accuracy and adaptability, but they also have problems such as strong data dependence and high computing resource requirements. 2) Multi-view based methods: This method shoots objects or human bodies from multiple perspectives and then combines multi-view data for 3D reconstruction. These methods perform well in obtaining accurate poses, but in practical applications they are often limited by scene complexity and camera configuration requirements. 3) Sensor fusion-based methods: These methods combine multiple sensor data such as RGB cameras, depth cameras (such as Kinect), inertial measurement units (IMUs), etc. to enhance the accuracy and robustness of 3D pose estimation. However, the increase in sensors also brings about an increase in system complexity and cost, and errors are prone to occur in the process of multi-sensor data fusion.
[0004] Based on deep learning methods, MotionBERT uses existing 2D and 3D data for pre-training, and then conducts fusion training of multiple subtasks such as posture estimation and action recognition. WHAM uses SLAM technology to fit camera features, and finally estimates the entire person's movements and ground movement trajectory for a motion video. Similar posture estimation interfaces of Google and MMLab can perform good 3D posture estimation of people in the video. However, these technologies will still have partial distortion when more complex movements occur, resulting in inaccurate predictions.
[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention
[0006] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a three-dimensional posture estimation method based on a pre-trained generation model.
[0007] To achieve the above object, the present invention adopts the following technical solutions:
[0008] In a first aspect of the present invention, a three-dimensional pose estimation method based on a pre-trained generative model comprises:
[0009] S1, pre-training action generation model processing stage:
[0010] Extracting features of 3D motion data from input 3D motion capture data through the 3D encoding layer of the motion generation model;
[0011] Extract the features of 2D motion data from the 2D image or video projected from the 3D motion data through the 2D encoding layer of the motion generation model;
[0012] The extracted 3D and 2D features are fused through an information sharing layer to enhance the model’s understanding of the action;
[0013] S2, 3D posture estimation model processing stage:
[0014] Through the 2D decoding layer of the 3D posture estimation model, the features of the received 2D motion data are decoded to restore the 2D skeleton information;
[0015] Combining the output of the 2D decoding layer and the features fused by the information sharing layer, the 3D posture is further decoded by the 3D decoding layer of the 3D posture estimation model to obtain the 3D posture.
[0016] In a second aspect of the present invention, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the three-dimensional pose estimation method based on a pre-trained generative model.
[0017] In a third aspect of the present invention, a computer program product includes a computer program, and when the computer program is executed by a processor, the three-dimensional posture estimation method based on the pre-trained generative model is implemented.
[0018] The present invention has the following beneficial effects:
[0019] The method of the present invention is more consistent with the real human body movement rules for predicting actions and realizes end-to-end fast prediction. After the model of the present invention is trained, for a video to be estimated, the existing 2D posture estimation model can be used to extract the 2D skeleton information in the video, and then the model of the present invention is used to promote the 2D skeleton to a 3D skeleton, thereby realizing the estimation of the entire 3D posture.
[0020] The innovative three-dimensional human posture estimation method of the present invention significantly improves the prediction accuracy of complex actions and the degree of fit to the real human motion law by combining the pre-trained generation model with the multi-task learning strategy. Utilizing a large amount of 3D motion capture data and its 2D projection, the motion generation model of the invention can effectively extract and fuse the features of 3D and 2D motion data from two-dimensional images or videos, and enhance the model's understanding of the action through the information sharing layer. In addition, the diffusion model is preferably used as the generation framework, and the feature extraction and posture prediction are optimized by simulating the random process of data distribution and step-by-step denoising. In practical applications, the present invention can use the existing 2D posture estimation model to extract the 2D skeleton information in the video and upgrade it to a 3D skeleton, achieving end-to-end fast prediction while maintaining adaptability to difficult actions and complex environments. Compared with traditional methods, the present invention shows lower average key point position error and higher prediction accuracy on the Human3.6M and 3DPW datasets, proving its first-class level and great potential in the field of three-dimensional human posture estimation. In addition, as a pre-training method, the present invention is universal and can improve existing models and give them better prior knowledge without excessive computational costs, making it possible to optimize the model within controllable resources.
[0021] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is an algorithm architecture diagram of a 3D human posture estimation method based on a pre-trained generative model according to an embodiment of the present invention.
[0023] Figure 2 The following is a simplified flowchart of implementing 3D posture estimation in an embodiment of the present invention.
[0024] Figure 3 This is a diagram of the prediction results of the embodiment of the present invention on the Human3.6M and 3DPW datasets. DETAILED DESCRIPTION
[0025] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
[0026] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0027] See also Figure 1 and Figure 2 The embodiment of the present invention provides a 3D posture estimation method based on a pre-trained generative model, which can be used for (but not limited to) 3D human posture estimation, including:
[0028] S1, pre-trained motion generation model processing stage: extract the features of 3D motion data from the input 3D motion capture data through the 3D encoding layer of the motion generation model; extract the features of 2D motion data from the 2D image or video projected from the 3D motion data through the 2D encoding layer of the motion generation model; fuse the extracted 3D and 2D features through the information sharing layer to enhance the model's understanding of the motion;
[0029] S2, 3D posture estimation model processing stage: Through the 2D decoding layer of the 3D posture estimation model, the features of the received 2D motion data are decoded to restore the two-dimensional skeleton information; combined with the output of the 2D decoding layer and the features fused by the information sharing layer, the three-dimensional posture is further decoded through the 3D decoding layer of the 3D posture estimation model.
[0030] In some embodiments, a diffusion model is used as a generation framework to perform denoising on the fused extracted features to predict accurate three-dimensional poses. Specifically, the process includes: initializing the action generation model, using the diffusion model as a generation framework to simulate the random process of data distribution; performing a forward denoising step on the input 3D and 2D action data, gradually adding Gaussian noise until the data distribution is close to a normal distribution; performing denoising at time step t based on the DDPM framework, predicting and removing noise to gradually restore the original state of the data; in the denoising process, the model learns how to predict the noise added in step t based on the denoising result of step t+1 to achieve reverse generation of data; through iterative denoising steps, gradually reducing noise and refining data features until a clear three-dimensional pose is restored.
[0031] In some embodiments, step S1 includes: projecting 3D motion capture data to 2D to generate 2D motion data of multiple perspectives; applying noise to the 3D motion data and the 2D motion data projected therefrom to generate 3D noise and 2D noise; processing the corresponding 3D noise and 2D noise through a 3D coding layer and a 2D coding layer to extract intermediate features; using an information sharing layer to integrate the intermediate features extracted by the 3D coding layer and the 2D coding layer to form a fused feature; step S2 includes: using a 3D decoding layer and a 2D decoding layer to respectively restore 3D and 2D clean motion data according to the fused features.
[0032] In some embodiments, the 3D encoding layer, the 2D encoding layer, the information sharing layer, the 3D decoding layer and the 2D decoding layer are all based on layers of a transformer architecture.
[0033] Furthermore, for paired 3D and 2D motion sequences, four MSE losses are cross-established for the 3D coding layer, 2D coding layer, 3D decoding layer, and 2D decoding layer, and the weighted sum of the four MSE losses is used as the final loss function.
[0034] The innovative three-dimensional human posture estimation method of the present invention significantly improves the prediction accuracy of complex actions and the degree of fit to the real human motion law by combining the pre-trained generation model with the multi-task learning strategy. Utilizing a large amount of 3D motion capture data and its 2D projection, the motion generation model of the invention can effectively extract and fuse the features of 3D and 2D motion data from two-dimensional images or videos, and enhance the model's understanding of the action through the information sharing layer. In addition, the diffusion model is preferably used as the generation framework, and the feature extraction and posture prediction are optimized by simulating the random process of data distribution and step-by-step denoising. In practical applications, the present invention can use the existing 2D posture estimation model to extract the 2D skeleton information in the video and upgrade it to a 3D skeleton, achieving end-to-end fast prediction while maintaining adaptability to difficult actions and complex environments. Compared with traditional methods, the present invention shows lower average key point position error and higher prediction accuracy on the Human3.6M and 3DPW datasets, proving its first-class level and great potential in the field of three-dimensional human posture estimation. In addition, as a pre-training method, the present invention is universal and can improve existing models and give them better prior knowledge without excessive computational costs, making it possible to optimize the model within controllable resources.
[0035] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.
[0036] Most of the existing posture estimation models are directly end-to-end learning, or use reconstruction as a pre-training task. The present invention is based on the 3D posture estimation method of the pre-trained generative model, and designs a more effective pre-training task based on the pre-training of the generative model. The main learning process is divided into two steps. The first step is to train a motion generation model based on a large amount of 3D motion capture data and its 2D projection. The second step is to train the 3D posture estimation model using the motion generation model as the pre-training weight.
[0037] Figure 1 The algorithm architecture of the 3D human body posture estimation method based on the pre-trained generative model of the embodiment of the present invention is shown. The present invention uses the generative model as the pre-trained model for 3D posture estimation, and on this basis, includes a fusion training method combining 3D denoising and 2D denoising, and multi-task training during prediction task training.
[0038] In the task of training the action generation model, the generation framework adopted by the preferred embodiment is the diffusion model. The core of the model is to continuously add Gaussian noise to a data distribution. When the number of steps tends to infinity, the data distribution becomes a normal distribution. On the contrary, conditional denoising is performed on a normal distribution, and it restores the original image. The model is trained to know how to denoise. Based on the DDPM framework, the denoising in the tth step is equivalent to adding a noise. The added noise is a posterior distribution based on the denoising result of the t+1th step, and its mean is related to the noise of the denoising in the tth step. Therefore, the task of model training is to predict the noise of the denoising in the tth step based on the denoising result of the t+1th step. The structural diagram of the generation model is shown in the figure. Figure 1 As shown in the figure, when training the generative model, in order to better integrate 2D and 3D information, the 3D information is projected to multiple 2D perspectives, and the 3D action information and 2D action information are denoised. There are five modules in the middle, namely, 3D encoding layer E 3D , 2D coding layer E 2D , information sharing layer E share , 3D decoding layer D 3D , 2D decoding layer D 2D , each layer is a layer transformer module. The input can be 3D noise or 2D noise, and the intermediate features are obtained after passing through the respective encoding layers together with the number of noise addition steps.
[0039] f1=E 3D (x t,3D ,),f1=E 2D (x t,2D ,)
[0040] Then f2 can be obtained through the same information sharing layer.
[0041] f2=E share (f1)
[0042] f2 is the extracted effective information, and the clean motion data can be restored through the respective decoding layers.
[0043]
[0044] When constructing the loss, the paired input 3D and 2D motion sequences can be cross-constructed into four MSE losses, and the final loss is the weighted sum of the four MSE losses. Considering the current largest 3D motion capture dataset, the AMASS dataset is used to train the motion generation model.
[0045] Perform 3D pose estimation as Figure 2 As shown, using the 2D encoder E 2D , information sharing layer Eshare , 3D decoding layer D 3D , for the number of noise steps, let it be 0, and the 3D action sequence can be predicted using the 2D action sequence. That is:
[0046]
[0047] By using the pre-training results of the generative model, the 3D posture output by the model can be more consistent with the actual 3D action sequence distribution, allowing the model to converge to a better extreme point.
[0048] When training the model, in order to consolidate the robustness of the model, the method of projecting the training 3D data to multiple perspectives for training was adopted. At the same time, the multi-task training method commonly used in the past was introduced to simultaneously learn the tasks of 3D pose estimation, action recognition, and 3D patch recovery. For the other two tasks, the two tail transformer layers were removed and the new transformer layer and linear layer were reinitialized.
[0049] After the model is trained, for a video to be estimated, the existing 2D posture estimation model can be used to extract the 2D skeleton information in the video, and then the model of the present invention can be used to upgrade the 2D skeleton to a 3D skeleton, thereby realizing the estimation of the entire 3D posture.
[0050] experiment
[0051] We tested the method on two classic 3D pose estimation datasets and compared it with previous cutting-edge methods. The main datasets used are Human3.6M and 3DPW. Human3.6M contains more than 3 million video frames. It is an indoor motion capture system with 3D human key point information collected by four-view cameras. The 3DPW dataset is an outdoor collection of pose data collected by cameras and IMU systems. The total data volume is 60 video sequences with about 30,000 frames of images. Due to the complexity of outdoor scenes, the tasks on this dataset are more difficult.
[0052] The evaluation index of Human3.6M mainly uses the mean key point position error (MPJPE). There are a total of fifteen action categories showing different action characteristics. Different categories are measured separately and the overall average value is calculated. The comparison results of the present invention and other methods are shown in Tables 1 and 2 below:
[0053] Table 1
[0054]
[0055] Table 2
[0056]
[0057] In Table 1, the 2D posture sequence estimated by the CPN model is used as input, and in Table 2, the real 2D key point information is used as the input condition. The best value of each item is marked with a darker mark. The data in Table 1 shows that in actual application scenarios, based on the entire algorithm flow of 2D posture estimation, the method of the present invention can obtain cutting-edge 3D posture estimation results. Table 2 studies the upper limit of the method based on Table 1, that is, assuming that the 2D posture is absolutely accurate, the estimation error of the 3D posture, the method of the present invention also achieves a good effect.
[0058] For the dataset 3DPW, the main evaluation indicators used are the mean key point position error (MPJPE), the Protoss alignment mean key point position error (PA-MPJPE), and the human mesh vertex error (PVE), and the measurement unit is millimeter (mm). The comparison with other methods is shown in Table 3:
[0059] Table 3
[0060]
[0061] The present invention still uses the 2D posture sequence estimated by the model as input, while some other methods use pictures as input. The model of the present invention shows the best experimental results in all indicators, demonstrating the accuracy of the 3D posture predicted by the present invention and its adaptability to difficult actions. It can still make correct and reasonable estimates when dealing with complex environments and actions with occlusion.
[0062] The visualization prediction results of the present invention on the two datasets of Human3.6M and 3DPW are shown in Figure 3 The left side shows the result of 2D posture estimation for the image, the red solid line in the right figure is the real data result, and the blue dotted line is the estimation result of the present invention. It can be seen that the posture estimated by the present invention is basically consistent with the real posture in terms of overall action. Due to the prior information brought by the action generation, the estimated action also strongly fits the actual human body action, and the overall accuracy is higher, that is, it can better utilize the three-dimensional correlation information of the human body to perform 3D posture estimation.
[0063] Alternative Example
[0064] During the entire training and modeling process, there are many alternative directions that can be optimized. The core of the present invention is to use a generative model as the pre-trained weight of the prediction model. The following mainly describes other implementation methods of the model from three directions, including different backbone networks, different generation strategies, and different training methods.
[0065] 1) Different backbone networks. It is preferred to use the transformer module as the intermediate encoding network layer, but each layer can be changed according to the form of the input data. For example, GRU, LSTM, etc. in the language model can be used, or linear layers and convolutional layers can be used for extraction.
[0066] 2) Different generation strategies. The entire generation model uses the diffusion model, which is currently the most widely used and most effective. However, there are still many types of generation models, such as variational autoencoders (VAE), generative adversarial networks (GANs), and normalization flows. Different generation models have different effects on different data. In terms of the training mode of the present invention, the best experimental effect is to use the diffusion model.
[0067] 3) Different training methods. When training the denoising model, an end-to-end training method is adopted to directly train the entire model. Another implementation method is to first train the 3D encoding layer E 3D , 2D coding layer E 2D , 3D decoding layer D 3D , 2D decoding layer D 2D , projecting 3D and 2D data into the same latent space, and then learning the denoising model on the latent space. This method is difficult to learn because it is difficult to project the data of the two modalities into the same latent space to establish an association, so this method was not adopted in the end.
[0068] In summary, the present invention proposes an innovative method for 3D human posture estimation, which breaks through the limitation of previous reliance on regression models through a modeling method based on generative models. The method can use a large amount of motion capture data to pre-train 3D human posture, and the generative model can obtain better pre-training weights, thereby improving the accuracy and efficiency of prediction. Experimental results show that compared with traditional methods, the present invention has reached the first-class level in the field in terms of prediction effect, showing great potential and application value.
[0069] In terms of solving industry pain points, the present invention effectively addresses problems such as insufficient training data, inconsistency between prediction results and human posture structure, and poor fit between images and posture estimation. Through the pre-training method, the present invention not only improves the accuracy of posture estimation, but also has versatility, can enhance the performance of existing models, and give them stronger prior knowledge. In addition, the present invention performs well in computing cost control and can achieve model optimization under limited resources.
[0070] The application scenarios of the present invention are wide-ranging, covering multiple fields such as entertainment, medical treatment, autonomous driving, sports analysis and manufacturing. In the fields of entertainment and games, it can be used in VR / AR systems and motion capture technology to enhance immersion and animation effects. In the medical field, it can be used to analyze human morphology to assess health conditions, assist rehabilitation training and surgical navigation. In the field of autonomous driving, it can be used to monitor driver status and environmental perception to improve safety. In the field of sports analysis, it can be used to optimize sports performance and help prevent injuries. In the fields of manufacturing and robotics, it can be used for the operation of industrial robots and human-machine collaboration. In addition, the present invention can also save costs and improve production efficiency through motion capture in game and film production.
[0071] In general, the present invention provides an efficient, accurate and widely used solution in the field of three-dimensional human posture estimation through its technical innovation mechanism, which has good application prospects and high application value.
[0072] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.
[0073] The present invention also provides a control device, including a processor and a computer program storage device. Sequence storage medium wherein the processor is used to execute the computer program at least as described above described method.
[0074] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.
[0075] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0076] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0077] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0078] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0079] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; The aforementioned storage media include: mobile storage devices, read-only Memory (ROM, Read-Only Memory), random access memory (RAM), disk or CD-ROM, etc. can store program code.
[0080] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0081] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.
[0082] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.
[0083] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0084] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A three-dimensional pose estimation method based on a pre-trained generative model, characterized in that: include: S1, pre-training action generation model processing stage: Extracting features of 3D motion data from input 3D motion capture data through the 3D encoding layer of the motion generation model; Extract the features of 2D motion data from the 2D image or video projected from the 3D motion data through the 2D encoding layer of the motion generation model; The extracted 3D and 2D features are fused through an information sharing layer to enhance the model’s understanding of the action; S2, 3D posture estimation model processing stage: Through the 2D decoding layer of the 3D posture estimation model, the features of the received 2D motion data are decoded to restore the 2D skeleton information; Combining the output of the 2D decoding layer and the features fused by the information sharing layer, the 3D posture is further decoded by the 3D decoding layer of the 3D posture estimation model to obtain the 3D posture.
2. The three-dimensional pose estimation method based on the pre-trained generative model according to claim 1, characterized in that: A diffusion model is adopted as the generative framework and the fused extracted features are denoised to predict accurate 3D pose.
3. The three-dimensional pose estimation method based on the pre-trained generative model as claimed in claim 2, characterized in that: The diffusion model is used as the generation framework to denoise the fused extracted features to predict accurate 3D poses, including: Initialize the action generation model and use the diffusion model as the generation framework to simulate the random process of data distribution; For the input 3D and 2D motion data, perform the forward noise addition step and gradually add Gaussian noise until the data distribution is close to the normal distribution; At time step t, based on the DDPM framework, denoising is performed to predict and remove noise to gradually restore the original state of the data; In the denoising process, the model learns how to predict the noise added in step t based on the denoising result in step t+1, thus realizing the reverse generation of data; Through iterative denoising steps, the noise is gradually reduced and the data features are refined until a clear 3D posture is restored.
4. The three-dimensional pose estimation method based on the pre-trained generative model as claimed in claim 2 or 3, characterized in that: Step S1 includes: projecting 3D motion capture data to 2D to generate 2D motion data of multiple viewing angles; applying noise to the 3D motion data and the 2D motion data projected therefrom to generate 3D noise and 2D noise; processing the corresponding 3D noise and 2D noise through a 3D coding layer and a 2D coding layer to extract intermediate features; and integrating the intermediate features extracted by the 3D coding layer and the 2D coding layer using an information sharing layer to form fusion features; Step S2 includes: using the 3D decoding layer and the 2D decoding layer to respectively restore 3D and 2D clean motion data according to the fusion features.
5. The three-dimensional pose estimation method based on the pre-trained generative model according to any one of claims 1 to 4, characterized in that: The 3D encoding layer, the 2D encoding layer, the information sharing layer, the 3D decoding layer and the 2D decoding layer are all layers based on the transformer architecture.
6. The three-dimensional pose estimation method based on a pre-trained generative model according to any one of claims 1 to 4, characterized in that: For paired 3D and 2D motion sequences, four MSE losses are cross-established for the 3D encoding layer, 2D encoding layer, 3D decoding layer, and 2D decoding layer, and the weighted sum of the four MSE losses is used as the final loss function.
7. The three-dimensional pose estimation method based on the pre-trained generative model as claimed in claim 4, characterized in that: The 3D posture estimation includes directly using the 2D action sequence to predict the 3D action sequence without adding noise, that is, realizing accurate 3D posture recovery through continuous processing from the 2D encoding layer, the information sharing layer to the 3D decoding layer.
8. The three-dimensional pose estimation method based on a pre-trained generative model according to any one of claims 1 to 7, characterized in that: Using a multi-task training method, the model not only learns 3D pose estimation, but also simultaneously learns action recognition and 3D patch restoration tasks.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional pose estimation method based on the pre-trained generative model as described in any one of claims 1 to 8 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional pose estimation method based on the pre-trained generative model as described in any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Monocular three-dimensional human body posture estimation method and device based on diffusion model and feature fusion
CN119810917A
