Video-based underwater 3D human pose estimation method and device

Abstract

This invention discloses a video-based underwater 3D human pose estimation method and apparatus. In the preprocessing stage, a Part Attention Regressor (MAR) is used to dynamically estimate the Smart-Small Pose (SMPL) human parameters, which is particularly adept at handling common underwater occlusion problems. Furthermore, a 2D keypoint detector using YOLO for bounding box detection and HRNet for keypoint regression enhances feature extraction under challenging underwater image conditions. In the optimization stage, an underwater variational autoencoder is used to adjust the SMPL model, learning from a range of possible underwater pose distributions. The optimization process incorporates constraints to align the final SMPL model with the detected 2D keypoints, minimizing the difference between the adjusted SMPL model and the original model, and ensuring the consistency of SMPL parameters over time. During the optimization stage, different optimization terms are assigned appropriate weights, and gradient descent is used for optimization to obtain the final result.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and proposes a video-based underwater 3D human pose estimation method and device. Background Technology

[0002] Early human motion pose estimation methods used stereo vision to reconstruct human geometric motion. These methods relied on dense pairs of view data, consuming significant computational resources and often exhibiting low robustness. Following the introduction of the SMPL statistical human model, numerous single-view estimation methods emerged. The representative SMPLify network uses a 2D pose convolutional network to estimate human keypoints and perform gradient-based optimization, allowing the model to better fit the observed image data. To address the occlusion problem in images, the PARE network supervises 49 masks of body parts through partial segmentation and uses an attention mechanism to leverage useful information from the body and surrounding pixels to obtain more accurate results.

[0003] Although monocular human pose estimation algorithms have made considerable progress, images obtained from underwater environments currently face refraction and reflection problems caused by the water surface. These problems significantly lead to a large discrepancy between the parameter results obtained by existing SMPL parameter regressors based on monocular images and the actual human pose, resulting in unbelievable effects and failing to achieve the goal of high-fidelity human motion reconstruction. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a video-based underwater three-dimensional human pose estimation method and device.

[0005] The objective of this invention is achieved through the following technical solution: Firstly, this invention provides a video-based underwater three-dimensional human pose estimation method, which includes the following steps:

[0006] (1) Obtain video frame sequences of underwater human motion and estimate SMPL parameters of underwater human posture;

[0007] (2) Obtain prior information on underwater human movement, process underwater human motion images containing both air and water media through an image encoder, extract high-dimensional feature information, fuse it with the SMPL parameters of underwater human posture obtained in step (1), and input it into the posture encoder to relearn and adjust posture features in the latent space.

[0008] (3) The attitude decoder decodes new human posture parameters from the latent space to obtain the estimated underwater three-dimensional human posture.

[0009] Further, in step (1), the attention of the body parts corresponding to the 2D heatmap is calculated and used as the weight of the 3D feature map to regress the final SMPL model and perform SMPL pose parameter estimation.

[0010] Furthermore, in step (2), the underwater human motion image includes the part of the human body in the air and the part in the water. The image encoder extracts high-dimensional feature information and distinguishes the water part affected by refraction and the air part that is not affected at the feature level, generating a more useful feature representation for eliminating the effect of refraction in the subsequent decoding stage.

[0011] Further, in step (2), the image feature vector from which the high-dimensional feature information is extracted is subjected to an inner product operation with the estimated SMPL pose parameters to obtain a new vector after feature fusion, which is then input into the pose encoder.

[0012] Furthermore, in step (3), the high-level feature representation learned by the posture encoder is used to distinguish the water part affected by refraction and the air part that is not affected in the human posture. The posture decoder corrects the estimation error caused by water surface refraction and reflection and decodes the new human posture parameters from the latent space.

[0013] Further, in step (1), the reprojection supervision error of the SMPL parameters is calculated based on 2D key points as supervision; the change in the length of the human skeleton is calculated based on the SMPL parameters, and the rate of change between 2D key points is measured to calculate the human pose velocity; the smoothness error in the video frame sequence is calculated based on the skeleton length and pose velocity; in step (3), the mean square error between the human pose parameters obtained by the pose decoder and the estimated SMPL pose parameters is calculated to obtain the prior loss of the SMPL parameters; the final loss function is obtained by combining the reprojection supervision error, smoothness error and prior loss.

[0014] Secondly, the present invention also provides a video-based underwater three-dimensional human pose estimation device, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the video-based underwater three-dimensional human pose estimation method.

[0015] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned video-based underwater three-dimensional human pose estimation method.

[0016] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned video-based underwater three-dimensional human pose estimation method.

[0017] The beneficial effects of this invention are:

[0018] 1. For underwater human motion scenarios, realistic underwater human motion reconstruction is achieved through monocular RGB sequences. This overcomes the challenges posed by water occlusion and low image quality to a certain extent, thereby significantly improving the accuracy of underwater 3D human pose estimation.

[0019] 2. To address the estimation error caused by water surface refraction and reflection, a WM-VAE network was designed to constrain the estimated motion parameters. In addition, this network can quickly generate a large number of underwater human motion parameters, enabling the creation of simulation datasets and solving the problem of scarce underwater human motion datasets.

[0020] 3. By using high-precision underwater human motion reconstruction as the observation result, we can provide underwater professional athletes or coaches with visualized motion trajectories to help them better standardize or learn underwater sports movements. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a video-based underwater 3D human pose estimation method provided by the present invention;

[0023] Figure 2 A schematic diagram of the underwater human motion generation module provided by the present invention;

[0024] Figure 3 This is a schematic diagram of the underwater human posture estimation results in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the visualization results obtained by the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of an underwater three-dimensional human posture estimation device based on video provided by the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0029] like Figure 1 As shown, this invention proposes a video-based underwater 3D human pose estimation method. For underwater human motion scenarios, it achieves realistic underwater human motion reconstruction using monocular RGB sequences. This method overcomes, to some extent, the challenges posed by water occlusion and low image quality. However, in underwater motion scenarios, the actual position of the human body differs from its observed position in the image due to water refraction. This problem affects the results obtained by ordinary image-based SMPL regressors, leading to inaccurate results.

[0030] The overall execution flow of this invention comprises two main stages: a preprocessing stage and an optimization stage. The preprocessing stage calculates the initial human SMPL parameters and 2D keypoint coordinates. Specifically:

[0031] Initial SMPL parameter estimation: PARE was chosen as the regressor for the single-frame SMPL model. This regressor regresses the final SMPL model by calculating the body part attention corresponding to the 2D heatmap and using it as weights for the 3D feature map. When some body parts are occluded, the network can implicitly eliminate their contribution to the final result. Due to the prevalence of partial occlusion in underwater scenes, PARE's advantages are particularly significant. When using a weak perspective camera model, the regressed SMPL model can be represented as Θ={θ,β,R,t}, where θ represents pose, β represents shape, and R and t are rotation and translation matrices, respectively.

[0032] 2D Keypoint Detection: A pre-trained YOLO detector is used to extract bounding boxes of the human body in each frame. HRNet is then selected as the backbone network for human keypoint detection. HRNet preserves high-resolution features of the image during inference, providing more accurate and complete location information. This feature is crucial for addressing low-quality images caused by motion blur and water splash occlusion in underwater motion scenes. The predicted 2D keypoints are represented as a set of x... i ∈R 2×K , where i represents the i-th 2D key point.

[0033] The optimization phase involves refining the SMPL motion parameters and addressing issues in underwater scenes such as occlusion and reflections to obtain more accurate results. Details are as follows:

[0034] Underwater human motion data is often hampered by issues such as water wave occlusion, surface reflection, and refraction. Human motion in each frame of video is distorted to some extent by refraction, and occlusion can easily lead to unreasonable SMPL parameter estimation results. To address this, we propose the Underwater Human Motion Generation Module (WM-VAE). This module learns a low-dimensional latent space suitable for underwater motion scenarios, with different latent vectors corresponding to different human poses. The module aims to understand and represent the latent space of human poses, thereby generating realistic pose samples and providing continuous pose manipulation and control. It can rapidly generate large amounts of high-quality underwater human motion data, providing data support for related tasks. The module incorporates prior information about underwater motion into the final reconstruction result, effectively avoiding unreasonable output results caused by water wave occlusion, surface reflection, and refraction. Three loss mechanisms are used in each optimization step to further enhance the robustness of inferred human motion.

[0035] Prior supervision of underwater motion: In scenarios involving underwater human movement, part of the body is typically in the air, while the lower half is underwater. The captured images, especially of the lower half, suffer from water refraction, causing a discrepancy between the actual position and the observed position in the image. This problem renders the results of a standard image-based SMPL regressor inaccurate due to the influence of refraction.

[0036] like Figure 2 As shown, the underwater human motion generation module network utilizes a variational autoencoder (VAE) framework that includes an encoder and a decoder. The image encoder first processes the input underwater human motion image, extracting its high-dimensional feature information. Since the human body in the image is situated in both air and water, the encoder can learn to distinguish between the parts affected by refraction (lower body) and the unaffected parts (upper body) at the feature level, thereby generating more useful feature representations. The feature vector after encoder processing contains the distinguishing information between the air and underwater parts. This information is used to eliminate the effects of refraction in the subsequent decoding stage. In addition, it can also reduce noise introduced by the underwater environment.

[0037] The SMPL motion parameters obtained by the pre-trained regressor do not correct for refraction error. The feature vector of the image is combined with the pose parameters obtained by the SMPL regressor to perform an inner product operation to obtain a new vector after feature fusion, which is then input into the pose encoder.

[0038] The pose encoder further encodes the initial SMPL parameters into a compact latent representation. The purpose of this step is to relearn and adjust pose features in the latent space. Through the high-level feature representation learned by the pose encoder, the network is able to better distinguish between the parts of the human pose affected by refraction and those that are not.

[0039] The pose decoder decodes new human pose parameters from the latent space. This process utilizes the high-level feature representations learned by the pose encoder to correct estimation errors caused by water refraction and reflection, thus making the final result more accurate.

[0040] When training the WM-VAE network, only human underwater motion datasets (such as SwimXYZ, CMU-MocapSub125, 126) are used for training. The image encoder part uses ResNet as the network skeleton, and the data size of the output results is consistent with that of the SMPL regressor.

[0041] The initial pose parameters θ of the SMPL model and the underwater human motion feature vector obtained through the image encoder are used as inputs to the WM-VAE to obtain a more reasonable pose. Then, by calculating θ and The mean squared error (MSE) between the two frames is used to obtain the prior loss for the initial SMPL parameters. The formula is shown below, where i represents the number of video frames:

[0042]

[0043] Calculate the reprojection supervision error. The 2D keypoints predicted in the preprocessing stage will be used as supervision. The reprojection loss error of the SMPL parameters is calculated using the following formula:

[0044]

[0045] Where, x i ∈R 2×K This represents the pixel coordinates of the i-th predicted 2D keypoint. Let represent the pixel coordinates of the i-th 2D keypoint reprojected from the SMPL model Θ. For a given Θ, the pixel coordinates of the reprojected 2D keypoints can be calculated using the following formula, where ... Let t represent the video frame space rotation matrix, and t represent the video frame space translation vector. This represents the joint regression matrix of 3D key points in the human SMPL model, where Π represents the orthogonal projection calculation.

[0046]

[0047] Calculating the predicted smoothness supervision error: The smoothness error in a video frame sequence consists of two components: skeleton length and pose velocity. Skeleton length is calculated by evaluating the change in skeleton length between the current frame and the previous frame, using the SMPL parameter. During human motion, the skeleton length should remain constant. Pose velocity is calculated by measuring the rate of change between keypoints, serving as an indicator of the degree of pose change. Smaller velocity values ​​indicate smoother pose transitions.

[0048] By optimizing the motion smoothness loss, more accurate and stable results can be obtained. The formula for calculating the smoothness error is as follows, where L represents the skeleton length, V represents the velocity of the human keypoints, and θ... t This represents the motion parameters of the human SMPL in the t-th video frame.

[0049]

[0050] Optimization of human SMPL parameters: The shape parameter β is calculated by averaging the corresponding parameters in each frame of the entire underwater motion sequence. The pose parameter θ in all frames is optimized by optimizing the three loss functions mentioned above. These three loss functions are assigned appropriate weights to obtain the final loss function, as shown in the formula below, where λ1, λ2, and λ3 represent the weights of their respective terms:

[0051] L tota; =λ1L prior +λ2L reproj +λ3L smooth

[0052] Ultimately, the pose estimation problem involves minimizing the loss function and gradually approaching the optimal solution through an iterative optimization process. In SwimXYZ, the visualization results obtained by this algorithm are as follows: Figure 4 As shown. Gradient descent is used in the optimization process. When the change in θ before and after an iteration is less than a preset threshold, the optimization iteration stops, and this is taken as the final result.

[0053] After training, the underwater human motion generation module, through randomization of the latent space vectors, produces a new human SMPL, as shown below. Figure 3 As shown, most of the movements are those of human swimming.

[0054] Corresponding to the aforementioned embodiment of a video-based underwater 3D human pose estimation method, the present invention also provides an embodiment of a video-based underwater 3D human pose estimation device.

[0055] See Figure 5 The present invention provides a video-based underwater three-dimensional human pose estimation device, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a video-based underwater three-dimensional human pose estimation method in the above embodiment.

[0056] The embodiment of the underwater 3D human pose estimation device based on video provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which includes the video-based underwater 3D human pose estimation device provided by this invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0057] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0058] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0059] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a video-based underwater three-dimensional human pose estimation method as described in the above embodiments.

[0060] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0061] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned video-based underwater three-dimensional human pose estimation method.

[0062] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0063] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A video-based underwater 3D human pose estimation method, characterized in that, The method includes the following steps: (1) Obtain the video frame sequence of underwater human motion and estimate the SMPL parameters of underwater human posture; specifically: use 2D key points as supervision to calculate the reprojection supervision error of SMPL parameters; calculate the change of human skeleton length based on SMPL parameters, measure the rate of change between 2D key points to calculate human posture velocity, and calculate the smoothness error in the video frame sequence based on skeleton length and posture velocity. (2) Obtain prior information on underwater human movement, process underwater human motion images containing both air and water media through an image encoder, extract high-dimensional feature information, fuse it with the SMPL parameters of underwater human posture obtained in step (1), and input it into the posture encoder to relearn and adjust posture features in the latent space. (3) The attitude decoder decodes new human posture parameters from the latent space, calculates the mean square error between the human posture parameters obtained by the attitude decoder and the estimated SMPL posture parameters, and obtains the prior loss of the SMPL parameters; the final loss function is obtained by combining the reprojection supervision error, smoothness error and prior loss, and the estimated underwater three-dimensional human posture is obtained by optimization.

2. The underwater 3D human pose estimation method based on video according to claim 1, characterized in that, In step (1), the attention of the body parts corresponding to the 2D heatmap is calculated and used as the weight of the 3D feature map to regress the final SMPL model and estimate the SMPL pose parameters.

3. The underwater 3D human pose estimation method based on video according to claim 1, characterized in that, In step (2), the underwater human motion image includes the part of the human body in the air and the part in the water. The image encoder extracts high-dimensional feature information and distinguishes the water part affected by refraction and the air part that is not affected at the feature level, generating a more useful feature representation for eliminating the effect of refraction in the subsequent decoding stage.

4. The underwater 3D human pose estimation method based on video according to claim 1, characterized in that, In step (2), the image feature vector from which the high-dimensional feature information is extracted is subjected to an inner product operation with the estimated SMPL pose parameters to obtain a new vector after feature fusion, which is then input into the pose encoder.

5. The underwater 3D human pose estimation method based on video according to claim 1, characterized in that, In step (3), the high-level feature representation learned by the posture encoder is used to distinguish the water part affected by refraction and the air part that is not affected in the human posture. The posture decoder corrects the estimation error caused by water surface refraction and reflection and decodes the new human posture parameters from the latent space.

6. A video-based underwater 3D human pose estimation device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements a video-based underwater three-dimensional human pose estimation method as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a video-based underwater three-dimensional human pose estimation method as described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a video-based underwater three-dimensional human pose estimation method as described in any one of claims 1-5.

Images