A 3D human body parametric model estimation method combining regression and optimization

Abstract

The present invention discloses a three-dimensional human body parametric model estimation method that combines regression and optimization. By using three-dimensional human body joints as a bridge, the regression algorithm and optimization algorithm commonly used in the task of three-dimensional human body parametric model estimation are combined. In order to give full play to their respective strengths, the present invention uses the regression algorithm to perform neural network regression on the key points of the three-dimensional human body joints and the optimization algorithm to perform inverse dynamics solution on the predicted three-dimensional human body joint key points. The optimization branch implements step-by-step iterative solution in the neural network and serves as one of the layers, namely the optimization layer. Through the designed optimization algorithm, the inverse kinematics optimization layer we designed can be trained together with the neural network and the optimization results can be fed back through the gradient to train the parameters of the neural network. Thereby achieving the effect of end-to-end three-dimensional human body parametric model estimation.

Description

Technical Field

[0001] The present invention relates to a method for estimating an end-to-end three-dimensional human body parameterized model by combining the respective advantages of optimization and regression, and belongs to the field of computer vision. Background Art

[0002] In recent years, 3D human parametric model estimation has been widely used in video analysis, camera surveillance, human-computer interaction, virtual reality, and augmented reality. Monocular 3D human parametric model estimation aims to estimate a parametric model representing human pose and shape from a single image or monocular video. Currently, the main techniques fall into two categories: optimization-based methods and regression-based methods. Optimization-based methods use an iterative optimization approach to fit the output of a parametric model (such as the SMPL model) to 2D input observations. They can produce precise grid-image alignment and maintain consistency between input data and predictions. However, due to their underdetermined and non-convex nature, these methods often only find local minima, and their optimization results depend on the initial values of the iterations. Conversely, regression-based methods treat the problem of human parametric model estimation as a regression problem. These methods typically train deep neural networks to directly regress the model parameters, thus avoiding the complex iterative optimization process. However, despite their efficiency, regression-based methods often struggle to ensure alignment between the estimated grid and the image.

[0003] Some existing studies have attempted to combine optimization processes with regression techniques. These works either use post-optimization schemes to improve regression results during the inference phase or use optimization algorithms to provide supervision signals for the training loop. The former approach tends to rely heavily on the quality of the regression results, while the latter approach is often subject to problems such as inaccurate supervision and initialization sensitivity. Recently, a method called HybrIK derives an analytical solution for inverse kinematics by torsional swing decomposition, thus avoiding the iterative calculation process. However, this method requires the assumption of consistent bone lengths, and this assumption sacrifices some inverse kinematics accuracy.

[0004] In summary, we know that the main challenges faced by 3D human body parametric model estimation technology so far are:

[0005] 1) Optimization-based methods often fall into local minima and their optimization results depend on the initial value of the iteration

[0006] 2) The estimation results of regression-based methods often cannot guarantee the alignment between the estimated grid and the image.

[0007] 3) Current methods based on the combination of optimization and regression are heavily dependent on the quality of regression results or suffer from problems such as inaccurate supervision and initialization sensitivity, and cannot fully leverage the respective advantages of the two methods. Summary of the Invention

[0008] The purpose of the present invention is to solve the difficult problems faced by the three-dimensional human body parameter model estimation technology pointed out in the background technology.

[0009] In order to achieve the above object, the technical solution of the present invention is to provide a three-dimensional human body parameter model estimation method combining regression and optimization, characterized in that it includes the following steps:

[0010] Step 1: Crop a given image so that the cropped image contains only one person;

[0011] Step 2: Input the data obtained in the previous step into the working model, which includes a backbone network and multiple head networks, where:

[0012] The backbone network is used to extract features from the data input into the working model to obtain feature tensors shared by multiple head networks;

[0013] Multiple head networks including a torsion angle head network, a shape parameter head network, and a three-dimensional joint head network are used to obtain the predicted values of the torsion angle φ, the shape parameter β, and the three-dimensional joint coordinates P, respectively. p k represents the kth key joint in the human body parameter model, K represents the total number of key joints in the human body parameter model, and the human body parameter model provides a differentiable function M(θ,β), where θ is the relative axis angle of the key joint;

[0014] Step 3: Use Rodrigues' formula to convert the relative axis angle θ into a relative rotation matrix Among them, par(i) represents the parent joint of the i-th key joint, R par(i),i represents the i-th local rotation matrix relative to its parent joint;

[0015] The global rotation matrix of the i-th key joint is expressed as R i , then R i Recursively calculated from the relative rotation matrix:

[0016] R i =R par(i) R par(i),i

[0017] Where R par(i) Represents the parent joint rotation matrix of i key joints;

[0018] Step 4: Convert the global rotation matrix R i Decomposed into twist rotation R tw and swing rotation R sw , then:

[0019]

[0020] Where: α i represents the swing rotation angle of the i-th key joint; φ i represents the torsion angle of the i-th key joint rotation, which is predicted by the working model;

[0021] Among them, the pelvic joint and lobe joint are not included in the above torsion-pendulum decomposition, and the spine sub-joints, left hip joint and right hip joint are used for closed-form solution;

[0022] Step 5: Remove the pelvic joint and lobe joints, apply the torsion pendulum decomposition described in step 4, and define the forward dynamics process as:

[0023] Q=FK(R sw (α); R tw (φ),T,P):=f(α)

[0024] Where Q represents the 3D human joint reconstructed from the human body parameter model output, α represents the swing rotation angle, f(·) represents the forward model operator used for simplified representation, and FK(·) represents the forward kinematics process;

[0025] Step 6: Express the optimization objective function as:

[0026]

[0027] Through multiple iterations, the optimization objective function converges to obtain the local optimal value α of the swing rotation angle. * ,in:

[0028] The swing rotation angle α obtained at the kth iteration k =α k-1 +Δα k , Δα k is the k-th iteration obtained in α k-1 The Gauss-Newton direction at , and:

[0029]

[0030] Where: J k-1 Denotes the Jacobian matrix calculated from the k-1th iteration, the Jacobian matrix J k-1 With the swing rotation angle α, torsional rotation R tw , the inputs of the predicted values of the static pose T and the 3D joint coordinates P are associated and differentiable; σ is a constant term; I is the 3x3 identity matrix; ΔP k-1 =Pf(α k-1 ) represents the residual term.

[0031] Preferably, after step 1 and before step 2, the method further includes the following steps:

[0032] Combine all the cropped images obtained in step 1 into a dataset After that, randomly sample a batch of pictures to get Among them, N b is the batch size, is the i-th cropped picture, Input into the working model described in step 2.

[0033] Preferably, the backbone network processes data in five stages: the first stage is the preprocessing of the input, including a convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer; the subsequent four stages are composed of three, four, six and three BasicBlock layers respectively, wherein each BasicBlock layer has two layers of convolution, a downsampling and a residual connection operator.

[0034] Preferably, the torsion angle head network and the shape parameter head network use an adaptive average pooling layer with a kernel of 1 to process the feature tensor output by the backbone network, and then use two fully connected layers and two dropout layers to obtain the torsion angle φ and shape parameter β respectively;

[0035] The three-dimensional joint head network uses a deconvolution layer structured by three deconvolution operations, three batch normalization layers, and three ReLU activation functions. The deconvolution layer is followed by a convolution layer with a K×N convolution kernel of 1. Finally, a soft-argmax operation is performed to output the three-dimensional joint coordinates P, where N is the depth dimension of each key joint.

[0036] Preferably, the loss function L of the working model is reg The calculation is as follows:

[0037] L reg =ω1L shape +ω2L twist +ω3L joint

[0038] Among them, L shape is the 2-norm loss with shape parameter β, L twist is the 2-norm loss of the torsion angle φ, L joint is the 1-norm loss of the three-dimensional joint coordinates between the true values, and ω1, ω2, ω3 are the corresponding loss weights.

[0039] Preferably, the twist rotation R tw The analytical solution of is obtained by the Rodriguez formula, namely:

[0040]

[0041] Among them, D sw (·) is the closed-form solution of the oscillatory rotation matrix, yes The corresponding skew-symmetric matrix, I is the 3x3 unit matrix, α represents the swing rotation angle, and are the body part vector for the given rest pose and the target body part vector, respectively.

[0042] Preferably, the swing rotation R sw The body part vector The rotation around the axis itself with a torsion angle φ.

[0043] Preferably, in step 6, only a few iterations are needed to make the optimization objective function converge.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention leverages the respective advantages of optimization-based and regression-based algorithms to estimate a 3D human body parametric model. The present invention has significantly lower computational overhead than most existing optimization-based methods and provides more accurate mesh-image correspondence than existing regression-based methods.

[0046] (2) The present invention can effectively handle non-convex optimization in the optimization layer. It is worth noting that existing optimization layer solvers cannot effectively handle non-convex optimization problems. In addition, benefiting from the iterative closed-form solution via Gauss-Newton iteration, its running time and convergence time are significantly improved.

[0047] (3) The present invention achieves state-of-the-art performance in popular 3D human body parameterized model estimation benchmark algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is the overall structure diagram of the inverse kinematics optimization layer;

[0049] Figure 2 Schematic diagram of forward and backpropagation for inverse kinematics optimization layer training;

[0050] Figure 3 Graph of the regression branch structure for inverse kinematics optimization layer. DETAILED DESCRIPTION

[0051] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0052] The method for estimating a 3D human body parametric model by combining regression and optimization disclosed in this embodiment combines the advantages of optimization and regression to perform end-to-end 3D human body parametric model estimation. In the model, the regression branch predicts 3D human body joint points, and the inverse kinematics optimization layer embeds the optimization process into the neural network of the regression branch as one of its layers, namely the optimization layer, thereby using the estimated 3D human body joint points to improve the 3D human body mesh estimation. The method for estimating a 3D human body parametric model by combining regression and optimization disclosed in this embodiment specifically includes the following steps:

[0053] Step 1: Use a mature person detection algorithm on the input image to obtain bounding boxes for each person in the image. Then, based on the bounding boxes, crop the corresponding person according to the bounding box position, so that the cropped image contains only one person. The cropped image is then preprocessed and the training optimizer and training schedule are configured. In this example, we set the initial learning rate to 0.001 and reduce it by a factor of 10 at the 90th and 120th training rounds, respectively. We also use the Adam solver as an example and train for a total of 200 rounds.

[0054] In this embodiment, the series of pre-processing performed on the cropped image specifically includes the following steps:

[0055] Step 101: Scale the image to 256×256 size according to the original image ratio by adding black edges.

[0056] Step 102: Randomly flip the image horizontally with a probability of 0.5.

[0057] Step 103: Rotate the image randomly at various angles with a probability of 0.5.

[0058] Step 104: Randomly erase the image with a probability of 0.5 to simulate occlusion.

[0059] Step 105: Randomly crop the image.

[0060] Step 106: Normalize the images, where the means are set to 0.485, 0.456, and 0.406, and the variances are set to 0.229, 0.224, and 0.225.

[0061] Step 2: At the beginning of the initial task t = 0, where t is the current task number, all the cropped images obtained in step 1 are merged into the dataset D train , randomly sample a batch of pictures to get N b is the batch size, x i is the i-th picture and is input into the working model based on convolutional neural network (CNN).

[0062] Step 3: The working model includes a backbone network and multiple head networks to regress different predicted values of three-dimensional joints P, torsion angle Φ and shape parameters β. P represents the three-dimensional joints of the human body. Where K = 23 represents the 23 key nodes used in the SMPL model. In this invention, we use the SMPL model as the human body parameter model, which provides a differentiable function M(θ, β), where the shape parameters are given by 10 principal component parameters are used for representation. Here we use the head network for regression. The relative axis angle of the joint is given by Parameter representation.

[0063] The backbone network extracts features to obtain a shared feature map. Here, we use ResNet34 as the basis and make some replacements. The 3D joint head network uses 3 deconvolution layers followed by 1×1 convolution to further extract features from the feature map. Softmax is then used to obtain the predicted scores of the 23 body joint positions. To generate a three-dimensional heat map, q(·) is the softmax function used by the working model. The torsion angle Φ and shape parameter β use a fully connected layer to directly regress their respective parameters on the shared feature map. The activation function used here is the ReLU function. The regression branch structure of the inverse kinematics optimization layer is shown in the figure. Figure 3 In summary, the working model consists of five stages in the backbone network and three head networks:

[0064] The first stage is the preprocessing of the input, including a convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer. The convolution kernel size is 7×7, the number is 64, and the stride is 2. The kernel size of the maximum pooling layer is 3×3 and the stride is 2.

[0065] The subsequent four stages consist of three, four, six, and three BasicBlock layers, respectively. Each BasicBlock layer contains two convolution layers (including batch normalization and ReLU functions, with a kernel size of 3×3), an optional downsampling layer, and a residual connection operator.

[0066] Therefore, assuming that the batch size we feed into the backbone network is 64, the feature dimensions of the output after the final stage of the backbone network are completed will be 64×512×8×8.

[0067] The three head networks share a backbone network outputting a feature tensor with a dimension of 64×512×8×8. These include the torsion angle head network, the shape parameter head network, and the 3D joint head network. The torsion angle and shape parameter head networks use an adaptive average pooling layer with a kernel of 1 to process the feature tensor, converting it into a 64×512 feature vector. Subsequently, two fully connected layers and two dropout layers are used, respectively, to reduce the network's output dimension to 64×1024. Using two fully connected layers, the torsion angle output and shape parameter output are 64×46 and 64×46, respectively.

[0068] Unlike the torsion angle and shape parameter head networks, the 3D joint head network uses a deconvolutional layer structure consisting of three deconvolution operations, three batch normalization layers, and three ReLU activation functions. The deconvolutional layer outputs a dimension of 64×256×64×64. This is followed by a convolutional layer with 1536 convolution kernels and a kernel size of 1, which resizes the output to 64×1536×64×64. The second dimension of 1536 is to accommodate the 24 joints and the depth dimension of each joint of 64. Finally, a soft-argmax operation is performed to output the coordinates of the 3D key nodes.

[0069] So far, we have constructed the regression branch of our entire neural network structure, which outputs the 3D joint P, the torsion angle Φ, and the shape parameter β. The shape parameter β is used to obtain the static pose T through the SMPL model. Then, by combining P, T, and Φ, we execute the designed inverse dynamics optimization layer to solve the relative rotation R of the 3D pose, that is, the pose parameter θ. The forward and back propagation process of the entire training is as follows: Figure 2 shown.

[0070] In the regression branch, we use a deep neural network to predict the 3D human joint P, the torsion angle Φ and the shape parameter β. The 3D human joint P is obtained from the 3D heat map by the differentiable soft-argmax operation to estimate the 3D joint coordinates P:

[0071]

[0072] in, represents the normalized 3D heatmap, whose depth, height, and width resolutions are denoted as D, H, and W respectively. We then use l1 loss to supervise the predicted 3D joint coordinates as shown below:

[0073]

[0074] in, Represents the supervision value of the three-dimensional joint, K represents the number of three-dimensional joints, and here we use 23 key nodes according to the SMPL model.

[0075] Since the twist angle has a highly nonlinear characteristic, it is difficult to directly regress the twist angle of the limb from the network. Here we choose to convert the twist angle into its corresponding cosine and sine values, namely cosφ k and cosφ k , where φ k Represents the torsion angle of the kth joint, so the regression target becomes a two-dimensional vector Instead of the scalar value φ k , thus avoiding the discontinuity problem. We then use l2 loss to supervise the predicted two-dimensional vector:

[0076]

[0077] in, Represents the true value of the torsion angle of the regressed limb.

[0078] The SMPL model obtains a rest pose skeleton with an additional offset according to the shape parameter β:

[0079]

[0080] Where T represents the static pose skeleton, is the mesh vertex in the average rest pose, B S (·) is the blend shape function provided by SMPL, and W represents the regression function from mesh vertices to skeleton coordinates. Therefore, during the training phase, we only need to supervise the shape parameter β:

[0081]

[0082] in, represents the true value of the shape parameter β.

[0083] In summary, in the regression branch, we use a deep neural network to calculate the total loss function of the three-dimensional joints, shapes, and distortion angles and the true values as follows:

[0084] L reg =ω1L shape +ω2L twist +ω3L joint

[0085] Among them, L shape is the 2-norm loss with shape parameter β, L twist is the 2-norm loss of the torsion angle φ, L joint is the 1-norm loss of the regression joint between the true values, and ω1, ω2, ω3 are the corresponding loss weights.

[0086] Step 4: After obtaining the regressed three-dimensional joint P, torsion angle Φ and shape parameter β, IKOL can be used to obtain the corresponding solution angle, where the angle is given by θ∈R 3×23 Parameter representation is usually converted into a relative rotation matrix using the Rodrigues formula Among them, par(i) represents the parent joint of the i-th body joint, R par(i),i ∈SO(3) represents the i-th local rotation matrix relative to its parent joint, and SO(3) represents the three-dimensional rotation group. We denote the i-th global rotation matrix as R i ∈SO(3). In this way, the global rotation matrix of the i-th body joint can be recursively calculated from the relative rotation matrix as:

[0087] R i =R par(i) R par(i),i

[0088] Where R par(i) Represents the parent joint rotation matrix of i key joints.

[0089] It is worth noting that estimating the rotation from 3D human joints P is an inherently underdetermined problem due to the lack of torsion angles in the skeleton representation. Therefore, we follow existing work [hybrik] to apply twist and swing decomposition, decomposing the rotation R into the twist rotation R tw and swing rotation R sw This is because when solving inverse kinematics, some body joints are usually assigned lower degrees of freedom (DoF), such as 1 or 2 DoF, to simplify the problem. In this implementation, we consider the general case, assuming that each body joint has a full 3 DoF, so the rotation R is decomposed into the twist rotation R tw and swing rotation R sw . Given a resting pose, the body part vector and the target body part vector The solution process of rotation R can be expressed as:

[0090]

[0091] Where φ is the torsion angle estimated by the neural network, D sw (·) is the closed-form solution of the oscillatory rotation matrix, D tw (·) Convert φ into a twist rotation matrix. Here, the rotation matrix R should satisfy the equality condition

[0092] Assume that the swing rotation angle is about the axis For rotation, we know the direction axis of the swing rotation angle Perpendicular to and Therefore, the direction axis and and The mathematical relationship between them can be expressed as:

[0093]

[0094] Therefore, the swing rotation R sw The analytical solution can be solved by the Rodrigues formula, that is:

[0095]

[0096] in, yes The corresponding skew-symmetric matrix, I is the 3x3 identity matrix, and α represents the swing rotation angle.

[0097] Similar to the swing angle, the twist angle corresponds to the rotation around Rotating. Therefore, The axis itself is the axis, with φ as the angle, and the twist rotation R tw The same can be obtained.

[0098] In summary, the decomposed rotation matrix is expressed as follows:

[0099]

[0100] in, represents the swing rotation angle of the i-th rotation, and The twist rotation angle is the i-th rotation. The swing rotation angle is calculated by the optimization branch we designed. The twist rotation angle is regressed by the head network predicted by the regression branch.

[0101] The pelvic joint and the leaf joints are not included in the above torsion-pendulum decomposition. Because the pelvic joint has three sub-joints, a closed-form solution is used using the spine sub-joints, the left hip joint, and the right hip joint. The solution here is provided by Hybrik, specifically to find a rigid rotation that aligns the two sets of joints (the spine sub-joints, the left hip joint, and the right hip joint) in the best way. Here, we assume that the root joint of the predicted pose is aligned with the rest pose. Therefore, the problem is formulated as:

[0102]

[0103] Among them, ||·|| Frepresents the Frobenius norm, P0 = [p0, p1, p2], T0 = [t0, t1, t2], where p0, p1, p2 are the network-estimated spine sub-joint, left hip joint, and right hip joint coordinates, respectively. t0, t1, t2 are the network-estimated spine sub-joint, left hip joint, and right hip joint coordinates of the resting pose skeleton with an additional offset based on the network-estimated shape parameter β. R0 represents the global rotation matrix of the body. The above equation can be easily solved in closed form as follows:

[0104] R0=VU T

[0105] Among them, V and U represent the The left and right singular matrices obtained by singular value decomposition are:

[0106]

[0107] Step 5: Remove the pelvic joint and the lobe joints, apply the torsion-pendulum decomposition, and the forward kinematics (FK) process is defined as follows:

[0108] Q=FK(R sw (α); R tw (φ),T,P):=f(α)

[0109] in, represents the 3D human joints reconstructed from the SMPL output, P represents the regressed 3D human joints estimated from the regression branch, represents the swing angle, f(·) represents the forward model operator used to simplify the representation, and FK(·) represents the forward kinematics process. In this case, we use the forward kinematics process provided by the SMPL model.

[0110] After torsion pendulum decomposition, the optimization objective function in IKOL is expressed as follows:

[0111]

[0112] Step 6: Taylor expand the nonlinear forward model operator f(·). k-1 is expressed as the swing angle of the k-1th iteration. To do this, the Taylor series is used in α k-1 Expand to approximate the equation, that is:

[0113] f(α)≈f(α k-1 )+J k-1 (α-α k-1 )

[0114] Among them, J k-1 represents the Jacobian matrix calculated from the k-1th iteration. Here, the Jacobian matrix J k-1With the swing angle α, torsional rotation R tw , the inputs of the rest pose T and the regressed joint P are correlated and differentiable.

[0115] After the above linearization operation, the objective function is transformed into the following linear function:

[0116]

[0117] Where ΔP k-1 =Pf(α k-1 ) represents the residual term, Δα k =α-α k-1 Indicates that in α k-1 The Gauss-Newton direction at , Δα represents the optimization objective function variable.

[0118] Step 7: Given an initial guess α0, the pendulum angle is iteratively updated as follows:

[0119] α k =α k-1 +Δα k

[0120] It is worth noting that α k-1 and Δα k is analytically differentiable.

[0121] Since the objective function is transformed into a linear problem, in each iteration, we can calculate the Gauss-Newton direction by the following formula:

[0122]

[0123] in, σ is a constant term used to avoid ill-posed solutions.

[0124] In each iteration, for example, the kth iteration, the swing angle iteration is based on α k =α k-1 +Δα k When the number of iterations is large enough, that is, when k→∞, we can get the local optimal value α * , the derivation is as follows:

[0125]

[0126] Since the Gauss-Newton algorithm is a second-order method, we found in practice that it only takes a few iterations to converge. Therefore, we set the number of iterations to 5 in the entire optimization layer based on empirical evidence. Here, in our experiments, we believe that the entire optimization process has converged when the iteration reaches the 5th time. To better understand how the Gauss-Newton algorithm is differentiable here, we check its gradient with respect to one of the input parameters P, that is,

[0127]

[0128] Due to Δα k It is derived from the Gauss-Newton direction equation, and its gradient can be calculated as:

[0129]

[0130] in,

[0131] After 5 fixed iterations, the inverse dynamics optimization layer can output the joint input rotation R corresponding to the parameterized model SMPL. sw The loss function used for the optimization result is designed to calculate the deviation between the rotation and the true value, as shown below:

[0132] in, Represents the true value of the rotation. The predicted value α used in the above formula * The results are output by the inverse dynamics optimization layer algorithm we designed, and the gradients are propagated back through the Gauss-Newton differentiation method.

[0133] In summary, the inverse dynamics optimization layer uses the regression loss L reg and optimize the loss L opt The weighted sum of supervision, the total loss L total The derivation is:

[0134] L total =L reg +ω4L opt

[0135] Here, ω4 represents the corresponding optimization loss weight. It is worth noting that the optimization objective function used by the inverse dynamics optimization layer is highly non-convex, making it difficult to train with the regression branch. In this paper, through our designed Gauss-Newton differentiation method, we can use the regression branch to train this non-convex optimization function in an end-to-end framework.

[0136] The present invention discloses a method for end-to-end three-dimensional human body parametric model estimation that combines the respective advantages of optimization and regression. By using three-dimensional human joints as a bridge, the regression algorithm and the optimization algorithm commonly used in the task of three-dimensional human body parametric model estimation are combined. In order to give full play to their respective strengths, the present invention uses the regression algorithm to perform neural network regression on the key points of the three-dimensional human body joints and the optimization algorithm to perform inverse dynamics solution on the predicted three-dimensional human body joint key points. The optimization branch implements step-by-step iterative solution in the neural network, and serves as one of the layers, namely the optimization layer. Through the designed optimization algorithm, the inverse kinematics optimization layer we designed can be trained together with the neural network and the optimization results can be fed back to the parameters of the training neural network through gradients. Thereby achieving the effect of end-to-end three-dimensional human body parametric model estimation.

Claims

1. A three-dimensional human body parameter model estimation method combining regression and optimization, characterized in that: The following steps are involved: Step 1: Crop a given image so that the cropped image contains only one person; Step 2: Input the data obtained in the previous step into the working model, which includes a backbone network and multiple head networks, where: The backbone network is used to extract features from the data input into the working model to obtain feature tensors shared by multiple head networks; Multiple head networks including a torsion angle head network, a shape parameter head network, and a three-dimensional joint head network are used to obtain the predicted values of the torsion angle φ, the shape parameter β, and the three-dimensional joint coordinates P, respectively. p k represents the kth key joint in the human body parameter model, K represents the total number of key joints in the human body parameter model, and the human body parameter model provides a differentiable function M(θ,β), where θ is the relative axis angle of the key joint; Step 3: Use Rodrigues' formula to convert the relative axis angle θ into a relative rotation matrix Among them, par(i) represents the parent joint of the i-th key joint, R par(i),i Represents the i-th local rotation matrix relative to its parent joint; The global rotation matrix of the i-th key joint is expressed as R i , then R i Recursively calculated from the relative rotation matrix: R i =R par(i) R par(i),i Where R par(i) Represents the parent joint rotation matrix of i key joints; Step 4: Convert the global rotation matrix R i Decomposed into twist rotation R tw and swing rotation R sw , then: Where: α i represents the swing rotation angle of the i-th key joint; φ i represents the torsion angle of the i-th key joint rotation, which is predicted by the working model; Among them, the pelvic joint and lobe joint are not included in the above torsion-pendulum decomposition, and the spine sub-joints, left hip joint and right hip joint are used for closed-form solution; Step 5: Remove the pelvic joint and lobe joints, apply the torsion pendulum decomposition described in step 4, and define the forward dynamics process as: Q=FK(R sw (a);R tw (φ),T,P):=f(α) Where Q represents the 3D human joint reconstructed from the human body parameter model output, α represents the swing rotation angle, f(·) represents the forward model operator used for simplified representation, and FK(·) represents the forward kinematics process; Step 6: Express the optimization objective function as: Through multiple iterations, the optimization objective function converges to obtain the local optimal value α of the swing rotation angle. * ,in: The swing rotation angle α obtained at the kth iteration k =α k-1 +Δα k , Δα k is the k-th iteration obtained in α k-1 The Gauss-Newton direction at , and: Where: J k-1 Denotes the Jacobian matrix calculated from the k-1th iteration, the Jacobian matrix J k-1 With the swing rotation angle α, torsional rotation R tw , the inputs of the predicted values of the static pose T and the 3D joint coordinates P are associated and differentiable; σ is a constant term; I is a 3x3 identity matrix; ΔP k-1 =Pf(α k-1 ) represents the residual term.

2. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: After step 1 and before step 2, the method further includes the following steps: Combine all the cropped images obtained in step 1 into a dataset After that, randomly sample a batch of pictures to get Among them, N b is the batch size, is the i-th cropped picture, Input into the working model described in step 2.

3. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: The backbone network processes data in five stages: the first stage is the preprocessing of the input, including a convolution layer, a batch normalization layer, a ReLU activation function, and a maximum pooling layer; the subsequent four stages are composed of three, four, six, and three BasicBlock layers, respectively, where each BasicBlock layer has two convolution layers, a downsampling layer, and a residual connection operator.

4. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: The torsion angle head network and the shape parameter head network use an adaptive average pooling layer with a kernel of 1 to process the feature tensor output by the backbone network, and then use two fully connected layers and two dropout layers to obtain the torsion angle φ and shape parameter β respectively; The three-dimensional joint head network uses a deconvolution layer structured by three deconvolution operations, three batch normalization layers, and three ReLU activation functions. The deconvolution layer is followed by a convolution layer with a K×N convolution kernel of 1. Finally, a soft-argmax operation is performed to output the three-dimensional joint coordinates P, where N is the depth dimension of each key joint.

5. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: The loss function L of the working model reg The calculation is as follows: L reg =ω1L shape +ω2L twist +ω3L joint Among them, L shape is the 2-norm loss with shape parameter β, L twist is the 2-norm loss of the torsion angle φ, L joint is the 1-norm loss of the three-dimensional joint coordinates between the true values, and ω1, ω2, ω3 are the corresponding loss weights.

6. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: The twist rotation R tw The analytical solution of is obtained by the Rodriguez formula, namely: Among them, D sw (·) is the closed-form solution of the oscillatory rotation matrix, yes The corresponding skew-symmetric matrix, I is the 3x3 unit matrix, α represents the swing rotation angle, and are the body part vector for the given rest pose and the target body part vector, respectively.

7. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: The swing rotation R sw The body part vector The rotation around the axis itself with a torsion angle φ.

8. The method for estimating a three-dimensional human body parameter model by combining regression and optimization according to claim 1, wherein: In step 6, only a few iterations are needed to make the optimization objective function converge.

Images