A parameterized human body model fitting method based on human body point cloud and measurement parameters

Abstract

This invention discloses a parametric human body model fitting method based on human point clouds and measurement parameters, comprising the following steps: rendering the human point cloud at fixed distances and different viewpoints to obtain full-body human images from multiple viewpoints; predicting 2D joints of the human body in the full-body human images using a deep neural network; performing triangulation on the 2D joints to predict 3D joints; and matching the parametric human body model SMPL or SMPL-X onto the human point cloud based on the human point cloud, 3D joints, and input measurement parameters. The parametric human body model generated by this invention is relatively aligned with the human point cloud in pose and meets the constraints of the human point cloud and the requirements of the measurement parameters in body shape. Furthermore, this invention also supports directly generating accurate body shape parametric human body models by inputting only measurement parameters. The obtained parametric human body model can be used for dataset construction, virtual try-on, and other applications.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a parametric human body model fitting method based on human point clouds and measurement parameters. Background Technology

[0002] In recent years, sensing technologies for capturing unstructured 3D point clouds have become ubiquitous and more accurate, driving the development of 3D human reconstruction technology. Among them, fitting a parametric human model to a point cloud of a human wearing a bodysuit is a challenging task in computer vision, as it requires locating the positions of various joints in the human point cloud and aligning the mesh vertices as closely as possible to the human point cloud. In learning-based methods, fitting of the SMPL(-X) model can be divided into two categories: (1) using deep neural networks to predict the pose and body shape parameters of SMPL(-X); (2) searching for the parameters of SMPL(-X) through an iterative optimization method of gradient descent, optimizing an objective function to match the human point cloud as closely as possible. These two methods each have their advantages and disadvantages. The former requires a dataset with 3D annotations for training, but its nonlinear representation capability is stronger; the latter, although it does not require an additional dataset, requires finding suitable optimization terms and various prior knowledge constraints.

[0003] Existing technique 1, "HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling," constructs a multi-view camera array to collect point cloud data of a person wearing tight clothing from different perspectives and then fuses them to obtain a high-precision human point cloud. To register it with the SMPL mesh model, it uses 2D joint prediction from multiple perspectives, obtains 3D joints through triangulation, and then optimizes the pose and body shape parameters of the SMPL by minimizing the objective function. However, the provided dataset lacks scan data of people wearing tight clothing, making it difficult to evaluate the accuracy of the registered SMPL model in terms of body shape. Furthermore, this method requires high-quality scan data; otherwise, the reconstructed SMPL model will still have some discrepancies with the human point cloud.

[0004] Existing technique 2, "3D Human Body Reshaping with Anthropometric Modeling," proposes a method for generating parametric human body models of specific body types using partial anthropometric parameters. During the training phase, a correlation mask is learned using a feature-selective local mapping technique based on a human body mesh model and measurement parameters from the dataset. The vertex mapping matrix is ​​further learned from the parameters selected by the correlation mask through linear regression. During the inference phase, the user inputs a limited number of measurement parameters, and missing values ​​are handled through multiple interpolation. Finally, the human body model is generated using the correlation mask and mapping matrix during the mapping phase. However, this method cannot be applied to SMPL(-X) human body models, and the error between the reconstructed 3D human body measurement parameters and the input parameters is relatively high.

[0005] Existing technology 3, "Accurate 3D Body Shape Regression using Metric and Semantic Attributes," focuses on predicting more accurate SMPL body shape parameters from a single image. First, it uses an existing regressor to predict the SMPL's pose and body shape parameters. Then, it uses a simple neural network to input the body shape parameters to predict semantic body shape features. The predicted SMPL body shape is then used to obtain partial anthropometric parameters through virtual measurement technology. Finally, the semantic body shape and measurement parameters are combined to constrain the SMPL's body shape parameters. However, the semantic body shape features it uses are based on the subjective evaluation of multiple annotators, making it difficult to accurately and objectively represent human body shape characteristics. Although it effectively improves the accuracy of body shape prediction, its error remains relatively large.

[0006] Most existing fitting methods for the SMPL(-X) model struggle to reconstruct accurate body shapes, or require additional vertex offsets. Deep neural network-based methods, which require pre-built 3D datasets for training, still yield poor body shape reconstruction results. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned deficiencies in the prior art by providing a parametric human body model fitting method based on human point cloud and measurement parameters. This method can achieve precise control of the SMPL(-X) body shape by introducing constraints on measurement parameters. It can not only provide strong constraints on the body shape in the fitting process, but also directly generate the SMPL(-X) model of the target body shape through measurement parameters.

[0008] The objective of this invention can be achieved by adopting the following technical solutions:

[0009] A parametric human body model fitting method based on human body point cloud and measurement parameters, the fitting method comprising the following steps:

[0010] S1. Render the human body point cloud at a fixed distance and from multiple different perspectives to obtain a full-body human image from each different perspective.

[0011] S2. For the full-body human image from each viewpoint, predict the 2D key point set and the confidence level corresponding to each key point from each viewpoint using the deep neural network OpenPose.

[0012] S3. For the 2D joint set under all viewpoints, the 2D joint set is obtained by filtering the viewpoints and excluding low-confidence 2D joints. Then, the 3D joints are reconstructed using the triangulation algorithm. The 3D joints with reprojection loss exceeding the first threshold are optimized by excluding noisy viewpoints.

[0013] S4. Given the initial pose parameters θ0 and body shape parameters β0 of the parametric human model SMPL(-X), hereinafter referred to as the SMPL(-X) model, when the zero-vector pose parameters are provided, the generated human model presents a T-pose with the palms extended. Feature points required for measurement are predefined on the SMPL(-X) model. The required measurement parameters are obtained by measuring the SMPL(-X) model under the T-pose generated with specific body shape parameters. Then, the SMPL(-X) model is fitted. In the fitting process, the SMPL(-X) model is first translated to the mid-hip point of the reconstructed 3D joint to complete the initial alignment. Then, the reconstructed 3D joint, human point cloud and input measurement parameters are used as constraints to minimize the objective function through iterative optimization to obtain the optimized pose parameters θ and body shape parameters β of the SMPL(-X) model.

[0014] Furthermore, in step S1, the entire human point cloud is translated along its center of gravity to a position with the origin as the center of gravity, ensuring that the rendered human body is located in the center of the image (coordinate unit: mm). Then, the camera distance is set to 1200mm, and an aspect ratio of a = 1 and a vertical field of view are used. Near plane z n =10mm, far plane z f =10000mm perspective projection, while ensuring the whole human body is in the image, covers more pixel area. The camera is facing the front of the human body to start rendering. Then, the human body point cloud is rotated 30 degrees counterclockwise around the y-axis each time, for a total of 11 rotations. Finally, 12 full-body human body images with a resolution of 512x512 are rendered from different perspectives.

[0015] Furthermore, in step S2, 12 full-body images are used as input to the deep neural network OpenPose to predict the set of 2D human joints for each full-body image and the confidence level of each 2D joint. Let K be the number of 2D human joints predicted by OpenPose. OpenPose can predict two sets of joints for body parts: Body25 (25 joints) or COCO (18 joints). It also supports the prediction of 42 hand joints or 70 face joints. Based on the clarity of the full-body image, it is determined whether to add hand joints or face joints after selecting the set of joints for the selected body parts.

[0016] Furthermore, step S3 is as follows:

[0017] S31. Select N sets of 2D joint points from 12 full-body human images. Due to self-occlusion of the human body in some views, the confidence of the joint points predicted by OpenPose is lower than that of other views. Views with severe self-occlusion of the human body are excluded by filtering. Then, since the low confidence of 2D joint points will affect the accuracy of the reconstructed 3D joint points, all 2D joint points with confidence below the second threshold are excluded from the 2D joint point set of these N views, resulting in the 2D joint point selection set.

[0018] S32. From the 2D joint selection set, reconstruct K 3D joints using triangulation: Let... Let V be the set of 2D joints consisting of the remaining viewpoints of the current joint, where N' represents the number of remaining viewpoints, i represents the i-th viewpoint among the remaining viewpoints, and V i Indicates the corresponding viewpoint number. This indicates that the joint is in viewpoint number V i 2D joints below and They represent The x and y coordinates are set for each viewpoint V. i 3×3 camera rotation matrix Translation vector Where vector and Representing the rotation matrix Columns 1, 2, and 3, and They represent The x, y, and z coordinates are set, and the same 4×4 perspective projection matrix is ​​set for all 12 viewpoints. in 'a' represents the aspect ratio. Indicates the vertical field of view. z n z represents the near-plane position distance. f The distance to the far-plane position is expressed by the following formula for the viewing angle V. i The 2D key points are subjected to inverse image space transformation and inverse projection transformation to obtain the 3D direction vector of the camera viewpoint:

[0019]

[0020] in Representing perspective V i The direction vector below, and They represent The x and y coordinates, W and H represent the width and height of the full-body image, respectively;

[0021] For each perspective V i List the following two equations:

[0022]

[0023]

[0024] Where x' represents the 3D joint position to be determined; N' remaining viewpoints list 2N' equations, each viewpoint V i Starting from the corresponding camera position, generate a path along the corresponding 2D joint point position based on the respective viewpoint. A ray is drawn in the direction of the ray, generating a total of N' rays. The least squares method is used to find the 3D keypoint x' with the smallest distance to these N' rays. Then, x' is projected onto the image space of the remaining viewpoints to obtain the reprojected 2D keypoint set. Calculate the reprojected 2D joints using the following formula:

[0025]

[0026] in, It is a column vector located in the projection space. and express The x, y, z, and w coordinates, for Perform perspective division, that is, divide the whole by This allows its x, y, and z components to represent 3D coordinates in the projection space. The viewpoint is numbered V. i Reprojection of 2D joints;

[0027] Finally, the average pixel error is calculated using the following formula:

[0028] Where e is the average pixel error, and ||·||2 is used to calculate the Euclidean distance of the vector. The smaller the average pixel error, the more accurate the calculated 3D joints. This depends on the quality of the full-body image and the confidence level of the 2D joints predicted by OpenPose.

[0029] S33. If the average pixel error e is less than the first threshold, or the number of remaining viewpoints N' is less than 4, the final predicted 3D joint point x' is obtained; otherwise, the following method is used to eliminate noisy viewpoints until the condition is met: For the 2D joint point set composed of the remaining viewpoints of the current joint... Let Q i for The point set obtained by excluding the 2D joints corresponding to the i-th viewpoint from the remaining viewpoints, for Q i 3D joints x were reconstructed using triangulation. i Then, reprojection is performed and the average pixel error e is calculated. i Let m be the set of the average pixel errors to be calculated. The minimum value e in m The corresponding subscript, if e m Less than the first threshold, or the point set Q m If the number of remaining viewpoints is less than 4, the final predicted 3D keypoint x is obtained. m ', otherwise Q m As input to the noise-removing viewpoint method and continued to be executed, the noise-removing viewpoint can effectively reduce the average pixel error obtained by reprojection calculation of the reconstructed 3D joints, thereby helping to align the 3D joints of the subsequent SMPL(-X) model with the reconstructed 3D joints, as well as the alignment between the point cloud and the SMPL(-X) model.

[0030] Furthermore, in step S4, the measurement method obtains a total of 8 measurement parameters, including height, perineal height, arm span, upper limb length, chest circumference, waist circumference, hip circumference, and thigh circumference. Then, it is expanded to support more measurement parameters, including hip width, waist width, thigh length, upper arm length, and shoulder width. Since the measurement process is performed on the SMPL(-X) model, some measurement parameters will inevitably have certain differences from the actual measurement parameters.

[0031] Furthermore, the measurement method in step S4 requires pre-defining feature points from the vertices of the SMPL(-X) model, including: head vertex, left heel point, left middle finger tip point, right middle finger tip point, left shoulder point, perineum point, right chest point, navel point, perineum point, and left thigh point. If it is necessary to further expand the measurement categories, more feature points for measurement can be defined. Then, length-related and circumference-related measurements are performed under a unified T-pose. Since the palm of SMPL is in a straight and open state under the default T-pose, while the palm of SMPL-X is in a bent state under the default T-pose, the palm pose of SMPL-X needs to be changed to the same straight and open state as SMPL.

[0032] Among them, length-related measurements include height, arm span, upper limb length, and perineal height. Height is obtained by calculating the vertical distance from the top of the head to the left heel on the y-axis. Arm span is obtained by calculating the horizontal distance from the tip of the left middle finger to the tip of the right middle finger on the x-axis. Upper limb length is obtained by calculating the distance from the left shoulder to the tip of the left middle finger. Perineal height is obtained by calculating the vertical distance from the perineum to the left heel on the y-axis.

[0033] The measurements related to circumference include chest circumference, waist circumference, hip circumference, and left thigh circumference. First, a feature point is selected for lateral cutting. Using the normal vector n = (0,1,0), a planar mesh capable of completely cutting the SMPL(-X) model is determined through point normals. Then, the SMPL(-X) model and the planar mesh are intersected to obtain the set of intersecting vertices. The convex hull vertex set is obtained by removing duplicate vertices and using a convex hull algorithm. Finally, the circumference of the convex hull vertex set is calculated. The intersection detection and convex hull vertex set calculations do not participate in gradient propagation. However, due to the large computational load of intersection detection, the CPU computation speed is slow. Therefore, a space-based accelerated structure, the BVH tree, can be constructed for the SMPL(-X) model, and the intersection detection calculation can be performed on the GPU, significantly improving the computation speed. However, different output order results in different convex hull vertex sets and multiple measurement results. To reduce this impact, multiple intersection detections and convex hull calculations are performed, and the maximum value of the measurement results is taken as the final circumference.

[0034] Before measuring the bust, waist, hip, and left thigh circumference, horizontal segmentation is required using the right chest point, navel point, perineum point, and left thigh point, respectively. Furthermore, when measuring the left thigh circumference, an SMPL(-X) model containing only the left half of the body needs to be defined and its intersection with the planar mesh performed to avoid self-intersection of the vertices of the legs affecting the intersection detection results. Figure 2 As shown.

[0035] Furthermore, the minimization objective function is defined as follows:

[0036] E = λCD E CD +λ p E p +λ a E a +λ m E m

[0037] Where E is the objective function, E CD E represents the chamfer distance loss term. p For the distance loss term of 3D joints, E a E represents the Euler angle prior loss term for the whole skeleton. m For the measurement parameter constraint loss term, λ CD , λ p , λ a and λ m This indicates the weights corresponding to each loss term;

[0038] Chamfer distance loss term E CD The following formula is used to calculate the average distance from the downsampled point cloud to the nearest point in the SMPL(-X) model, and the sum of the average distances from the SMPL(-X) model points to the nearest point in the downsampled point cloud, to evaluate the similarity between the SMPL(-X) model and the point cloud:

[0039]

[0040] in, This represents all vertices of the SMPL(-X) model. express The vertex in This represents all the vertices in the human body point cloud. This indicates that the human body point cloud retains pre-set parameters through downsampling of the farthest point. Compared to uniform sampling, downsampling using the furthest point better reflects the overall contour of the point cloud. Since the input human point cloud may have an excessively large number of points, it can directly affect the overall calculation speed. Downsampling can reduce the computational load of this loss term. express The vertex in This represents the number of vertices in the SMPL(-X) model, and min represents finding the minimum distance between a single vertex and all points in the set.

[0041] 3D joint distance loss term E p The distance error between the 3D joints of SMPL(-X) and the 3D joints obtained by triangulation is calculated using the following formula:

[0042]

[0043] Where M(θ,β) represents the generation of the SMPL(-X) model using the pose parameter θ and body shape parameter β, J k (·) represents the mapping function from the SMPL(-X) model to the k-th 3D joint, ω represents the 3×1 global camera rotation Euler angle vector, Rot(·) represents the mapping function from the 3×1 rotation Euler angle to the 3×3 rotation matrix, t represents the 3×1 global camera translation vector, and x′ ( k) represents the k-th 3D joint point obtained from triangulation, w k To control the weight of the k-th joint and ensure the sum of all joint weights is 1, there is a certain deviation between the joint positions selected by SMPL(-X) and those predicted and triangulated by OpenPose (e.g., left and right hip points). There are also cases where inaccurate 2D joint positions predicted by OpenPose lead to inaccurate 3D joint positions obtained through triangulation (e.g., heel and toe points). The weights of these significantly affected 3D joints can be set to 0. Furthermore, arm alignment is often difficult; therefore, in the later stages of iterative optimization, the weights of arm-related 3D joints can be increased, some 3D joints can be set to 0, and the weight λ can be adjusted. p ;

[0044] E of the Euler angle prior loss term for the whole skeleton a Euler angles used to constrain bones that exceed their range of motion are calculated using the following formula:

[0045] E a =avg(exp(ReLU(θ-θ)) u )+ReLU(θ l -θ))-L)

[0046] Where, θ u and θ l represents the upper and lower bounds of the Euler angles of the pose parameter θ, respectively; ReLU(·) is used to find the maximum value of each component of the matrix with respect to 0; exp(·) represents the natural exponential function of each component of the matrix; L represents a matrix of all 1s with the same dimension as θ; and avg(·) represents the average value of all components of the matrix.

[0047] Measurement parameter constraint loss term E mThe body shape parameters used to further constrain SMPL(-X) are to make the measurement parameters obtained by the SMPL(-X) model as close as possible to the input measurement parameters. This loss term is obtained by calculating the square of the difference between each input measurement parameter and the corresponding measurement parameter obtained on the SMPL(-X) model and summing them. The input measurement parameters are any combination of eight items: height, arm span, perineal height, upper limb length, chest circumference, waist circumference, hip circumference and thigh circumference. If only this loss term is used in the fitting process, it is used to generate the corresponding body shape SMPL(-X) model from the measurement parameters. When aligning to the human point cloud or model, if the input measurement values ​​are known, the body shape of SMPL(-X) is further optimized while aligning using a large weight.

[0048] Since changes in arm span will simultaneously lead to changes in upper limb length, if the arm span and upper limb length input at the same time are not reasonable, it will cause the measurement parameter constraint loss term to be difficult to converge. It is recommended to choose either arm span or upper limb length as input. Similarly, changes in hip circumference will simultaneously lead to changes in thigh circumference. It is recommended to choose either hip circumference or thigh circumference as input.

[0049] First, initial values ​​need to be provided for the pose parameter θ, body shape parameter β, global camera rotation Euler angle vector ω, and global camera translation vector t. Set θ = θ0, β = β0, ω = (0,π,0), and t is the difference vector between the hip midpoint of the SMPL(-X) model and the hip midpoint obtained by triangulation reconstruction.

[0050] To minimize the objective function E, it is necessary to optimize the global camera rotation Euler angle vector ω, the global camera translation vector t, the pose parameter θ and body parameter β of the SMPL(-X) model. Gradient descent is performed using the L-BFGS optimizer, with the optimizer employing a strong wolfe linear search method and a learning rate of 0.2, and weights λ. CD , λ p , λ a and λ m The recommended values ​​are 5, 100, 30000 and 10000000, and the number of iterations can be 60-100. The iteration can be terminated early when the loss no longer decreases.

[0051] The present invention has the following advantages and effects compared with the prior art:

[0052] 1. This invention, through further expansion of the virtual measurement module, enables the measurement of height, leg length, body measurements (bust, waist, hip, hip), thigh circumference, arm span, and upper limb length of the SMPL(-X). Because this module is differentiable, it can also support the generation of an SMPL(-X) model of the target body shape by providing measurement parameters. The generated model, after measurement, closely approximates the input values, resulting in more reasonable body shape parameters and lower error compared to existing methods for generating parameterized human body models using measurement parameters.

[0053] 2. This invention, while fitting the SMPL(-X) model, can additionally introduce constraints based on measurement parameters, and achieve interpretable SMPL(-X) body shape optimization and generation through these measurement parameters. By introducing differentiable virtual measurements, it is possible to further optimize body shape parameters during iterative optimization, thereby obtaining a smoother human body model with a better body shape. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 This is a reference diagram of the coordinate system used in this invention;

[0056] Figure 2 This is a reference diagram of a human body model used in this invention for intersection detection between the left half of the human body and a plane.

[0057] Figure 3 This is a flowchart of the SMPL(-X) model fitting process combining human point cloud and measurement parameters in a specific embodiment of the present invention.

[0058] Figure 4 This is a flowchart of the SMPL(-X) model for generating a target body shape by measuring parameters in a specific embodiment of the present invention;

[0059] Figure 5 This is a rendering of the SMPL-X model obtained after fitting in a preferred embodiment of the present invention.

[0060] Figure 6 This is a rendering of the SMPL-X model of four target body types generated by four sets of male measurement parameters in a preferred embodiment of the present invention.

[0061] Figure 7 This is a rendering of the SMPL model of four target body types generated by four sets of female measurement parameters in a preferred embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1

[0064] This embodiment provides a method for fitting an SMPL(-X) model corresponding to body shape and posture based on human point cloud and some measurement parameters. The body shape parameter β is a 10-dimensional vector. The implementation process is as follows: Figure 3 As shown, the specific process is as follows:

[0065] S1. Translate the entire human point cloud along its center of gravity to a position with the origin as the center of gravity, in mm. Then set the camera distance to 1200mm, using an aspect ratio of a=1 and a vertical field of view of [missing information]. Near plane z n =10mm, far plane z f =10000mm perspective projection, the camera starts rendering the human body from the front, and then the human body point cloud is rotated 30 degrees counterclockwise around the y-axis each time, for a total of 11 rotations, finally rendering 12 full-body human body images with a resolution of 512x512 from different perspectives.

[0066] S2. Using 12 full-body images as input to the deep neural network OpenPose, predict the Body25 2D joint set (containing 25 joints) for each full-body image and the confidence level of each 2D joint.

[0067] S31. Select 2D joint sets from 12 full-body human images, with 7 perspectives numbered 1, 2, 6, 7, 8, 11 and 12 respectively. Then, exclude all 2D joints with confidence scores below the second threshold of 0.35 from these 7 perspective 2D joint sets to obtain the 2D joint set.

[0068] S32. From the 2D joint selection set, 25 3D joints are reconstructed using triangulation: Let... Let V be the set of 2D joints consisting of the remaining viewpoints of the current joint, where N' represents the number of remaining viewpoints, i represents the i-th viewpoint among the remaining viewpoints, and V i Indicates the corresponding viewpoint number. This indicates that the joint is in viewpoint number V i 2D joints below and They represent The x and y coordinates are set for each viewpoint V. i 3×3 camera rotation matrix Translation vector Where vector and Representing the rotation matrix Columns 1, 2, and 3, and They represent The x, y, and z coordinates are set, and the same 4×4 perspective projection matrix is ​​set for all 12 viewpoints. in a = 1 indicates the aspect ratio. Z represents the vertical field of view. n =10mm represents the near-plane position distance, z f =10000mm represents the distance to the far-plane position. The angle of view V is calculated using the following formula. i The 2D key points are subjected to inverse image space transformation and inverse projection transformation to obtain the 3D direction vector of the camera viewpoint.

[0069]

[0070] in Representing perspective V i The direction vector below, and They represent The x and y coordinates, W = 512 and H = 512 represent the width and height of the full-body image, respectively.

[0071] For each perspective V i List the following two equations:

[0072]

[0073]

[0074] Where x' represents the position of the 3D joint to be determined.

[0075] Given N' remaining viewpoints, 2N' equations are derived. The 3D keypoints x' are then solved using the least squares method. Finally, x' is projected onto the image space of the remaining viewpoints to obtain the reprojected 2D keypoint set. The reprojection of 2D joints is calculated using the following formula.

[0076]

[0077] in, It is a column vector located in the projection space. and express x, y, z, and w coordinates The viewpoint is numbered V. i Reprojection of 2D joints.

[0078] Finally, the average pixel error is calculated using the following formula:

[0079]

[0080] Where e is the average pixel error, and ||·||2 is used to calculate the Euclidean distance between vectors.

[0081] S33. If the average pixel error e is less than the first threshold 5, or the number of remaining viewpoints N' is less than 4, the final predicted 3D joint point x' is obtained; otherwise, the following method is used to eliminate noisy viewpoints until the condition is met: For the 2D joint point set composed of the remaining viewpoints of the current joint... Let Q i for The point set obtained by excluding the 2D joints corresponding to the i-th viewpoint from the remaining viewpoints, for Q i 3D joints x were reconstructed using triangulation. i Then, reprojection is performed and the average pixel error e is calculated. i Let m be the set of the average pixel errors to be calculated. The minimum value e in m The corresponding subscript, if e m Less than the first threshold of 5, or the point set Q m If the number of remaining viewpoints is less than 4, the final predicted 3D keypoint x is obtained. m ', otherwise Q m It is used as input to the noise elimination perspective method and continues to be executed.

[0082] S4. First, translate the SMPL(-X) model to the midpoint of the hip joint of the reconstructed 3D joint to complete the initial alignment.

[0083] Measurement of the SMPL(-X) model requires the feature points to be defined in advance from the vertices of the SMPL(-X) model. The complete definition is shown in Table 1.

[0084] Table 1. Feature point definition table required for SMPL(-X) measurement

[0085]

[0086] Measurements of the SMPL(-X) model need to be performed under a uniform T-pose. Figure 1 The T-pose and coordinate axes of the SMPL(-X) model are shown.

[0087] Among them, length-related measurements include height, arm span, upper limb length, and perineal height. Height is obtained by calculating the vertical distance from the top of the head to the left heel on the y-axis. Arm span is obtained by calculating the horizontal distance from the tip of the left middle finger to the tip of the right middle finger on the x-axis. Upper limb length is obtained by calculating the distance from the left shoulder to the tip of the left middle finger. Perineal height is obtained by calculating the vertical distance from the perineum to the left heel on the y-axis.

[0088] Among them, the circumference-related measurements include chest circumference, waist circumference, hip circumference and left thigh circumference. The circumference-related measurements first require selecting a feature point for horizontal cutting. Combined with the normal vector n=(0,1,0), a planar mesh that can completely cut the SMPL(-X) model is determined by the point normal formula. Then, the SMPL(-X) model and the planar mesh are intersected to obtain the set of intersecting vertices. The set of convex hull vertices is obtained by removing duplicate vertices and using the convex hull algorithm. Finally, the circumference of the convex hull vertex set is calculated.

[0089] Before measuring the bust, waist, hip, and left thigh circumference, horizontal segmentation is required using the right chest point, navel point, perineum point, and left thigh point, respectively. Furthermore, when measuring the left thigh circumference, an SMPL(-X) model containing only the left half of the body needs to be defined and its intersection with the planar mesh performed to avoid self-intersection of the vertices of the legs affecting the intersection detection results. Figure 2 As shown.

[0090] Since the intersection detection is performed on the GPU, different output results will result in different convex hull vertex sets and multiple measurement results. It is necessary to perform 30 intersection detections and convex hull calculations for the bust, waist, hip, and left thigh circumference respectively, and take the maximum value of the measurement results as the final circumference.

[0091] The objective function to be minimized is defined as follows: E = λ CD E CD +λ p E p +λ a E a +λ m E m

[0092] Where E is the objective function, E CD E represents the chamfer distance loss term. p For the distance loss term of 3D joints, E a E represents the Euler angle prior loss term for the whole skeleton. m For the measurement parameter constraint loss term, λ CD , λ p , λ a and λ m This indicates the weights corresponding to each loss term.

[0093] Chamfer distance loss term E CD Calculate using the following formula:

[0094]

[0095] in, This represents all vertices of the SMPL(-X) model. express The vertex in This represents all the vertices in the human body point cloud. This indicates that the human body point cloud retains pre-set parameters through downsampling of the farthest point. The vertex of the number, express The vertex in This represents the number of vertices in the SMPL(-X) model, and min represents finding the minimum distance between a single vertex and all points in the set.

[0096] 3D joint distance loss term E p Calculate using the following formula:

[0097]

[0098] Where M(θ,β) represents the generation of the SMPL(-X) model using the pose parameter θ and body shape parameter β, J k (·) represents the mapping function from the SMPL(-X) model to the k-th 3D joint, ω represents the 3×1 global camera rotation Euler angle vector, Rot(·) represents the mapping function from the 3×1 rotation Euler angle to the 3×3 rotation matrix, t represents the 3×1 global camera translation vector, and x′ (k) w represents the k-th 3D joint obtained by triangulation. k To control the weight of the k-th joint, and to ensure that the sum of the weights of all joints is 1, Table 2 lists the weights of each 3D joint in Phase 1 and Phase 2:

[0099] Table 2. Weights of 3D joints in Phase 1 and Phase 2

[0100]

[0101]

[0102] In this context, Stage 1 and Stage 2 represent the first and second halves of the iterative optimization process, respectively. For example, in 100 iterations, the first 50 iterations use Stage 1 weights, and the last 50 iterations use Stage 2 weights.

[0103] E of the Euler angle prior loss term for the whole skeleton a Calculate using the following formula:

[0104] E a =avg(exp(ReLU(θ-θ)) u )+ReLU(θ l -θ))-L)

[0105] Where, θ u and θ l These represent the upper and lower bounds of the rotation Euler angles for the pose parameter θ, respectively. ReLU(·) is used to find the maximum value of each component of the matrix with respect to 0. exp(·) represents the natural exponential function of each component of the matrix. L represents a matrix of all 1s with the same dimension as θ. avg(·) represents the average value of all components of the matrix. Table 3 defines the range of motion for some skeletal points. For undefined skeletal points, the minimum and maximum Euler angles are both 0.

[0106] Table 3. Definition of the Range of Motion of Some Skeletal Points in SMPL(-X)

[0107]

[0108]

[0109] Measurement parameter constraint loss term E m The result is obtained by calculating the squares of the differences between height, arm span, perineal height, chest circumference, waist circumference, and hip circumference and the corresponding measurement parameters obtained on the SMPL(-X) model, and then summing them.

[0110] In this embodiment, it is first necessary to provide initial values ​​for the pose parameter θ, body shape parameter β, global camera rotation Euler angle vector ω, and global camera translation vector t. Set θ = θ0, β = β0, ω = (0,π,0), and t is the difference vector between the hip midpoint of the SMPL(-X) model and the hip midpoint obtained by triangulation reconstruction, where θ0 and β0 are both 0 vectors.

[0111] To minimize the objective function E, it is necessary to optimize the global camera rotation Euler angle ω, the global camera translation vector t, the pose parameter θ and body parameter β of the SMPL(-X) model. Gradient descent is performed using the L-BFGS optimizer, with the optimizer employing a strong wolfe linear search method, a learning rate of 0.2, a maximum number of iterations of 100, and weights λ. CD , λ p , λ a and λ m The values ​​for height, arm span, perineal height, chest circumference, waist circumference, and hip circumference were 167.6cm, 166cm, 75.5cm, 90.5cm, 85cm, and 92cm, respectively, when the loss no longer decreased. Optimization can be terminated early when the loss stops decreasing. The results are as follows. Figure 5As shown in Table 4, the results of the measurements taken using the input measurement parameters and the generated SMPL-X model are compared.

[0112] Table 4. Comparison of measurement results between input measurement parameters and generated SMPL-X model (unit: cm)

[0113] Measurement parameters Input value Measured values error height 167.600 167.620 0.020 perineal high 166.000 166.249 0.249 wingspan 75.500 75.603 0.103 chest circumference 90.500 90.531 0.031 waistline 85.000 84.952 -0.048 Hip circumference 92.000 92.055 0.055

[0114] Some implementations require pre-transformation of the human body point cloud into coordinates, including scaling and rotating the point cloud to ensure its orientation is consistent with the coordinates of the human body point cloud. Figure 1 Consistent.

[0115] In some implementations, the input is not a human point cloud but a mesh model, and all vertices of the mesh model can be directly regarded as point clouds.

[0116] Some implementations have body shape parameters with more than 10 dimensions, up to a maximum of 300 dimensions.

[0117] Example 2

[0118] This embodiment provides a method for generating an SMPL(-X) model corresponding to a body type based on measurement parameters. The body type parameter β is a 10-dimensional vector. The implementation process is as follows: Figure 4 As shown. The process is as follows:

[0119] Since the input data does not contain human point clouds, steps S1 to S33 do not need to be repeated.

[0120] S4. Measurement of the SMPL(-X) model requires the feature points to be defined in advance from the vertices of the SMPL(-X) model. The complete definition is shown in Table 1 above.

[0121] Measurements of the SMPL(-X) model need to be performed under a uniform T-pose. Figure 1 The T-pose and coordinate axes of the SMPL(-X) model are shown.

[0122] Among them, length-related measurements include height, arm span, upper limb length, and perineal height. Height is obtained by calculating the vertical distance from the top of the head to the left heel on the y-axis. Arm span is obtained by calculating the horizontal distance from the tip of the left middle finger to the tip of the right middle finger on the x-axis. Upper limb length is obtained by calculating the distance from the left shoulder to the tip of the left middle finger. Perineal height is obtained by calculating the vertical distance from the perineum to the left heel on the y-axis.

[0123] Among them, the circumference-related measurements include chest circumference, waist circumference, hip circumference and left thigh circumference. The circumference-related measurements first require selecting a feature point for horizontal cutting. Combined with the normal vector n=(0,1,0), a planar mesh that can completely cut the SMPL(-X) model is determined by the point normal formula. Then, the SMPL(-X) model and the planar mesh are intersected to obtain the set of intersecting vertices. The set of convex hull vertices is obtained by removing duplicate vertices and using the convex hull algorithm. Finally, the circumference of the convex hull vertex set is calculated.

[0124] Before measuring the bust, waist, hip, and left thigh circumference, horizontal segmentation is required using the right chest point, navel point, perineum point, and left thigh point, respectively. Furthermore, when measuring the left thigh circumference, an SMPL(-X) model containing only the left half of the body needs to be defined and its intersection with the planar mesh performed to avoid self-intersection of the vertices of the legs affecting the intersection detection results. Figure 2 As shown.

[0125] Intersection detection is performed on the GPU. Different output results will result in different convex hull vertex sets and multiple measurement results. It is necessary to perform 30 intersection detections and convex hull calculations for the bust, waist, hip, and left thigh circumference respectively, and take the maximum value of the measurement results as the final circumference.

[0126] Since the input data does not include human point clouds, the objective function to be minimized is λ. m E m E m For the measurement parameter constraint loss term, λ m This represents the weights corresponding to each loss term. Measurement parameter constraint loss term E m The difference between each input measurement parameter and the corresponding measurement parameter obtained on the SMPL(-X) model is calculated by squared summation. The input measurement parameters are any combination of the eight items: height, arm span, perineal height, upper limb length, chest circumference, waist circumference, hip circumference, and thigh circumference.

[0127] In this embodiment, the initial value of the body size parameter β needs to be set to a vector of 0. Then, the objective function λ is minimized. m E m At this point, the body size parameter β needs to be optimized. The L-BFGS optimizer is used for gradient descent, with the optimizer employing a strong wolfe linear search method, a learning rate of 0.2, a maximum number of iterations of 60, and weights λ. m The optimization can be terminated early when the loss reaches 10,000,000 and stops decreasing.

[0128] Table 5 shows a comparison of the results of the four sets of male measurement parameters with the results of the generated SMPL-X model. The mean error is the average of the absolute values ​​of the errors between the measured values ​​and the input values ​​obtained from five separate measurements using the same measurement parameters to generate the SMPL-X model. The visualization results of the generated SMPL-X model are shown below. Figure 6 As shown.

[0129] Table 5. Comparison of male measurement parameters and results measured using the generated SMPL-X model (unit: cm)

[0130]

[0131] Table 6 shows a comparison between the four sets of female measurement parameters and the results of the generated SMPL model. The mean error is the average of the absolute values ​​of the errors between the measured values ​​and the input values ​​obtained from five separate measurements using the same parameters to generate the SMPL model. The visualization results of the generated SMPL model are shown below. Figure 7 As shown.

[0132] Table 6. Comparison of Female Measurement Parameters and Results from the Generated SMPL Model (Unit: cm)

[0133]

[0134] Table 7 shows the mean absolute error between the measured values ​​and the input values ​​obtained from the SMPL model generated by the eight measurement parameters. The best method for each measurement parameter is indicated in bold.

[0135] Table 7. Comparison of Mean Absolute Error of Measurement Parameters and SMPL Model Generation (Unit: cm)

[0136]

[0137] Therefore, it is evident that the SMPL(-X) model generated by this invention achieves good results even when all eight measurement parameters are input simultaneously, with lower errors on some parameters, demonstrating stronger scalability compared to other methods. Some implementations have body shape parameters with more than 10 dimensions, reaching up to 300 dimensions.

[0138] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A parametric human body model fitting method based on human point cloud and measurement parameters, characterized in that, The fitting method includes the following steps: S1. Render the human body point cloud at a fixed distance and from multiple different perspectives to obtain a full-body human image from each different perspective. S2. For the full-body human image from each viewpoint, predict the 2D key point set and the confidence level corresponding to each key point from each viewpoint using the deep neural network OpenPose. S3. For the 2D joint set under all viewpoints, the 2D joint set is obtained by filtering the viewpoints and excluding low-confidence 2D joints. Then, the 3D joints are reconstructed using the triangulation algorithm. The 3D joints with reprojection loss exceeding the first threshold are optimized by excluding noisy viewpoints. S4. Given the initial pose parameters θ0 and body shape parameters β0 of the parametric human model SMPL(-X), hereinafter referred to as the SMPL(-X) model, when the zero-vector pose parameters are provided, the generated human model presents a T-pose with the palms extended. Feature points required for measurement are predefined on the SMPL(-X) model. The required measurement parameters are obtained by measuring the SMPL(-X) model under the T-pose generated with specific body shape parameters. Then, the SMPL(-X) model is fitted. In the fitting process, the SMPL(-X) model is first translated to the mid-hip point of the reconstructed 3D joint to complete the initial alignment. Then, the reconstructed 3D joint, human point cloud and input measurement parameters are used as constraints to minimize the objective function through iterative optimization to obtain the optimized pose parameters θ and body shape parameters β of the SMPL(-X) model.

2. The parametric human body model fitting method based on human point cloud and measurement parameters according to claim 1, characterized in that, In step S1, the entire human point cloud is translated along its center of gravity to a position with the origin as the center of gravity, with coordinate units in mm. Then, the camera distance is set to 1200mm, and an aspect ratio of a=1 and a vertical field of view are used. Near plane z n =10mm, far plane z f =10000mm perspective projection, the camera starts rendering the human body from the front, and then the human body point cloud is rotated 30 degrees counterclockwise around the y-axis each time, for a total of 11 rotations, finally rendering 12 full-body human body images with a resolution of 512x512 from different perspectives.

3. The parametric human body model fitting method based on human point cloud and measurement parameters according to claim 2, characterized in that, In step S2, 12 full-body human images are used as input to the deep neural network OpenPose to predict the set of 2D human joints for each full-body human image and the confidence level of each 2D joint. Let K be the number of 2D human joints predicted by OpenPose.

4. The parametric human body model fitting method based on human point cloud and measurement parameters according to claim 3, characterized in that, The process of step S3 is as follows: S31. Select N sets of 2D joints from 12 full-body human images, and then exclude all 2D joints with confidence scores below the second threshold from these N sets of 2D joints to obtain the 2D joint selection set. S32. From the 2D joint selection set, reconstruct K 3D joints using triangulation: Let... Let V be the set of 2D joints consisting of the remaining viewpoints of the current joint, where N' represents the number of remaining viewpoints, i represents the i-th viewpoint among the remaining viewpoints, and V i Indicates the corresponding viewpoint number. This indicates that the joint is in viewpoint number V i 2D joints below and They represent The x and y coordinates are set for each viewpoint V. i 3×3 camera rotation matrix Translation vector Where vector and Representing the rotation matrix Columns 1, 2, and 3, and They represent The x, y, and z coordinates are set, and the same 4×4 perspective projection matrix is ​​set for all 12 viewpoints. in 'a' represents the aspect ratio. Indicates the vertical field of view. z n z represents the near-plane position distance. f The distance to the far-plane position is expressed by the following formula for the viewing angle V. i The 2D key points are subjected to inverse image space transformation and inverse projection transformation to obtain the 3D direction vector of the camera viewpoint: in Representing perspective V i The direction vector below, and They represent The x and y coordinates, W and H represent the width and height of the full-body image, respectively; For each perspective V i List the following two equations: Where x' is the position of the 3D joint to be determined; Given N' remaining viewpoints, 2N' equations are derived. The 3D keypoints x' are then solved using the least squares method. Finally, x' is projected onto the image space of the remaining viewpoints to obtain the reprojected 2D keypoint set. Calculate the reprojected 2D joints using the following formula: in, It is a column vector located in the projection space. and express x, y, z, and w coordinates The viewpoint is numbered V. i Reprojection of 2D joints; Finally, the average pixel error is calculated using the following formula: Where e is the average pixel error, and ||·||2 is used to calculate the Euclidean distance between vectors; S33. If the average pixel error e is less than the first threshold, or the number of remaining viewpoints N' is less than 4, the final predicted 3D joint point x' is obtained; otherwise, the following method is used to eliminate noisy viewpoints until the condition is met: For the 2D joint point set composed of the remaining viewpoints of the current joint... Let Q i for The point set obtained by excluding the 2D joints corresponding to the i-th viewpoint from the remaining viewpoints, for Q i 3D joints x were reconstructed using triangulation. i Then, reprojection is performed and the average pixel error e is calculated. i Let m be the set of the average pixel errors to be calculated. The minimum value e in m The corresponding subscript, if e m Less than the first threshold, or the point set Q m If the number of remaining viewpoints is less than 4, the final predicted 3D keypoint x is obtained. m ', otherwise Q m It is used as input to the noise elimination perspective method and continues to be executed.

5. The method for fitting a parametric human body model based on human point cloud and measurement parameters according to claim 4, characterized in that, The measurement method in step S4 obtains eight measurement parameters: height, perineal height, arm span, upper limb length, chest circumference, waist circumference, hip circumference, and thigh circumference. Then, it is expanded to support more measurement parameters, including hip width, waist width, thigh length, upper arm length, and shoulder width.

6. The method for fitting a parametric human body model based on human point cloud and measurement parameters according to claim 5, characterized in that, The measurement method in step S4 requires pre-defining feature points from the vertices of the SMPL(-X) model, including: head vertex, left heel point, left middle finger tip point, right middle finger tip point, left shoulder point, perineum point, right chest point, navel point, perineum point, and left thigh point. If it is necessary to further expand the measurement categories, more feature points for measurement can be defined, and then length-related and circumference-related measurements can be performed under a unified T-pose. Among them, length-related measurements include height, arm span, upper limb length, and perineal height. Height is obtained by calculating the vertical distance from the top of the head to the left heel on the y-axis. Arm span is obtained by calculating the horizontal distance from the tip of the left middle finger to the tip of the right middle finger on the x-axis. Upper limb length is obtained by calculating the distance from the left shoulder to the tip of the left middle finger. Perineal height is obtained by calculating the vertical distance from the perineum to the left heel on the y-axis. Among them, the measurements related to circumference include chest circumference, waist circumference, hip circumference and left thigh circumference. The measurement of circumference first requires selecting a feature point for horizontal cutting. Combined with the normal vector n=(0,1,0), a planar mesh that can completely cut the SMPL(-X) model is determined by the point normal formula. Then, the SMPL(-X) model and the planar mesh are intersected to obtain the set of intersecting vertices. The set of convex hull vertices is obtained by removing duplicate vertices and using the convex hull algorithm. Finally, the circumference of the convex hull vertex set is calculated. Before measuring the bust, waist, hip, and left thigh circumference, it is necessary to perform horizontal segmentation using the right chest point, navel point, perineum point, and left thigh point, respectively. When measuring the left thigh circumference, it is necessary to define an SMPL(-X) model with only the left half of the body and perform intersection detection with the planar mesh to avoid the situation where the vertices of the two legs self-intersection affect the intersection detection results.

7. The method for fitting a parametric human body model based on human point cloud and measurement parameters according to claim 5, characterized in that, The minimization objective function is defined as follows: E=λ CD E CD +λ p E p +λ a E a +λ m E m Where E is the objective function, E CD E represents the chamfer distance loss term. p For the distance loss term of 3D joints, E a E represents the Euler angle prior loss term for the whole skeleton. m For the measurement parameter constraint loss term, λ CD , λ p , λ a and λ m This indicates the weights corresponding to each loss term; Chamfer distance loss term E CD Calculate using the following formula: in, This represents all vertices of the SMPL(-X) model. express The vertex in This represents all the vertices in the human body point cloud. This indicates that the human body point cloud retains pre-set parameters through downsampling of the farthest point. The vertex of the number, express The vertex in This represents the number of vertices in the SMPL(-X) model, and min represents finding the minimum distance between a single vertex and all points in the set. 3D joint distance loss term E p Calculate using the following formula: Where M(θ,β) represents the generation of the SMPL(-X) model using the pose parameter θ and body shape parameter β, J k (·) represents the mapping function from the SMPL(-X) model to the k-th 3D joint, ω represents the 3×1 global camera rotation Euler angle vector, Rot(·) represents the mapping function from the 3×1 rotation Euler angle to the 3×3 rotation matrix, t represents the 3×1 global camera translation vector, and x′ (k) w represents the k-th 3D joint obtained by triangulation. k To control the weight of the k-th joint, the sum of the weights of all joints is 1; E of the Euler angle prior loss term for the whole skeleton a Calculate using the following formula: And a =avg(exp(ReLU(θ-θ) u )+ReLU(θ l -θ))-L) Where, θ u and θ l represents the upper and lower bounds of the Euler angles of the pose parameter θ, respectively; ReLU(·) is used to find the maximum value of each component of the matrix with respect to 0; exp(·) represents the natural exponential function of each component of the matrix; L represents a matrix of all 1s with the same dimension as θ; and avg(·) represents the average value of all components of the matrix. Measurement parameter constraint loss term E m The difference between each input measurement parameter and the corresponding measurement parameter obtained on the SMPL(-X) model is calculated by squared summation. The input measurement parameters are any combination of the eight items: height, arm span, perineal height, upper limb length, chest circumference, waist circumference, hip circumference and thigh circumference. First, initial values ​​need to be provided for the pose parameter θ, body shape parameter β, global camera rotation Euler angle vector ω, and global camera translation vector t of SMPL(-X). Set θ = θ0, β = β0, ω = (0,π,0), and t is the difference vector between the hip midpoint of the SMPL(-X) model and the hip midpoint obtained by triangulation reconstruction. When minimizing the objective function E, it is necessary to optimize the global camera rotation Euler angle vector ω, the global camera translation vector t, the pose parameter θ and body shape parameter β of the SMPL(-X) model.

Images