A method and system for three-dimensional human body reconstruction optimization based on shadow constraints

Abstract

A three-dimensional human body reconstruction optimization method and system based on shadow constraint, the method comprising: obtaining an initial SMPL human body model, a scene foreground contour, a depth map and a shadow mask; based on the shadow mask, constructing discrete shadow pixels into a shadow grid through a pin-and-needle stitching algorithm; based on the shadow grid, adopting a coarse search-fine optimization two-stage strategy to obtain a convergent illumination direction estimation; performing scale alignment on the depth scale of the initial SMPL human body model and the real scene by introducing a global scale factor; constructing a sunlight direction orthogonal view angle based on the geometric center of the shadow grid and the illumination direction estimation, and forming two groups of complementary observations with the original camera view angle; performing joint optimization by constructing a joint energy function containing a double-view geometric consistency constraint and a pose prior constraint; and optimizing SMPL pose parameters by using a CMA-ES algorithm to obtain an optimized three-dimensional human body model.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, specifically to a 3D human body reconstruction optimization method and system based on shadow constraints. Background Technology

[0002] Monocular 3D human reconstruction technology, as an intersection of computer vision and computer graphics, has broad application prospects in fields such as virtual reality, film and television production, and medical rehabilitation. Existing technologies are mostly based on parametric human models such as SMPL (Skinned Multi-Person Linear), using deep learning methods to regress pose and shape parameters. However, they suffer from the following key problems: First, pose uncertainty: when the human body exhibits self-occlusion or limb crossing, different 3D poses may produce similar 2D projections, leading to pose estimation bias; second, high module coupling: existing optimization methods require modification of the original reconstruction network structure, making integration difficult and adaptable to different reconstruction frameworks. Summary of the Invention

[0003] This invention provides a method and system for optimizing 3D human body reconstruction based on shadow constraints. By using a stitch-like stitching algorithm and dual-view joint constraints, it achieves accurate optimization of 3D human body reconstruction results without modifying the original reconstruction network. It is an optimization scheme that can use shadow constraints to solve scale and pose problems and has plug-and-play characteristics, thereby improving the accuracy and physical consistency of monocular 3D human body reconstruction.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Firstly, a method for optimizing 3D human body reconstruction based on shadow constraints includes the following steps: S1. Obtain the initial SMPL human body model, scene foreground outline, depth map, and shadow mask; based on the shadow mask, construct a shadow mesh from discrete shadow pixels using a stitch-stitching algorithm; S2. Based on the shadow grid, a two-stage strategy of coarse search and fine optimization is adopted to obtain a converged illumination direction estimate; S3. Based on illumination direction estimation and shadow mesh, optimize the initial SMPL human body model, including: The initial SMPL human model is scaled to match the depth scale of the real scene by introducing a global scale factor; an orthogonal viewpoint of the sunlight direction is constructed based on the geometric center and illumination direction estimation of the shadow mesh, forming two complementary observations with the original camera viewpoint; joint optimization is performed by constructing a joint energy function that includes dual-view geometric consistency constraints and pose prior constraints; the SMPL pose parameters are optimized using the CMA-ES algorithm to obtain the optimized 3D human model.

[0005] Preferably, in S1, obtaining the shadow mask includes: The 3D human body mesh, ground geometry, camera intrinsic and extrinsic parameters, and sunlight source parameters are input into the Blender rendering engine. Two images are generated under the conditions of enabling and disabling sunlight shadow casting, respectively. Pixel-level difference processing is performed on the two images to obtain the initial color difference map. The first filtering result is obtained by filtering the non-shaded pixels of the initial color difference image using a differential mask; In the RGB color space, a color difference threshold range is defined; only pixels whose color difference values ​​in the initial color difference image are within the color difference threshold range are retained to obtain the second filtering result. The depth map is used to convert pixels into three-dimensional points in the world coordinate system. Ground height values ​​and height tolerances are set, and ground area pixels whose height values ​​are within the range determined by the ground height values ​​and height tolerances are filtered to obtain the third filtering result. The intersection of the first, second, and third filtering results yields the effective shadow mask.

[0006] Preferably, in S1, constructing a shadow mesh from discrete shadow pixels using a needle-and-thread stitching algorithm includes: Traverse the valid shadow mask, extract the valid shadow vertices of each row, sort them by column coordinates, establish the mapping relationship between pixel coordinates and 3D point cloud index, and obtain the vertex set of each row; The two-pointer method is used to match and determine the vertices of adjacent rows to generate adjacent vertex pairs; Triangular patches are constructed based on adjacent vertex pairs; the construction process of triangular patches is extended row by row to form a shadow mesh covering the shadow area.

[0007] As a preferred embodiment, constructing triangular patches based on adjacent vertex pairs in S1 includes: Matching is performed between vertices in adjacent rows; when the absolute value of the difference between the column coordinates of two vertices is less than or equal to a preset pixel threshold, the two vertices are determined to be adjacent vertices; connect two vertices in the previous row with one vertex in the current row or connect two vertices in the current row with one vertex in the previous row to generate a triangular facet.

[0008] Preferably, S2 includes: S21. In the coarse search stage, the upper hemisphere space is uniformly discretized into multiple candidate lighting directions to form an initial candidate direction set; for each candidate direction, the matching energy between the predicted shadow and the actual shadow is calculated, and the direction with the smallest energy is selected as the optimal direction for the coarse search. S22. In the fine optimization stage, the optimal candidate direction of the initial candidate direction set is used as the initial value. The illumination direction vector is iteratively optimized in the continuous space by the CMA-ES algorithm to obtain a converged illumination direction estimate.

[0009] As a preferred embodiment, the depth scale alignment between the initial SMPL human model and the real scene in S3 includes: Obtain the visible surface vertices of the initial SMPL human body model; construct the scale alignment error function based on the visible surface vertices and the foreground point cloud using the least squares criterion; obtain the closed-form solution of the global scale factor by differentiating the alignment error function with respect to the global scale factor and setting the gradient to zero; multiply the vertices of the initial SMPL human body model by the global scale factor.

[0010] As a preferred option, constructing orthogonal viewpoints in the direction of sunlight in S3 includes: The geometric center of the shadow grid is used as the observation target point; the camera viewpoint is placed at a distant position estimated along the negative illumination direction of the observation target point; the upward direction of the camera is estimated based on the principal axis direction of the human body bounding box and the illumination direction; three orthogonal unit axes of the camera coordinate system are constructed in sequence, and then the rotation matrix and translation vector are generated to complete the orthogonal camera extrinsic parameter configuration; this orthogonal viewpoint and the original perspective camera viewpoint together constitute a dual-view observation.

[0011] As a preferred option, the dual-view geometric consistency constraints in S3 include: Foreground constraints from the camera's perspective are used, and Dice loss is employed to measure the degree of matching between the projected contour of the SMPL model and the real foreground contour. The sunlight direction shadow constraint is based on the consistency between the model shadow and the real shadow, which is measured by the orthogonal projection of sunlight.

[0012] As a preferred embodiment, the joint energy function in S3 is constructed as follows: in, Foreground constraints for camera viewpoint, Constraints on shadow direction from sunlight. Multi-level attitude prior constraints are constructed as follows: in, Lightweight constraint loss for critical joints, used to suppress small abnormal rotations of critical joints. This is for extreme joint constraint loss, used to avoid non-physiological joint distortion. This is a global attitude regularization used to suppress overall attitude deviations. , These are positive and negative exponential penalties, respectively. , It is the set of degrees of freedom that can rotate extremely in a certain direction.

[0013] Secondly, a 3D human body reconstruction optimization system based on shadow constraints includes: The shadow mesh construction module is used to obtain the initial SMPL human body model, scene foreground outline, depth map and shadow mask; based on the shadow mask, the discrete shadow pixels are constructed into a shadow mesh through a stitch-like stitching algorithm; The illumination direction estimation module is used to obtain a converged illumination direction estimate based on the shadow grid and employing a two-stage strategy of coarse search and fine optimization. The scale alignment and viewpoint construction module is used to scale the initial SMPL human body model with the depth scale of the real scene by introducing a global scale factor; and to construct an orthogonal viewpoint in the sunlight direction based on the geometric center and illumination direction estimation of the shadow mesh, forming two sets of complementary observations with the original camera viewpoint. The joint optimization module is used to perform joint optimization by constructing a joint energy function that includes dual-view geometric consistency constraints and attitude prior constraints; the CMA-ES algorithm is used to optimize the SMPL attitude parameters to obtain the optimized 3D human body model. The aforementioned shadow constraint-based 3D human body reconstruction optimization system is used to implement the shadow constraint-based 3D human body reconstruction optimization method and its steps as described in the first aspect.

[0014] Compared with the prior art, the beneficial effects of the present invention are reflected in: 1. Unlike technical solutions that directly use discrete shadow pixels as two-dimensional masks in loss calculations or rely on general point cloud reconstruction to obtain irregular shadow geometry, this invention employs a stitch-and-paste algorithm: valid shadow vertices are extracted by scanning image rows and sorted by column coordinates. Adjacent vertices are matched between adjacent rows using two pointers. Then, triangular patches are generated row by row according to the rule of "two vertices in the previous row + one vertex in the current row" or "two vertices in the current row + one vertex in the previous row," constructing a continuous shadow mesh from the topologically untopological shadow point cloud. This shadow mesh serves as a projectable geometric carrier, enabling subsequent illumination direction estimation and sunlight direction shadow contour constraints to be calculated on a unified three-dimensional geometry. This avoids constraint failure caused by the instability of discrete point cloud projection, thereby improving the usability and optimization stability of shadow constraints.

[0015] 2. Unlike technical solutions that only utilize foreground contours or keypoints for pose optimization from the original camera's perspective, this invention constructs an orthogonal viewpoint complementary to the original camera's perspective based on the estimated sunlight direction. It also constructs a dual-view joint energy function that includes foreground contour consistency constraints from the camera's perspective and shadow contour consistency constraints from the sunlight direction. Combined with multidimensional pose priors, the CMA-ES algorithm is used to optimize the pose parameters of the initial SMPL human model. The dual-view constraint simultaneously anchors the human's 3D pose to two complementary sets of observations: foreground projection and shadow projection. Compared to a single-view constraint, this more effectively eliminates pose uncertainties caused by self-occlusion and projection ambiguity. Furthermore, after being integrated with the scale alignment process, it further reduces reconstruction errors such as PVE and MPJPE.

[0016] It should be noted that the above two technical effects arise from the combination of "shadow meshing geometric representation" and "foreground-shadow dual-view joint constraint", rather than the individual use or simple chaining of any existing module; the standardized interface is only used as an engineering integration method and is not the main inventive point of this invention. Attached Figure Description

[0017] Figure 1 This is a flowchart of the model structure of the present invention.

[0018] Figure 2 This is a flowchart of the optimized human body reconstruction process of the present invention. Detailed Implementation

[0019] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0020] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0021] For ease of expression, the following standardized terms are used in this invention: (1) "Initial SMPL human body model": refers to the SMPL parameterized human body model output by monocular 3D human body reconstruction methods such as WHAM and which has not yet been optimized by the shadow constraint of this invention, including the pose parameter θ, the shape parameter β and the 3D mesh vertex V generated by the parameters; (2) "Optimized SMPL human model": refers to the SMPL human model output after scale alignment and pose optimization by the present invention; (3) "Shadow mask": refers to a two-dimensional binary (or logical) mask on the image plane that identifies the actual shadow area; (4) "Shadow Mesh": refers to a three-dimensional mesh with triangular topological connections, which is constructed by stitching the three-dimensional points corresponding to the shadow mask using a stitching algorithm; (5) Unless otherwise stated, in pixel coordinates (u,v), u represents the column coordinate (horizontal direction) of the image and v represents the row coordinate (vertical direction) of the image.

[0022] Example 1: like Figure 1 , Figure 2The plug-and-play shadow-constrained 3D human body reconstruction optimization method shown includes the following steps: S1. Obtain the initial SMPL human body model (Skinned Multi-Person Linear), scene foreground outline, depth map and shadow mask; perform filtering on the shadow mask to obtain the effective shadow area, and then use the needle-and-thread stitching algorithm to construct a continuous triangular mesh from the discrete shadow pixels. The inputs for this step are: the initial SMPL human model output by the monocular reconstruction network, the scene RGB / rendered image (or its alternative observations), the depth map, and the camera parameters; the output for this step is: the effective shadow mask. And shadow meshes. Specifically, the initial input data obtained in the data preprocessing stage includes: the initial SMPL human model (including pose parameters θ, shape parameters β, and 3D mesh vertices V) obtained through monocular reconstruction models such as WHAM; and the scene foreground contours obtained based on Blender rendering or dedicated estimation models. Depth map D and shadow mask.

[0023] S1.1 Shadow Mask Generation and Triple Filtering The inputs to this sub-step are: 3D human body mesh (and / or scene mesh), ground geometry, camera intrinsic and extrinsic parameters, and sunlight source parameters; the output of this sub-step is: effective shadow mask M_shadow.

[0024] The shadow mask is generated using the Blender rendering engine. The 3D human mesh, ground geometry, camera intrinsics and extrinsic parameters, and sunlight source parameters are input into the Blender rendering engine. Two images are obtained by rendering with sunlight shadow casting enabled (use_shadow=True) and disabled (use_shadow=False). and The initial color difference map is obtained by performing pixel-level subtraction on the two: in, This represents the image rendered when shadow casting is enabled. This represents the image rendered when shadow casting is turned off; This represents the color difference map of corresponding pixels in two images.

[0025] To eliminate non-realistic shadow areas (such as false shadows caused by material reflections or indirect lighting), a triple-screening strategy is introduced: Image difference method: Filters out non-shadow pixels using a difference mask, i.e.: in, Represents the column coordinates of the image. Represents the row coordinates of the image; Indicates color difference image at the pixel level Color difference value at the location (can be RGB Euclidean distance or its single-channel amplitude); This indicates an indicator function that takes the value 1 if the condition is true and 0 otherwise. This is a differential mask.

[0026] Color difference range method: Define the color difference threshold range (Values ​​range from 10 to 25, Euclidean distance in RGB color space), only pixels that meet the conditions are retained, i.e.: in, , These are the lower and upper thresholds for color difference, respectively. This is a color difference range mask.

[0027] Ground-based filtering method: This method uses depth maps to convert pixels into 3D points in the world coordinate system. Let the ground height be... Define tolerance Filter the pixels in the ground area, that is: in, Indicates ground reference height; Indicates height tolerance; This is a mask for the ground region.

[0028] The final effective shadow mask is the intersection of the three filtering results: .

[0029] S1.2, Needle-and-thread stitching algorithm for constructing shadow mesh The input for this sub-step is: valid shadow mask. Pixel-to-3D point mapping obtained by backprojection of depth map The output of this sub-step is: Shadow mesh (set of vertices and set of triangles).

[0030] To address the problem of discrete shadow point clouds lacking topological structure, this invention proposes a stitch-like stitching algorithm. First, it iterates through the valid shadow masks. Extract the valid shadow vertices of each row and sort them by column coordinates. Establish an index mapping between pixels and the 3D point cloud to obtain the vertex set for each row: Where w is the column coordinate and vid is the global index of the vertex.

[0031] Then, a two-pointer method is used to match vertices in adjacent rows. Let the set of vertices in the current row be . The set of vertices in the previous row is Simultaneously traverse two sets, and when the difference between the column coordinates of two vertices satisfies ( When the value is 1 to 2 pixels, the two vertices are determined to be adjacent vertices.

[0032] Based on this, triangular patches are generated using either "two vertices from the previous row + one vertex from the current row" or "two vertices from the current row + one vertex from the previous row," and then extended row by row to form a continuous triangular mesh. Boundary handling is performed for cases of index out-of-bounds errors or excessively large column coordinate differences, extending triangular connections only along a single scanning direction to ensure the continuity and closure of the mesh structure. This mesh can completely preserve the spatial structure of the shadow region, providing a stable shadow mesh for subsequent projection modeling. .

[0033] S2. The illumination direction estimation adopts a two-stage strategy of "coarse search - fine optimization" to ensure a balance between accuracy and efficiency: The inputs for this step are: the initial SMPL human model, the shadow mesh (or its outline points), and ground / observation plane parameters; the output for this step is: a unit vector representing the direction of sunlight. .

[0034] S2.1, Coarse Search Stage The input to this sub-step is: the initial SMPL human model and real shadow geometry required for candidate direction sampling configuration and shadow matching energy; the output of this sub-step is: the optimal candidate direction obtained through coarse search. .

[0035] The upper hemisphere, centered on the geometric center of the illuminated person (or the area where their shadow falls), and with the vertically upward axis as its polar axis, is uniformly discretized according to azimuth and elevation angles to form a set of candidate illumination directions covering possible angles of sunlight incidence. In a preferred embodiment, the azimuth angle is... Twelve samples were uniformly taken, and six samples were uniformly taken within the effective elevation angle range of the upper hemisphere, resulting in a total of 72 candidate illumination directions. For each candidate direction, the matching energy between the predicted shadow and the actual shadow was calculated, and the direction with the lowest energy was selected as the optimal direction for coarse search. .

[0036] S2.2, Detailed Optimization Stage The input for this sub-step is: the optimal direction for the coarse search. The output of this sub-step is: fine lighting direction. .

[0037] by Using the initial values, the illumination direction vector is iteratively optimized in continuous space using the CMA-ES algorithm (Covariance Matrix Adaptation Evolution Strategy). .

[0038] Given initial SMPL model parameters Direction of light With the observation plane (Usually the ground), calculate the shadow position of each SMPL vertex by finding the intersection of the ray and the plane, where the ray equation is: Combining plane equations ( , For plane (Parameters), the intersection parameters can be solved: And thus the corresponding shadow prediction point is obtained: In the energy function construction stage, a two-way Chamfer distance metric is used to predict shadows. With realistic shadow outline The differences between them are determined, and the energy function is defined: The bidirectional Chamfer distance is specifically represented as... Based on this, the energy function is minimized iteratively optimized using the CMA-ES algorithm. The algorithm can adaptively adjust the search direction and step size, and finally converges to obtain the accurate direction of sunlight.

[0039] S3. Human Reconstruction Optimization: Based on Illumination Direction Estimation The initial SMPL human body model is scale-aligned, dual-view construction is performed, and joint optimization is carried out using the Shadow mesh to obtain the optimized SMPL human body model.

[0040] The inputs for this step are: initial SMPL human model, depth map and foreground outline, shadow mesh, and sunlight direction. Original camera parameters; the output of this step is: scale factor. and optimized attitude parameters .

[0041] S3.1 Depth Scale Alignment The inputs to this sub-step are: initial SMPL human model, depth map, scene foreground contour, and camera parameters; the output of this sub-step is: global scale factor. And the human body vertices after scale alignment.

[0042] To address the inherent scale ambiguity problem in monocular human reconstruction, this paper introduces a global scale factor τ∈R to construct an extended human state representation that integrates scale, pose, and shape. θ is the attitude parameter and β is the shape parameter.

[0043] The visible vertex determination method is as follows: Project the 3D vertices of the initial SMPL human body model onto the image plane via a camera to obtain the projected coordinates. ;like If a pixel falls within the foreground contour mask and its depth is valid (within a preset valid range), then that vertex is considered a visible surface vertex, and all visible surface vertices form a set. This judgment makes Foreground cloud obtained by backprojection of depth map Consistent across the observation domain.

[0044] Based on this, a scale alignment error function is constructed using the least squares criterion: in, Let i be the coordinates of the i-th visible surface vertex; The corresponding foreground depth point cloud coordinates (aligned according to the same image pixel index); N is the number of valid point clouds participating in the matching. This is achieved through the error function... about By taking the derivative and setting the gradient to zero, we can directly obtain the closed-form solution for the scaling factor: in, This represents the vector dot product. It multiplies the vertices of the initial SMPL human model by... This achieves a depth scale consistency with the real scene. For ease of comparison with experiments, the scale obtained by least squares solution based on foreground depth point cloud is called Scale Optimization 2 (Scale2); in contrast, the scale obtained by one-dimensional search based solely on lighting / projection viewpoint is called Scale Optimization 1 (Scale1).

[0045] S3.2, Constructing Orthogonal Perspectives in the Direction of Sunlight The inputs for this sub-step are: Shadow mesh and Sunlight direction. Human body spatial distribution (bounding box); the output of this sub-step is: the extrinsic parameters (rotation matrix) of the orthogonal camera in the sunlight direction. With translation vector ).

[0046] Based on the estimated direction of sunlight, an orthogonal camera perspective is constructed to unify the projected geometric relationship between the foreground human figure and the shadow area.

[0047] First, use the four vertices of the shadow mesh bounding box. Calculate its geometric center and use this center as the observation target point: And this center is taken as the observation target point. Let the normalized sunlight direction vector be... (Direction from the light source to the object), adjust the camera viewpoint. Arranged along the observation target point Distant location in the direction: in, This is the distance coefficient, preferably set to 200 (consistent with scene units) to ensure the camera is located outside the object.

[0048] Then take the main axis direction of the human body enclosure box. Candidate up directions orthogonal to the lighting direction are obtained by cross product with the lighting direction. And normalize to obtain the upward direction of the camera. Based on this, three orthogonal unit axes of the camera coordinate system are constructed sequentially. This leads to the generation of a rotation matrix. With translation vector The orthogonal camera extrinsic parameters were configured. This orthogonal viewpoint, together with the original perspective camera viewpoint, constitutes a dual-view observation.

[0049] S3.3, Dual-view Joint Optimization The inputs for this sub-step are: scale-aligned SMPL parameters, original camera parameters, extrinsic parameters of the sunlight orthographic camera, and true foreground contours. The output of this sub-step is: the optimized pose parameters (Shadow mesh). .

[0050] A joint optimization model integrating dual-view constraints and multi-dimensional attitude priors is constructed to achieve optimization of SMPL attitude parameters. Refined corrections.

[0051] First, a dual-view geometric consistency constraint is established. The camera-view foreground constraint uses Dice loss to measure the degree of matching between the projected contour of the SMPL model and the real foreground contour, defined as: In the formula, Indicates by posture Driven SMPL 3D mesh; For perspective camera projection operators, For the true foreground outline, The loss can be expressed as , , It is a set of two-dimensional contours.

[0052] The sunlight direction shadow constraint is based on the consistency between the sunlight orthogonal projection model shadow and the real shadow: in For the orthogonal projection operator of the sunlight direction. The shaded mesh constructed for S1.2.

[0053] To avoid abnormal rotation and non-physiological torsion of joints, multi-level posture prior constraints are further introduced. This invention defines the rotational degrees of freedom of the 24 joints in the SMPL model according to the constraint strength as follows: (1) Global attitude regularization: Apply L2 regularization to all joint rotation parameters, denoted as This is used to suppress excessive deviations in the overall posture; (2) Lightweight constraint loss of critical joints: refers to the set of joint degrees of freedom that have a significant impact on overall posture stability but allow for minor corrections. (For example, for a specific rotational component in a joint such as the shoulder), applying a lightweight L2 constraint can precisely suppress small abnormal rotations of key joints: (3) Extreme joint strong constraint loss: refers to the set of joint degrees of freedom that are prone to non-physiological flipping or excessively large angle rotation during optimization. Applying enhanced L2 constraints (to specified rotational components such as the main flexion axis of the knee joint) can effectively prevent non-physiological joint distortion. (4) Nonlinear exponential penalty: For the set of degrees of freedom that are prone to extreme rotations along a certain direction. , Apply positive and negative exponential penalties respectively: This is used to suppress positive and negative extreme rotations respectively. Based on this, a weighted fusion is performed to obtain the total attitude prior loss: Based on this, a joint energy function is constructed: The CMA-ES algorithm is used to minimize the problem, and two optimization modes are set: in the full-joint optimization mode, the pose parameters corresponding to all joints of SMPL are minimized. All are included in the update; the limb optimization mode only optimizes limb joints and fixes core joints such as the torso, while introducing an improvement rate threshold: Only retain those with an improvement rate of not less than The optimization results are then output, showing the optimized SMPL human body model. This module interfaces with the initial SMPL model output through a standardized interface, without requiring modification of the original reconstruction network structure. It can be flexibly integrated into various SMPL-based monocular 3D human body reconstruction frameworks as a general post-processing unit.

[0054] To verify the effectiveness of the above steps, shadow constraints were gradually introduced into the experiment based on artificial / scanned human data such as THuman and using the output of monocular methods such as WHAM as the baseline. The evaluation metrics used were PVE (Per-Vertex Error) and MPJPE (Mean Per Joint Position Error), with the unit being millimeters (mm).

[0055] Table 1 shows the changes in pose estimation error during the gradual introduction of shadow constraints. Wherein: "Baseline Model" represents the initial SMPL human model result without optimization by this invention; "Scale Optimization 1 ( ")" indicates that the scale factor is obtained solely through a one-dimensional search based on the lighting / projection perspective; "Scale Optimization 2 ( ")" indicates the result of the least-squares closed-form solution of the scale factor based on the visible surface vertices and the foreground depth point cloud in S3.1; "Scale Optimization 2 + Shadow Filtering" indicates the result of further optimization after introducing shadow quality filtering (screening out shadow samples with poor geometric integrity) on the basis of Scale2. Limb optimization and full joint optimization correspond to the two modes in S3.3, respectively.

[0056] As shown in Table 1, compared to the baseline model, the solution obtained using the least squares method... This can significantly reduce the two core metrics, PVE and MPJPE, compared to relying solely on one-dimensional search based on illumination perspective. The improvement effect on reconstruction error is relatively limited. This result fully demonstrates that relying solely on search strategies within the projection space cannot fundamentally solve the inherent scale ambiguity problem in monocular 3D reconstruction; in contrast, By constructing a 3D correspondence between SMPL surface points and foreground depth point clouds, scale optimization is directly applied to the level of real spatial geometry. This not only restores a more realistic overall human body size but also provides stable and reliable initialization support for subsequent posture optimization.

[0057] After achieving precise scale alignment, for better results... The method further introduces a shadow quality filtering strategy to further optimize model reconstruction performance while effectively reducing the fluctuation of optimization results. This result confirms that the actual effectiveness of shadow constraints highly depends on the geometric integrity of the shadow region. In real-world scenarios, shadow boundaries are easily affected by factors such as occlusion, truncation, and segmentation errors. If low-quality shadow data is directly included in the optimization process, incorrect projection relationships will mislead pose updates. By filtering out shadow regions with low geometric reliability, the optimization process can focus on utilizing highly consistent shadow information, fundamentally improving the stability and robustness of the overall optimization process.

[0058] Comparing experimental data from different optimization levels reveals that the full-joint level posture optimization scheme outperforms local optimization strategies targeting only limbs. Essentially, shadow constraints are the projection of the entire human body's contours onto the lighting direction. Their constraint characteristics are better suited to the calibration of the overall human structure and spatial layout, rather than the fine-tuning of local limb deformation. The optimization effect is more concentrated on the overall skeletal orientation calibration and global posture regularization. When the optimization target covers the entire joint system, the geometric consistency information contained in the shadows can be fully extracted and utilized, thereby achieving more significant performance gains. This conclusion can be further demonstrated through… Figure 1 The visualization results are intuitively verified.

[0059] Table 1: Changes in attitude estimation error during the gradual introduction of shadow constraints (unit: mm)

[0060] Further analysis of the statistical results in Table 2 shows that when only effective samples that have achieved optimization in each indicator are selected for statistical analysis, the model can achieve a lower error level. This conclusion indirectly confirms that, in ideal scenarios with sufficient shadow observation conditions and reliable geometric information, shadow constraints have greater optimization potential and can provide efficient global geometric supplementary information for monocular 3D human reconstruction tasks, further illustrating the room for improvement in model reconstruction accuracy.

[0061] Table 2: Upper limit of performance of shading constraints on valid samples (unit: mm)

[0062] It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of this invention.

[0063] Example 2: A shadow-constrained 3D human body reconstruction optimization system, comprising: The shadow mesh construction module is used to obtain the initial SMPL human body model, scene foreground outline, depth map and shadow mask; based on the shadow mask, the discrete shadow pixels are constructed into a shadow mesh through a stitch-stitching algorithm; The illumination direction estimation module is used to obtain a converged illumination direction estimate based on the shadow grid and employing a two-stage strategy of coarse search and fine optimization. The scale alignment and viewpoint construction module is used to scale the initial SMPL human model with the depth scale of the real scene by introducing a global scale factor; based on the geometric center of the shadow mesh and the estimation of the illumination direction, an orthogonal viewpoint in the sunlight direction is constructed, forming two sets of complementary observations with the original camera viewpoint. The joint optimization module is used to perform joint optimization by constructing a joint energy function that includes dual-view geometric consistency constraints and attitude prior constraints; the CMA-ES algorithm is used to optimize the SMPL attitude parameters to obtain the optimized 3D human body model.

Claims

1. A method for optimizing 3D human body reconstruction based on shadow constraints, characterized in that, Includes the following steps: S1. Obtain the initial SMPL human body model, scene foreground outline, depth map, and shadow mask; based on the shadow mask, construct a shadow mesh from discrete shadow pixels using a stitch-stitching algorithm; S2. Based on the shadow grid, a two-stage strategy of coarse search and fine optimization is adopted to obtain a converged illumination direction estimate; S3. Based on illumination direction estimation and shadow mesh, optimize the initial SMPL human body model, including: The initial SMPL human body model is scaled to match the depth scale of the real scene by introducing a global scale factor. Based on the geometric center and illumination direction estimation of the shadow mesh, an orthogonal viewpoint of the sunlight direction is constructed, forming two complementary observations with the original camera viewpoint. Joint optimization is performed by constructing a joint energy function that includes dual-view geometric consistency constraints and attitude prior constraints. The CMA-ES algorithm is used to optimize the SMPL attitude parameters, resulting in an optimized 3D human body model.

2. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 1, characterized in that, In S1, obtaining the shadow mask includes: The 3D human body mesh, ground geometry, camera intrinsic and extrinsic parameters, and sunlight source parameters are input into the Blender rendering engine. Two images are generated under the conditions of enabling and disabling sunlight shadow casting, respectively. Pixel-level difference processing is performed on the two images to obtain the initial color difference map. The first filtering result is obtained by filtering the non-shaded pixels of the initial color difference image using a differential mask; In the RGB color space, a color difference threshold range is defined; only pixels whose color difference values ​​in the initial color difference image are within the color difference threshold range are retained to obtain the second filtering result. The depth map is used to convert pixels into three-dimensional points in the world coordinate system. Ground height values ​​and height tolerances are set, and ground area pixels whose height values ​​are within the range determined by the ground height values ​​and height tolerances are filtered to obtain the third filtering result. The intersection of the first, second, and third filtering results yields the effective shadow mask.

3. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 1, characterized in that, In S1, the discrete shadow pixels are constructed into a shadow mesh using a needle-and-thread stitching algorithm, including: Traverse the valid shadow mask, extract the valid shadow vertices of each row, sort them by column coordinates, establish the mapping relationship between pixel coordinates and 3D point cloud index, and obtain the vertex set of each row; The two-pointer method is used to match and determine the vertices of adjacent rows to generate adjacent vertex pairs; Triangular patches are constructed based on adjacent vertex pairs; the construction process of triangular patches is extended row by row to form a shadow mesh covering the shadow area.

4. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 3, characterized in that, S1 constructs triangular facets based on adjacent vertex pairs, including: Matching is performed between vertices in adjacent rows; when the absolute value of the difference between the column coordinates of two vertices is less than or equal to a preset pixel threshold, the two vertices are determined to be adjacent vertices; connect two vertices in the previous row with one vertex in the current row or connect two vertices in the current row with one vertex in the previous row to generate a triangular facet.

5. The three-dimensional human body reconstruction optimization method based on shadow constraints according to claim 1, characterized in that, S2 include: S21. In the coarse search stage, the upper hemisphere space is uniformly discretized into multiple candidate lighting directions to form an initial candidate direction set; for each candidate direction, the matching energy between the predicted shadow and the actual shadow is calculated, and the direction with the smallest energy is selected as the optimal direction for the coarse search. S22. In the fine optimization stage, the optimal candidate direction of the initial candidate direction set is used as the initial value. The illumination direction vector is iteratively optimized in the continuous space by the CMA-ES algorithm to obtain a converged illumination direction estimate.

6. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 1, characterized in that, In S3, the depth scale alignment between the initial SMPL human model and the real scene includes: Obtain the visible surface vertices of the initial SMPL human body model; construct the scale alignment error function based on the visible surface vertices and the foreground point cloud using the least squares criterion; obtain the closed-form solution of the global scale factor by differentiating the alignment error function with respect to the global scale factor and setting the gradient to zero; multiply the initial SMPL human body model vertices by the global scale factor.

7. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 1, characterized in that, Constructing orthogonal viewpoints in the direction of sunlight in S3 includes: The geometric center of the shadow grid is used as the observation target point; the camera viewpoint is placed at a distant position estimated along the negative illumination direction of the observation target point; the upward direction of the camera is estimated based on the principal axis direction of the human body bounding box and the illumination direction; three orthogonal unit axes of the camera coordinate system are constructed in sequence, and then the rotation matrix and translation vector are generated to complete the orthogonal camera extrinsic parameter configuration; this orthogonal viewpoint and the original perspective camera viewpoint together constitute a dual-view observation.

8. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 1, characterized in that, The dual-view geometric consistency constraints in S3 include: Foreground constraints from the camera's perspective are used, and Dice loss is employed to measure the degree of matching between the projected contour of the SMPL model and the real foreground contour. The sunlight direction shadow constraint is based on the consistency between the model shadow and the real shadow, which is measured by the orthogonal projection of sunlight.

9. The method for optimizing 3D human body reconstruction based on shadow constraints according to claim 8, characterized in that, The joint energy function in S3 is constructed as follows: in, Foreground constraints for camera viewpoint, Constraints on shadows in the direction of sunlight. Multi-level attitude prior constraints are constructed as follows: in, Lightweight constraint loss for critical joints, used to suppress small abnormal rotations of critical joints. This is for extreme joint constraint loss, used to avoid non-physiological joint distortion. This is a global attitude regularization used to suppress overall attitude deviations. , These are positive and negative exponential penalties, respectively. , It is the set of degrees of freedom that can rotate extremely in a certain direction.

10. A three-dimensional human body reconstruction optimization system based on shadow constraints, characterized in that, The method and steps for implementing a shadow-constrained 3D human body reconstruction optimization method as described in claim 1 include: The shadow mesh construction module is used to obtain the initial SMPL human body model, scene foreground outline, depth map and shadow mask; based on the shadow mask, the discrete shadow pixels are constructed into a shadow mesh through a stitch-like stitching algorithm; The illumination direction estimation module is used to obtain a converged illumination direction estimate based on the shadow grid and employing a two-stage strategy of coarse search and fine optimization. The scale alignment and viewpoint construction module is used to scale the initial SMPL human body model with the depth scale of the real scene by introducing a global scale factor; and to construct an orthogonal viewpoint in the sunlight direction based on the geometric center and illumination direction estimation of the shadow mesh, forming two sets of complementary observations with the original camera viewpoint. The joint optimization module is used to perform joint optimization by constructing a joint energy function that includes dual-view geometric consistency constraints and attitude prior constraints; the CMA-ES algorithm is used to optimize the SMPL attitude parameters to obtain the optimized 3D human body model.

Images