Head three-dimensional reconstruction method and system based on semantic separation and PBR material prior

Abstract

This invention relates to a method and system for 3D head reconstruction based on semantic separation and PBR material prior, belonging to the field of 3D modeling technology. It includes: generating scalp and hair region masks for each frame of the target image using an image segmentation model; inputting a color image into a pre-trained deep learning model to obtain a 2D material prior map; initializing the position and physical material parameters of 3D Gaussian units in a 3D Gaussian scene model based on the depth map and material prior map, and mapping the Gaussian units to scalp and hair sets respectively; constructing physical constraint models for the scalp and hair regions respectively; establishing a boundary transition mechanism between the scalp and hair regions, and performing smooth interpolation processing on the material parameters within the transition region; and constructing a differentiable rendering pipeline to jointly optimize and adjust the geometric parameters and PBR material parameters of the 3D Gaussian scene model. This invention can reconstruct a 3D human head with normal physical response.

Description

Technical Field

[0001] This invention belongs to the field of 3D modeling technology and relates to a method and system for 3D reconstruction of a human head based on semantic separation and PBR (Physically Based Rendering) material prior. Background Technology

[0002] With the rise of the metaverse concept and the advancement of telemedicine technology, the demand for high-fidelity, photorealistic 3D human head reconstruction is becoming increasingly urgent. Especially in medical aesthetics, dermatology diagnosis (such as alopecia areata treatment evaluation), and personalized hair transplant plan design, not only is it required that the reconstructed 3D model have an accurate geometric structure, but also that its material properties (such as skin color, hair luster and texture) can truly reflect objective physical laws in order to conduct subsequent quantitative analysis.

[0003] Currently, 3D reconstruction technology is mainly divided into two categories: traditional geometry-based methods and emerging neural rendering-based methods.

[0004] Traditional geometry-based methods mainly include Structure from Motion (SfM) and Multi-View Stereo (MVS). Early hair reconstruction often employed optimization-based methods, such as using Gabor filters to extract two-dimensional orientation fields as constraints, or using Markov Random Fields (MRFs) to optimize and generate oriented point clouds. However, due to the extremely fragmented structure of hair and the existence of severe self-occlusion and complex light transmission phenomena, traditional stereo matching algorithms often fail in the hair region, resulting in noisy or void reconstruction results. Although recent studies have introduced guided hair strands for interpolation, the results are still unsatisfactory when dealing with sparse or complex hair styles.

[0005] Emerging methods based on neural rendering include Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). NeRF represents scenes using implicit neural networks and synthesizes high-quality images from new perspectives using volumetric rendering techniques. However, most original NeRF and its variants use uniform density and color fields, making it difficult to explicitly decouple geometry from materials. For materials like hair, which have strong anisotropic specular highlights, NeRF often "baks" the highlights into the color, resulting in unrealistic visual effects (such as "plasticity" or "hazy artifacts") when lighting or viewpoint changes. 3DGS, as an explicit point cloud representation method, achieves real-time rendering speed through a snowball algorithm. Although it is more flexible than NeRF in geometric representation, current 3DGS solutions mostly use spherical harmonics (SH) to fit colors, which is essentially still an overfit of view-dependent colors and does not truly recover the physical material properties of objects (albedo, roughness, metallicity).

[0006] When dealing with head reconstruction tasks, existing technologies mainly have the following problems:

[0007] 1. Material property confusion and simplified modeling

[0008] Existing methods typically treat the entire head as a homogeneous surface, failing to differentiate between "scalp" and "hair" at the material model level. The scalp, being biological tissue, exhibits subsurface scattering characteristics, macroscopically manifesting as predominantly diffuse reflection; while hair, composed of keratin fibers, has a smooth surface and strong specular reflection and anisotropic characteristics. Using a uniform material model (such as the standard Disney BRDF) to fit both leads to compromises in the optimization process: to fit the highlights of hair, the model might incorrectly increase the specular reflectivity of the scalp, making the skin appear oily or metallic; conversely, to fit the softness of the scalp, the model might suppress the highlights of the hair, making it appear like dry straw.

[0009] Part Two: The Pathological Nature and Initialization Challenges of Reverse Rendering

[0010] Inferring 3D geometry and material parameters from 2D images is a classic ill-posed problem. In the absence of known lighting and geometry, there is ambiguity in albedo and illumination. For example, a white object in darkness and a black object in light may produce the same pixel values. This ambiguity is even more pronounced in hair regions due to their complex geometry. Existing methods typically optimize material parameters starting with random noise, making them extremely prone to getting trapped in local optima. This leads to non-convergence of roughness and drift in metallic parameters, resulting in textures that may look acceptable from the training viewpoint but become completely distorted under new viewpoints or lighting conditions.

[0011] 3: Boundary transition artifacts

[0012] At the boundary between the scalp and hair (such as the hairline and the edge of alopecia areata), material properties undergo drastic changes. Existing methods lack a clear regional transition mechanism, and during optimization, unreasonable smooth transitions or severe noise are often generated at these boundaries, manifesting as color blurring, abrupt roughness changes, or localized bright spots. This is fatal for medical diagnosis, as the precise location and shape of the hairline are key indicators for assessing the degree of hair loss.

[0013] In summary, how to explicitly separate the semantics of scalp and hair in 3D reconstruction and introduce effective physical priors to constrain the pathological optimization process are pressing problems that current technology needs to solve. Summary of the Invention

[0014] In view of this, the purpose of this invention is to provide a method and system for 3D head reconstruction based on semantic separation and PBR material prior. By introducing semantic mask, the scene is decoupled at the physical level. The material prior generated by deep learning is used to solve the initialization problem. Through layered physical constraints and boundary transition mechanism, high-fidelity and physically reliable 3D head reconstruction is achieved.

[0015] To achieve the above objectives, the first aspect of this invention provides a method for 3D head reconstruction based on semantic separation and PBR material prior, the method comprising:

[0016] Acquire multi-view color images, depth maps, and camera pose information of the object to be reconstructed; generate scalp region masks and hair region masks for each frame of the image using an image segmentation model;

[0017] A color image is input into a pre-trained deep learning model to obtain a two-dimensional material prior map containing albedo, roughness, and metallicity.

[0018] A 3D Gaussian scene model is established as the expression carrier of geometry and appearance. The position and physical material parameters of the 3D Gaussian unit are initialized based on the depth map and material prior map, and the Gaussian unit is mapped to the scalp set and hair set according to the generated mask.

[0019] Physical constraint models were constructed for the scalp region and the hair region, respectively.

[0020] Establish a boundary transition mechanism between the scalp region and the hair region, calculate the blending weights based on geometric distance or learnable parameters within the transition region, and perform smooth interpolation on the material parameters based on the blending weights;

[0021] A differentiable rendering pipeline is constructed to jointly optimize and adjust the geometric parameters and PBR material parameters of the 3D Gaussian scene model.

[0022] On the other hand, the present invention provides a three-dimensional reconstruction system for a human head based on semantic separation and PBR material prior, which includes interconnected processors, input devices, output devices and memory.

[0023] The memory stores computer programs, which include program instructions. The processor is configured to invoke the program instructions to execute the method described in the first aspect.

[0024] The beneficial effects of this invention are as follows:

[0025] (1) Enhanced Realism: By explicitly decoupling the scalp and hair regions through semantic segmentation, and setting differentiated physical constraints for different optical characteristics (subsurface scattering of the scalp and anisotropic specularity of the hair), the material confusion problem caused by homogeneous modeling in traditional methods is solved. The scalp model reconstructed by this invention can avoid unrealistic "gloss" or metallic feel; at the same time, the hair region can present a delicate luster and correct specular stretching shape that conforms to its fiber structure characteristics, improving the visual realism and physical consistency of the model under lighting conditions.

[0026] (2) Optimization stability: The RGBX prior generated by the pre-trained material estimation network provides an initial starting point for the joint optimization of geometry and material, effectively constraining the ill-conditioned solution space. This enables the optimization process to converge quickly to a physically reasonable parametric manifold, avoiding problems such as parameter drift and roughness non-convergence caused by random initialization, and ensuring the accuracy and reliability of the final material parameters (such as albedo, roughness, and metallicity).

[0027] (3) Medical application value: For the area where the scalp and hair meet (such as the hairline), the boundary transition mechanism adopted in this invention can suppress unreasonable smoothing artifacts or noise while preserving the material abrupt change characteristics. This makes the geometric contours of key areas such as the hairline clear and the material transition natural, avoiding color smudging or false bright spots, and providing reliable three-dimensional data support for the measurement of alopecia areata area and the analysis of hair follicle density.

[0028] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0030] Figure 1 This is a schematic diagram of a three-dimensional head reconstruction method according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram for semantic mask generation;

[0032] Figure 3 Flowchart for prior generation and initialization of RGBX materials;

[0033] Figure 4 A comparative diagram of the physical constraint models of the scalp region and the hair region;

[0034] Figure 5 This is a schematic diagram of a differentiable rendering pipeline structure based on the Cook-Torrance model.

[0035] Figure 6 This is a schematic diagram of a three-dimensional head reconstruction system provided in an embodiment of the present invention. Detailed Implementation

[0036] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0037] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0038] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0039] Example 1

[0040] like Figure 1 As shown, this embodiment provides a method for 3D head reconstruction based on semantic separation and PBR material priors, mainly including four stages: data input and preprocessing, prior-based Gaussian field construction and initialization, differential physical property constraint modeling, and boundary-aware PBR joint optimization. The method is described in detail below:

[0041] I. Data Acquisition and Semantic Mask Generation

[0042] The quality of the data directly determines the upper limit of the reconstruction. This step aims to acquire multi-view data containing rich geometric and textural information and perform preliminary semantic analysis.

[0043] 1. Data Acquisition

[0044] In this embodiment, a mobile device equipped with a LiDAR (Light Detection and Ranging) sensor is used. The LiDAR sensor can provide a low-resolution but scale-accurate depth map, which is crucial for restoring the absolute scale of the head.

[0045] Acquire a video stream or photo sequence around the head in a 360-degree arc. The data includes color images (RGB, 4K resolution or higher), depth maps, camera intrinsics, and camera pose.

[0046] Pose calibration: Although mobile devices can provide an initial pose, in order to achieve sub-pixel level reconstruction accuracy, SfM software such as COLMAP can be used to further refine the pose.

[0047] 2. Semantic mask extraction

[0048] SAM (Segment Anything Model) or other SOTA (State-of-the-Art) semantic segmentation networks are used. These large models have strong generalization capabilities and can adapt to different hair colors, hairstyles and backgrounds.

[0049] like Figure 2 As shown, the semantic mask extraction process is as follows:

[0050] ① Each frame of image Input semantic segmentation model;

[0051] ② Use text prompts (such as "hair", "head", "scalp") or point prompts to guide the model segmentation.

[0052] ③ Output three binary masks: (All heads are masked). (hair area mask) and (Scalp area mask). In this invention, the scalp refers to the exposed scalp area caused by alopecia areata or baldness, i.e., the scalp area without hair coverage.

[0053] ④ Timing consistency optimization: Since single-frame segmentation may have jitter, optical flow can be introduced to align and fuse the masks of consecutive frames, eliminating outlier segmentation noise.

[0054] II. RGBX Material Prior Generation

[0055] To address the difficulty of initializing reverse rendering, this invention introduces the concept of "RGBX," which infers unknown physical properties X from RGB images. A pre-trained material estimation network (trained on large-scale PBR texture data) is used to process the input image, generating initial estimates of albedo, roughness, and metallicity. While these estimates may contain noise, they provide statistically consistent physical priors, effectively preventing parameter drift during the optimization process.

[0056] like Figure 3 As shown, this embodiment uses a deep learning model (such as RGB) The X model incorporates prior knowledge, and its input is a single-frame RGB image. The output is a material prior map. Includes three-channel albedo maps Single-channel roughness map Single-channel metallicity map .

[0057] The process of generating the material prior map is as follows:

[0058] ① Input all the viewpoint images obtained in step one into the deep learning model one by one. The deep learning model includes an encoder for extracting image features and a decoder for predicting physical properties. The decoder is then used to output pixel-level albedo estimates. Roughness estimate With metallicity estimate .

[0059] ② The generated material prior map This will serve as a "soft target" for subsequent iterative training processes.

[0060] Technical Insight: While single-frame prediction cannot guarantee consistency across multiple viewpoints (e.g., the roughness of the same skin may vary slightly depending on the angle), its statistical distribution (mean, variance) and local texture features are highly reliable. For example, the model can accurately identify that hair regions should have high specular reflection potential and anisotropic texture, while skin regions should be smooth and matte.

[0061] III. PBR Material Parameter Initialization and Semantic Mapping

[0062] A 3D Gaussian scene model is established, including not only geometric parameters such as position, rotation, and scaling, but also extended PBR material parameters. Using the prior image generated in step two, the material properties of each Gaussian unit are initialized through projection mapping. Simultaneously, based on semantic masks, the 3D Gaussian units are explicitly divided into "scalp sets" and "hair sets." This explicit set division allows for the application of completely different physical constraints to different materials.

[0063] First, construct a 3D Gaussian scene model:

[0064] 3D Gaussian Splatting (3DGS) is used as the basic representation. Each Gaussian unit... Defined as:

[0065]

[0066] in, The three-dimensional center position, Let covariance matrix be the variance matrix. Opacity.

[0067] PBR extension: Incorporating color attributes (Typically, the spherical harmonic coefficients SH) are replaced or enhanced with PBR attribute tuples:

[0068]

[0069] in, Albedo, For roughness, For metallicity, The direction of the normal.

[0070] Secondly, establish the initial representation of the scene:

[0071] Location 3DGS initialization is performed using sparse point clouds generated by SfM or point clouds back-projected from LiDAR depth maps.

[0072] Material : Each 3D point Project back onto the 2D plane of all visible viewpoints, index the material prior map generated in step two, and calculate the mean as the Gaussian element. Initial values ​​of material properties:

[0073]

[0074]

[0075]

[0076] in, This represents the initial value of albedo. This represents the initial roughness value. Indicates the initial value of metallicity. Indicates that the first can be observed The set of all camera viewpoints in a Gaussian unit. Indicates the first The reflective prior image predicted by a deep learning model from a single perspective. Indicates the first Roughness prior maps predicted by deep learning models from various perspectives Indicates the first Metallicity prior maps predicted by deep learning models from various perspectives. This represents the projection function that projects a point in three-dimensional space onto a two-dimensional image plane.

[0077] This initialization method allows the optimization starting point to be located directly near a "physically reasonable" manifold, greatly reducing the search space.

[0078] Finally, semantic set partitioning is performed:

[0079] Based on the semantic mask, all Gaussian units are divided into two mutually exclusive sets: the scalp set S and the hair set H.

[0080] The mapping rule is as follows: for Gaussian units The mask of its projection falling on the scalp area under all viewing angles is statistically analyzed. and hair area mask The percentage of pixels within the mask. If the probability of falling within the scalp area mask is 0.3 or higher than that of the hair area mask, then... If the probability of falling within the hair region mask is higher than 0.3 or more for the scalp region mask, then... .

[0081] Boundary marking is performed based on the semantic set of the partition, for those falling within... and Gaussian units with a probability difference less than 0.3, for example, those falling within... The probability is 0.4, falling on If the probability is 0.6, then these Gaussian units are labeled as "transition zone units" and belong to the set. .

[0082] IV. Physical Constraint Modeling of the Scalp Region

[0083] The rendering focus for the scalp is to depict its soft subsurface scattering, avoiding a plastic or metallic texture. The physical constraints of the scalp area include:

[0084] ① Roughness range constraint:

[0085] The skin surface has a rich but not sharp microstructure, resulting in multiple scatterings of light upon incidence. Therefore, a roughness range constraint is set, where the roughness... Exceeding (like This generates a huge gradient penalty to prevent the scalp from developing a mirror-like highlight. , These represent the lower and upper limits of the scalp roughness constraint range, respectively.

[0086] ②Metallicity hard constraint:

[0087] Biological tissues are non-conductors and can directly absorb metallic substances. Initialize it to 0, and apply very strong L2 regularization during the optimization process to keep it in a range close to 0, or freeze the parameter directly so that it does not participate in the optimization.

[0088] ③ Spatial Consistency:

[0089] Skin texture changes are low-frequency; the roughness of skin should not change drastically within a few millimeters of each other. Therefore, a K-nearest neighbor (KNN) graph can be constructed to calculate the roughness difference between each Gaussian cell and its neighbors.

[0090] Introducing a neighborhood smoothing loss function :

[0091]

[0092] in, This indicates a loss of scalp smoothness and roughness. Indicates a collection of scalp hairs. Represents Gaussian unit The spatial neighborhood set, and Representing Gaussian elements respectively and Roughness parameters, This represents the L1 norm (Manhattan distance), used to enhance sparsity constraints.

[0093] The neighborhood smoothing loss function can smooth out noise and fill in material gaps in sparse areas.

[0094] ④ Directional consistency constraint:

[0095] The normal to the skin surface should be a continuously varying surface normal, which can be addressed by introducing a direction consistency loss function. This is used to punish the high-frequency oscillations of the normal lines, so that the changes of the normal lines on the skin surface within the local neighborhood remain smooth, ensuring the spatial consistency of the scalp material.

[0096]

[0097] in, The local normals are calculated from Gaussian elements; The reference normal can be estimated from local geometry or a smooth field. The weighting coefficients represent the loss of directional consistency. It represents the square of the L2 norm (the square of the Euclidean distance).

[0098] Orientation consistency constraints can prevent local orientation instability caused by noise, making the scalp area present a more realistic continuity.

[0099] The final material constraints for the scalp area are expressed as follows:

[0100]

[0101] V. Physical Constraint Modeling of Hair Region

[0102] The difficulty in rendering hair lies in the unique reflections produced by its microscopic fibrous structure. This embodiment utilizes the geometric properties of 3D Gaussian to simulate this microstructure. 3D Gaussian itself is an anisotropic ellipsoid, and in the hair region, it is desired that this ellipsoid be "stretched" with its major axis along the direction of hair growth.

[0103] For the hair area, the following constraints are introduced:

[0104] ① Roughness range constraint: Set the roughness constraint range for the hair This keeps the hair's roughness within that range.

[0105] ②Metallicity Range Constraint: Set the metallicity constraint range for hair. This ensures that the metallicity of the hair remains within a certain range. Although hair is not metallic, in computer graphics approximations, appropriately increasing the metallicity parameter can simulate the high refractive index reflection of hair as a dielectric. Therefore, in this embodiment, the metallicity constraint range is set to [range missing]. This allows the metallicity of the hair to be optimized within this range to enhance shine.

[0106] ③ Orientation Consistency Constraint: Introduce an orientation consistency constraint to the hair region, forcing the principal axis direction of the Gaussian element to be consistent with the tangential direction of the hair strand. Let the rotation matrix of the Gaussian element be... The principal axis vector is The corresponding tangential direction of the hair is Then the directional consistency loss can be expressed as:

[0107]

[0108] Among them, the tangential direction of the hair strand It can be obtained by analyzing the gradient direction of hair texture in the image and back-projecting it into 3D space, or by using the direction of the line connecting the positions of adjacent Gaussian units. This represents the weighting coefficient of the hair region orientation consistency constraint. It means that when and When they are parallel, their dot product is 1 or -1, and the loss of directional consistency is 0; however, the loss is greatest when they are perpendicular.

[0109] By using directional consistency constraints, the Gaussian volume can be forced into long, thin "hair segments," so that during rendering, the specular reflection spot will appear as stripes stretched along the direction of the hair, presenting the correct anisotropic visual effect.

[0110] ④ Roughness Lower Limit Constraint: Although hair has microscopic roughness, its overall shine is strong. Set a roughness lower limit. (e.g., 0.15) and upper limit (e.g., 0.4). Set a lower roughness penalty loss. To prevent the model from pushing the roughness to 0 in order to fit certain noisy pixels (producing specular noise).

[0111]

[0112] in, As a penalty weight, The preset lower threshold for hair roughness. This is the current roughness value.

[0113] The final material constraints for the hair area are expressed as follows:

[0114]

[0115] VI. Boundary Transition Mechanism

[0116] The hairline is the area most prone to revealing flaws in reconstruction; direct spatial segmentation leads to "hard edges." This embodiment proposes a boundary transition mechanism that includes either a strong geometric guidance strategy or a weak transition strategy, as described below:

[0117] In the set Inside, each Gaussian unit It has two weights: scalp weight. and hair weight (Hair composition), and meets the requirements .

[0118] Calculate boundary transition material parameters ;

[0119]

[0120] in, This indicates the material parameters after mixing. Indicates the assumption of Gaussian unit Target material parameters when it belongs to the scalp Indicates the assumption of Gaussian unit The target material parameters when it belongs to hair.

[0121] A weak transition strategy refers to setting weights as learnable parameters in scenarios involving automatic segmentation or significant noise in hair masks, allowing the softness or hardness of the boundaries to be automatically adjusted during optimization. Specifically, under this strategy, the weights... and These parameters are set as learnable parameters and participate in gradient backpropagation as optimization variables. They are iteratively updated using the backpropagation algorithm based on the gradient of the total loss function. The specific calculation method is as follows:

[0122] Will Initialized to 0.5 as an optimizable parameter. During optimization, the network will automatically favor material combinations that better interpret the color of the current pixel. Meanwhile, the boundary transition material parameters... The calculation formula becomes:

[0123]

[0124] Constrained by the scalp, Constrained by hair. After mixing. They retain their respective physical characteristics while achieving a smooth transition in space.

[0125] Then, stability is maintained only through weaker fitting terms and weighted smoothing terms:

[0126]

[0127] in, For boundary transition constraint loss, The weighting coefficient for the material parameter fit item. For the first The current actual material parameters of each Gaussian element The parameters of the mixed target material are calculated based on the weights. These are the weighting coefficients of the weighted smoothing term. For the first The spatial neighborhood set of Gaussian units (obtained by taking the K nearest neighbors). For the first j Gaussian unit scalp weights.

[0128] The loss The weight field is encouraged to change smoothly in the local neighborhood, but the material parameters themselves are not directly subjected to high-frequency smoothing, thus ensuring that material details are preserved and avoiding discontinuity problems caused by noise.

[0129] The strong geometry-guided strategy refers to linearly interpolating the material weights of the scalp and hair based on the Euclidean distance from the Gaussian unit to the boundary when the hair mask quality is high and the reliability is high. Specifically, the weights are calculated as follows:

[0130]

[0131]

[0132] in, Represents Gaussian unit Euclidean distance to the boundary The preset transition band width, This represents the truncation function. At this point, the boundary transition material parameters are determined by... It can be calculated directly without setting boundary transition constraints for the loss. .

[0133] VII. Differentiable Rendering and Joint Optimization Based on PBR

[0134] A differentiable rendering pipeline is constructed, and the Cook-Torrance microsurface BRDF model is used to replace the traditional spherical harmonic function to jointly optimize scene parameters.

[0135] like Figure 5 As shown, the differentiable rendering pipeline includes:

[0136] ① Input the scene parameters and camera parameters of the 3D Gaussian model, where the scene parameters include position and PBR attributes.

[0137] ② Rasterization: Projecting 3D Gaussian onto screen space. This embodiment improves upon the original 3DGS rasterizer, in addition to accumulating color... It also accumulates material properties, as shown in the following formula:

[0138]

[0139] in, This represents the set of indices of all Gaussian units that intersect with the current pixel ray, sorted by depth. This represents the albedo at the pixel level. Similarly, the pixel-level normal can be obtained. Roughness and metallicity (G-Buffer).

[0140] Setting material prior loss :

[0141]

[0142] In the formula, , , These represent the pixel-level albedo, roughness, and metallicity obtained through rasterization, respectively. , , These are the pixel values ​​of the corresponding material prior images predicted by the pre-trained network. This represents the L1 norm (sum of absolute errors), used to calculate the difference between the predicted value and the rasterized value.

[0143] ③ Ambient Lighting: To resolve the ambiguity between lighting and materials, we assume that ambient light is far-field lighting and represent it using an HDR panoramic image. This embodiment uses the Split-Sum method to approximate the ambient light integral, separating the diffuse integral and specular integral in the rendering equation for calculation.

[0144] Diffuse reflection portion: Pre-calculated diffuse irradiance map, obtained by low-frequency convolution of the environment map;

[0145] Specular reflection component: Pre-calculated pre-filtered specular reflection map and environment BRDF lookup table (BRDF LUT). During rendering, the specular reflection components are quickly reconstructed by using the roughness index pre-filtered map and combining it with the view angle index BRDF LUT.

[0146] ④ Shading: Shading calculations are performed in screen space based on the generated G-Buffer. This embodiment uses the Cook-Torrance microsurface BRDF model for lighting calculations, which divides surface reflection into diffuse and specular reflection.

[0147] The specular reflection component is obtained by jointly calculating the normal distribution function (D), the geometric occlusion function (G), and the Fresnel term (F), as detailed below:

[0148]

[0149] In the formula, The direction of the incident light. Direction of sight It is a radius vector (half-vector). This refers to the macroscopic surface normal. The three physical meanings are as follows:

[0150] Normal distribution function (D): describes the statistical distribution probability of the normals of a micro-surface (usually using the GGX distribution), and determines the shape and extent of the specular highlights;

[0151] Geometric Shadowing Function (G): Describes the self-occlusion and masking effects between micro-surfaces (usually using the Smith shadowing function), and solves the problem of excessive illumination on rough surfaces at the swish angle;

[0152] Fresnel term (F): describes the physical phenomenon of reflectivity changing with the observation angle, used to simulate the strong reflective properties of non-metals at edges.

[0153] Final rendered output radiance for:

[0154]

[0155] in, The diffuse reflectance coefficient (depends on metallicity) ), Base color (depending on albedo) ), For the BRDF specular reflection term based on the Cook-Torrance model, Along the direction of incident light The ambient light radiance. Combining the aforementioned Split-Sum approximation method, the above integral is decomposed and calculated, thereby achieving efficient physically-based rendering.

[0156] ⑤ Loss function system:

[0157] Image reconstruction loss L1 Loss + SSIM Loss. This is the main loss driving the optimization.

[0158] Material prior loss : Provides physically reasonable "soft constraints" for PBR parameter optimization.

[0159] Dynamic weighting strategy: , for The weighting coefficients, The initial weight values ​​for the material prior loss are: The hyperparameter (decay constant) is used to control the weight decay rate. This represents the current number of training iterations. At the start of training (iter=0), the weights are large, forcing the model to learn the RGBX prior and relying on the RGBX prior to quickly converge the geometry; as training progresses, the weights decay exponentially, allowing the model to freely fit high-frequency geometric details of multi-view images.

[0160] Physical constraint loss: Material constraint loss in the scalp region Material constraint loss in the hair area Boundary transition constraint loss .

[0161] The total loss function is expressed as:

[0162]

[0163] ⑥ Based on the constructed total loss function, the position, rotation, scaling and PBR material parameters of the Gaussian unit are updated simultaneously using the backpropagation algorithm and the gradient descent method, and finally a high-quality 3D human head model is obtained.

[0164] Optimization strategy:

[0165] Phase 1 (Warm-up): Optimize only the geometric position and opacity The material parameters are frozen to prior values. A rough geometry is then established.

[0166] Phase Two (Joint Optimization): Unlock all parameters and enable PBR rendering. Enable all physical constraint losses.

[0167] Phase Three (Refinement): Gradually Reduce Weighting is used to refine high-frequency details.

[0168] Example 2

[0169] This embodiment provides a 3D head reconstruction system based on semantic separation and PBR material prior, such as Figure 6 As shown, the system includes a processor, input devices, output devices, and memory, which are interconnected.

[0170] The memory is used to store computer programs, which include program instructions. The processor is configured to call the program instructions to execute the three-dimensional head reconstruction method based on semantic separation and PBR material prior as described in Example 1.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for 3D head reconstruction based on semantic separation and PBR material prior, characterized in that, The method includes: Acquire multi-view color images, depth maps, and camera pose information of the object to be reconstructed; generate scalp region masks and hair region masks for each frame of the image using an image segmentation model; A color image is input into a pre-trained deep learning model to obtain a two-dimensional material prior map containing albedo, roughness, and metallicity. A 3D Gaussian scene model is established as the expression carrier of geometry and appearance. The position and physical material parameters of the 3D Gaussian unit are initialized based on the depth map and material prior map, and the Gaussian unit is mapped to the scalp set and hair set according to the generated mask. Physical constraint models were constructed for the scalp region and the hair region, respectively. Establish a boundary transition mechanism between the scalp region and the hair region, calculate the blending weights based on geometric distance or learnable parameters within the transition region, and perform smooth interpolation on the material parameters based on the blending weights; A differentiable rendering pipeline is constructed to jointly optimize and adjust the geometric parameters and PBR material parameters of the 3D Gaussian scene model.

2. The method according to claim 1, characterized in that, The deep learning model includes an encoder for extracting image features and a decoder for predicting physical properties. The decoder outputs pixel-level albedo estimates. Roughness estimate With metallicity estimate This yields a material prior map that includes albedo, roughness, and metallicity.

3. The method according to claim 1, characterized in that, A 3D Gaussian scene model is established, and the position and physical material parameters of the 3D Gaussian cells are initialized based on the depth map and material prior map, including: Using 3DGS as the basic 3D model, for each Gaussian element... Set its color attribute Replace with PBR attribute tuples: In the formula, This is an estimate of the albedo. This is the roughness estimate. This is an estimate of the metallicity. The direction of the normal; The three-dimensional center position, Let covariance matrix be the variance matrix. Opacity; Initialize using sparse point clouds generated by SfM or point clouds projected from depth maps; initialize each 3D point Projecting back onto a 2D plane from all visible viewpoints, indexing the generated material prior map, and calculating the mean as a Gaussian element. Initial values ​​of material properties: in, This represents the initial value of albedo. This represents the initial roughness value. Indicates the initial value of metallicity; Indicates that the first can be observed The set of all camera viewpoints in a Gaussian unit. Indicates the first The reflective prior image predicted by a deep learning model from a single perspective. Indicates the first Roughness prior maps predicted by deep learning models from various perspectives Indicates the first Metallicity prior maps predicted by deep learning models from various perspectives. This represents the projection function that projects a point in three-dimensional space onto a two-dimensional image plane.

4. The method according to claim 3, characterized in that, Based on the generated scalp region mask and hair area mask Divide all Gaussian elements; The mapping rule for Gaussian elements is as follows: for Gaussian elements... The statistics show that its projection falls on all viewpoints. and The percentage of pixels within; if it falls within The probability of falling within is higher than falling into If the probability within the range is 0.3 or higher, then it is a Gaussian unit. Classified to scalp collection If it falls The probability of falling within is higher than falling into If the probability within the range is 0.3 or higher, then it is a Gaussian unit. Divided into hair collection For those who fall and fall Gaussian units with a probability difference of less than 0.3 within a given range are designated as transition zone units and assigned to a set. .

5. The method according to claim 1, characterized in that, Establish a physical constraint model for the scalp region, including: Roughness interval constraint, constraining scalp roughness within an interval When the scalp roughness exceeds the constraint range, a gradient penalty is generated to prevent the scalp from having specular highlights. , These are the lower and upper limits of the scalp roughness constraint range, respectively; Hard constraint on metallicity, keeping scalp metallicity close to 0, or freezing scalp metallicity; Neighborhood smoothing constraint, setting neighborhood smoothing loss function : In the formula, This indicates a loss of scalp smoothness and roughness. Indicates a collection of scalp hairs. Represents Gaussian unit The spatial neighborhood set, and Representing Gaussian elements respectively and Roughness parameters, Represents the L1 norm; Directional consistency constraint, directional consistency loss function : In the formula, The local normals are calculated from Gaussian elements; For reference normal; The weighting coefficients represent the loss of directional consistency. This represents the square of the L2 norm.

6. The method according to claim 1, characterized in that, Establish a physical constraint model for the hair region, including: Roughness range constraint: Set the roughness constraint range for hair. This keeps the hair roughness within a controlled range. , These are the lower and upper limits of the hair roughness constraint range, respectively; Metallicity range constraint: Set the metallicity constraint range for hair. This keeps the metallicity of the hair within a controlled range. , These are the lower and upper limits of the hair metallicity constraint range, respectively; Directional consistency constraints, setting directional consistency losses Force the principal axis direction of the Gaussian element to be consistent with the tangential direction of the hair: In the formula, For Gaussian element rotation matrix The principal axis vector, The direction of the hair strand tangentially Collect the hair. The weighting coefficients represent the directional consistency constraints of the hair region; Roughness lower limit constraint, setting a roughness lower limit penalty loss : In the formula, As a penalty weight, This is the current roughness value.

7. The method according to claim 1, characterized in that, Establishing a boundary transition mechanism between the scalp and hair regions involves, within the transition region where the scalp and hair meet, each Gaussian unit... All have scalp weight and hair weight The scalp and hair weights are adjusted using either a strong geometry guidance strategy or a weak transition strategy, and the blending material parameters are calculated based on the adjusted weights. In the formula, Indicates the parameters of the mixed material. Indicates the assumption of Gaussian unit Target material parameters when it belongs to the scalp Indicates the assumption of Gaussian unit The target material parameters when it pertains to hair. Albedo, For roughness, Metallicity.

8. The method according to claim 7, characterized in that, When the hair mask contains noise, a weak transition strategy is used to adjust the scalp weight and hair weight, and the scalp weight is adjusted accordingly. and hair weight Set as learnable parameters, freely adjustable during optimization, and set boundary transition constraint losses. : Simultaneously mixing material parameters The calculation becomes: ; The weighting coefficient for the material parameter fit item. For the first The current actual material parameters of each Gaussian element These are the weighting coefficients for the weighted smoothing term; For the first j Gaussian unit scalp weight; A set of transition zone units; For the first The spatial neighborhood set of Gaussian units; When the hair mask quality is high, a strong geometry-guided strategy is used to adjust the scalp and hair weights, and the material weights of the scalp and hair are linearly interpolated based on the Euclidean distance from the Gaussian unit to the boundary. In the formula, Represents Gaussian unit Euclidean distance to the boundary The preset transition band width, This represents the truncation function.

9. The method according to claim 1, characterized in that, Constructing a differentiable rendering pipeline to jointly optimize and adjust the geometric parameters and PBR material parameters of a 3D Gaussian scene model includes: Input the 3D Gaussian model scene parameters and camera parameters, where the scene parameters include position and PBR material parameters: Differentiable rasterization is performed to project the 3D Gaussian model onto the screen space; a priori material loss is set during the rasterization process. : In the formula, , , These represent the pixel-level albedo, roughness, and metallicity obtained through rasterization, respectively. , , These are the pixel values ​​of the corresponding material prior images predicted by the pre-trained network; This represents the L1 norm, used to calculate the difference between the predicted and rendered values; Ambient lighting is calculated, assuming it is far-field lighting, represented by an HDR panoramic image. The Split-Sum method is used to approximate the ambient light integral, separating the diffuse integral and specular integral in the rendering equation. For the diffuse component, a diffuse irradiance map is pre-calculated by performing low-frequency convolution on the environment map. For the specular component, a pre-filtered specular map and an environment BRDF lookup table are pre-calculated. During rendering, the specular component is quickly reconstructed by indexing the pre-filtered map with roughness and combining it with the view angle indexed BRDF lookup table. Deferred shading in screen space is performed using the Cook-Torrance BRDF model; Construct the loss function: In the formula, For the total loss function, For image reconstruction loss, For material constraint loss in the scalp area, For material constraint loss in the hair area, For boundary transition constraint loss; For dynamic weighting coefficients: , The initial weight values ​​for the material prior loss are: To control the hyperparameter of the weight decay rate, This represents the current number of training iterations. Based on the constructed total loss function, the position, rotation, scaling, and PBR material parameters of the Gaussian unit are updated simultaneously using the backpropagation algorithm and the gradient descent method, ultimately obtaining a high-quality 3D human head model.

10. A three-dimensional head reconstruction system based on semantic separation and PBR material prior, characterized in that, The system includes interconnected processors, input devices, output devices, and a memory; the memory is used to store a computer program, the computer program including program instructions; the processor is configured to invoke the program instructions to execute the method of any one of claims 1 to 9.

Images