An implicit surface human modeling method based on weight mask strategy

Abstract

The application discloses a kind of implicit surface human modeling methods based on weight mask strategy, first based on 2D human photo generation implicit expression, then the implicit expression of the space point generated is input into renderer, render human prediction image and predicted mask.Optimize the surface of human special part, according to loss function supervision and train model;Finally, according to the model obtained, the generation of human surface is carried out.The method of the application introduces adjustable weight mask in different areas of the model, so that the details of each part of the human body are more accurately described, especially in areas with large curvature changes, such as the face, joints, etc., can better capture the small contour and shape changes.Through the weight mask strategy, more detailed optimization can be performed on the complex areas of the model, thereby avoiding distortion, distortion or stretching phenomenon when the curved surface is converted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer image processing and three-dimensional reconstruction, and particularly relates to a model combining a weight mask implicit surface, using computer vision technology and deep learning algorithms to generate a high-precision three-dimensional model of a human body. BACKGROUND

[0002] In recent years, neural implicit representations have rapidly developed in three-dimensional modeling and image synthesis. Compared with classic discrete counterparts such as meshes, point clouds, and voxels, neural implicit representations use neural networks to model scenes, which are continuous in space and show higher fidelity and flexibility. As the most popular implicit representation in neural rendering, Neural Radiance Fields (NeRF) has shown amazing results in various tasks since its first introduction. The original NeRF is over-fitted on a static scene by design, so it cannot model time-varying content.

[0003] The volume rendering technique in NeRF is a technique that generates an image by integrating the color and opacity of all sampling points on the light path. However, when applied to human body models, due to the complexity and diversity of human surface details, it is difficult to accurately capture every subtle surface feature by relying solely on volume rendering, because during the volume rendering process, the light accumulates a large amount of information as it passes through the object, and this accumulation of information can cause the surface details to be blurred.

[0004] Implicit surface model (I implicit surface model) is a technology widely used in computer graphics, computer vision, and machine learning. It defines a scalar field to represent the shape or object in three-dimensional space, where the value of the scalar field can be used to distinguish the space inside and outside the object. Specifically, for a given three-dimensional point, if the point is inside the object, the value of the scalar field at this point is negative; if it is outside the object, the value is positive; and when the value is zero, the point is exactly on the surface of the object. This representation method not only accurately captures complex geometric structures, but also facilitates various geometric operations and transformations. In recent years, with the rise of deep learning technology, implicit surface models have been further developed. By combining neural networks with implicit functions, researchers have developed a variety of new models, such as NeRF (Neural Radiance Fields) and DeepSDF (Deep Signed Distance Function), which can efficiently model and render three-dimensional scenes while maintaining high precision. In addition, these technologies have been widely applied in virtual reality, augmented reality, and autonomous driving, etc. many frontier fields, greatly promoting the progress and development of related technologies. SUMMARY

[0005] In order to solve the problem of surface blurring, the present application adopts a weight mask strategy to express the implicit surface model, which describes the shape of an object by defining a continuous scalar field, where the change of field value indicates the spatial conversion inside and outside the object, and can provide a clear surface definition, i.e. zero-level set, which is a set of points that make the field function value zero, which corresponds to the surface of the object. The surface S of an object can be represented by the zero-level set of its SDF as follows:

[0006]

[0007] In the rendering process, the intersection of the light and the object surface can be accurately determined by solving the zero-level set, thereby avoiding the surface blurring problem caused by inaccurate sampling.

[0008] An implicit surface human modeling method based on a weight mask strategy, comprising the following steps:

[0009] Step one: generating implicit expression based on 2D human photos.

[0010] Step two: inputting the generated implicit expression of the spatial points into a renderer to render a human prediction image and a prediction mask.

[0011] Step three: optimizing the surface of special parts of the human body, supervising and training the model according to the loss function;

[0012] Step four: generating the surface of the human body according to the obtained model.

[0013] Further, the specific method of step one is as follows:

[0014] Given N human body images with resolution (W, H) and corresponding camera intrinsics, extrinsics and human object mask The goal is to reconstruct the fine human surface.

[0015] For a given input human image on the sampling point t, first project it into the 3D space, get the 3D point on the ray emitted from the pixel as {p(t) = o + tv | t ≥ 0}, wherein o is the center of the camera, and v is the unit direction vector of the ray. Then, the sign distance value f(p(t)), the validity probability V(p(t)) and the RGB value c(p(t)) of the space point p(t) are respectively predicted by three fully connected neural networks (SDF-Net, Validity-Net, Color-Net), that is, the implicit expression (including three values) of the space point p(t) is obtained. Specifically, the three fully connected neural networks include:

[0016] SDF-Net: mapping function to represent the signed distance field.

[0017] Validity-Net: mapping function to represent the validity probability.

[0018] Color-Net: mapping function to predict the color of each point in the 3D space.

[0019] Further, the specific method of step two is as follows:

[0020] The three implicit expressions of the space point are transmitted to the renderer to render the human prediction image (I pred (o,v)) and the prediction mask (M pred (o,v)) from the implicit expression.

[0021] The prediction mask M pred (o,v) is inferred from the rendering weight w(p(t)) of each space point, and the human prediction image I pred (o,v) is calculated based on the RGB value c(p(t)) and the rendering weight w(p(t)):

[0022]

[0023] According to the above equation, a key problem in the rendering process is to find a suitable weight function w(p(t)). This task is divided into two steps:

[0024] 1) Construct a probability density function to estimate the volume density based on the sign distance value (f(p(t))).

[0025] 2) Estimate a suitable weight function w(p(t)) based on the volume density and the validity probability.

[0026] The construction of the probability density function is as follows:

[0027] The signed distance value is modified by flipping its sign in regions where the signed distance value increases along the camera ray. The probability density function is defined as:

[0028]

[0029] where v is the unit direction vector of the ray and n is the gradient of the signed distance function. The logistic density distribution function is denoted as and is expressed as follows:

[0030] φ s (x) = se -sx / (1 + e -sx ) 2

[0031] The definition of the probability density function ensures the same rendering behavior when a ray enters the surface from any direction.

[0032] The estimation weight function is specified as follows:

[0033] The weight function w(p(t)) has two properties: unbiased and occlusion-aware. The unbiased rendering weight function w(p(t)) is defined using the following first equation, and the occlusion-aware weight function is defined according to the opacity density p(t) in the following second equation.

[0034]

[0035] Solving the above two equations, the opacity p(t) can be obtained:

[0036]

[0037] To obtain the human body prediction image, the classical discretization scheme in differentiable rendering is adopted for the opacity and weight functions. For a set of spatial points {p i = o + t i v | i = 1,... m, t i i+1} along the ray, the rendered human body prediction image can be expressed as:

[0038]

[0039] where c i i.e., c(p(t i )) is the estimated color of the i-th sampling point; a i is the discrete opacity value when rendering the implicit representation of the signed distance value, which can be written as:

[0040]

[0041] ​Thus, a weight function w(p(t)) is established, which is unbiased and occlusion-aware, and supports the use of signed distance function (SDF) to represent the surface model of the rendered human body.

[0042] Further, the specific method of step three is as follows:

[0043] In order to render the special surface of the human body model, the validity probability (V(p(t i ))) of the 3D sampling point is multiplied by its opacity value in the rendering process. Then the discrete opacity value of the i-th sampling point is β i :

[0044] β i = a i · V(p(t i ))

[0045] Therefore, the final rendered pixel color of the surface of the human body model can be represented as:

[0046]

[0047] In the training process of the human body surface, five loss functions are used to supervise the rendered human body prediction image (I pred ) and the prediction mask (M pred ). The first three loss functions are RGB Loss, Mask Loss and Eikonal Loss. They are defined as:

[0048]

[0049] where BCE is the binary cross-entropy. i and j represent the center o and the unit direction vector v of the ray of the camera in the current training round. I gt represents the original image of the human body, and M gt represents the original object mask of the human body.

[0050] The fourth is the rendering probability loss function: in the physical world, the existence of the human body surface is binary (non-existent / non-existent). Therefore, the validity probability of the 3D sampling point is 0 (no surface) or 1 (surface). Therefore, the binary cross-entropy of the validity probability V(p(t)) is used as an additional regularization:

[0051]

[0052] The fifth is the sparsity loss function: for real-world human objects with open structures, the human body surface is sparsely distributed in 3D space. In order to prevent the network from predicting redundant human body surfaces, a sparsity loss function is introduced to promote the formation of open human body surfaces:

[0053]

[0054] The predicted human body image and the predicted mask are supervised according to the above-mentioned five loss functions in the training process, and the model is obtained.

[0055] Further, the specific method of step four is as follows:

[0056] A 2D human body photo is input into the trained model, the model generates a predicted human body image and a predicted mask for the input picture, the validity probability V(p(t)) of the symbolic distance function (SDF) is verified according to the predicted human body image and the predicted mask, the symbolic distance value of the 3D sampling point with V(p(t)) less than 0.5 is set to NAN, and the processed symbolic distance value is fed to the Marching Cubes algorithm to generate a final human body surface model.

[0057] The present application has the following advantages:

[0058] The present application proposes an implicit surface human body modeling method based on a weight mask strategy, aiming to improve the modeling accuracy and visual expressiveness of human body surface shape. Compared with traditional modeling techniques, this method introduces adjustable weight masks in different areas of the model, making the depiction of details of each part of the human body more accurate, especially in areas with large curvature changes, such as the face, joints, etc. This mask strategy applies different weight coefficients in different areas, thus adaptively smoothing the surface when building the model, and effectively reducing the appearance of edges, abruptness and noise interference in the traditional modeling process, making the generated surface transition more natural and smooth.

[0059] In addition, this method not only achieves higher realism in visual performance, but also significantly improves the accuracy and stability of the human body model. Through the weight mask strategy, the present application method can optimize the complex areas of the model more meticulously, thus avoiding distortion, distortion or stretching phenomena when the surface is converted. At the same time, the present application has flexibility in weight adjustment, which can control the detail level of different parts according to actual needs, so that the model is not only suitable for general animation production or virtual reality scenes, but also can meet the application scenarios of high-precision requirements such as medical simulation, motion capture and human-computer interaction. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 Predicted picture example for validity probability of different human body parts.

[0061] Figure 2 Process diagram of human body surface validity processing for the embodiment of the present application.

[0062] Figure 3 Experimental comparison results for the embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0064] An implicit surface human body modeling method based on a weighted mask strategy comprises the following steps:

[0065] Step 1: Generate implicit representation based on 2D human body photos:

[0066] Given N human images (gt is the abbreviation of ground truth), resolution (W, H) and corresponding camera intrinsics, extrinsics and human object masks The goal is to recreate a detailed human surface.

[0067] For a given sampling point t on an input human image, it is first projected into 3D space, and the 3D point on the ray emitted from the pixel is {p(t)=o+tv|t≥0}, where o is the center of the camera and v is the unit direction vector of the ray. Then, three fully connected neural networks (SDF-Net, Validity-Net, Color-Net) are used to predict the signed distance value f(p(t)), validity probability V(p(t)) and RGB value c(p(t)) of the spatial point p(t), respectively, to obtain the implicit expression of the spatial point p(t) (including three values). Specifically, the three fully connected neural networks include:

[0068] SDF-Net: Mapping Function to represent the signed distance field.

[0069] Validity-Net: Mapping Function To express the probability of effectiveness.

[0070] Color-Net: Mapping Function To predict the color of each point in 3D space.

[0071] Step 2: Input the generated implicit expression of the spatial points into the renderer to render the human body prediction image and prediction mask.

[0072] The three implicit representations of the spatial points are passed to the renderer to render the human body prediction image (I pred (o,v)) and the predicted mask (M pred (o,v)).

[0073] Prediction mask M pred(o, v) is inferred from the rendering weight w(p(t)) at each spatial point, while the human body is predicted from the image I pred (o, v) is computed based on the RGB value c(p(t)) and the rendering weight w(p(t)):

[0074]

[0075] According to the above equation, a key problem in the rendering process is to find a suitable weight function w(p(t)). This task is divided into two steps:

[0076] 1) Construct a probability density function to estimate the volume density based on the signed distance value (f(p(t)));

[0077] 2) Estimate a suitable weight function w(p(t)) based on the volume density and the probability of validity.

[0078] The construction of the probability density function is as follows:

[0079] The signed distance value is modified by flipping its sign in the region where the signed distance value increases along the camera ray. The probability density function is defined as:

[0080]

[0081] where v is the unit direction vector of the ray, and n is the gradient of the signed distance function. The logical density distribution function is denoted as φ(x), and the formula is as follows:

[0082] φ s (x) = se -sx / (1 + e -sx )2 2

[0083] The definition of the probability density function ensures the same rendering behavior when the light ray enters the surface from any direction.

[0084] The estimation of the weight function is as follows:

[0085] The weight function w(p(t)) has two properties: unbiased and occlusion-aware. The first formula is used to define the unbiased rendering weight function w(p(t)), and the second formula is used to define the occlusion-aware weight function based on the opacity density ρ(t).

[0086]

[0087] Solving the above two equations, the opacity ρ(t) can be obtained:

[0088]

[0089] In order to obtain the predicted human image, the classical discretization scheme in differentiable volume rendering is adopted for the opacity and weight functions. For a set of spatial points {p i =o+t i v|i=1,...m,t i <t i+1}, the rendered human body prediction image can be expressed as:

[0090]

[0091] where c i That is c(p(t i )), is the estimated color of the i-th sampling point; α i is the implicit expression of the signed distance value for the discrete opacity value when rendering, which can be written as:

[0092]

[0093] This results in an unbiased and occlusion-aware weight function w(p(t)) that enables rendering of the human body surface model using a signed distance function (SDF) representation.

[0094] Step 3: Optimize the surface of special parts of the human body, supervise and train the model according to the loss function;

[0095] In order to render some special surfaces of the human body model, such as closed surfaces and open surfaces, the validity probability of the 3D sampling points (V(p(t i ))) are multiplied by their opacity values ​​during rendering, e.g. Figure 1 (To render a human body prediction image using validity probability, Object1 and Object2 represent human body parts from different perspectives).

[0096] Then the discrete opacity value of the i-th sampling point is β i for:

[0097] β i =α i ·V(p(t i ))

[0098] So the final rendered pixel color of the human body model surface can be expressed as:

[0099]

[0100] In the process of training the human body surface, five loss functions are used to supervise the rendered human body prediction image (I pred ) and the prediction mask (M pred). The first three loss functions are RGB Loss, Mask Loss, and Eikonal Loss. They are defined as:

[0101]

[0102] where BCE is binary cross-entropy. i and j represent the center o and the unit direction vector v of the ray of the camera in the current training round, I gt represents the original image of the human body, M gt represents the original object mask of the human body.

[0103] The fourth is the rendering probability loss function: in the physical world, the existence of the human body surface is binary (non-existent / non-existent). Therefore, the effectiveness probability of the 3D sampling point is 0 (no surface) or 1 (surface). Therefore, the binary cross-entropy of the effectiveness probability V(p(t)) is taken as an additional regularization:

[0104]

[0105] The fifth is the sparse loss function: for real-world human objects with open structures, the human body surface is sparsely distributed in 3D space. In order to prevent the network from predicting redundant human body surfaces, a sparse loss function is introduced to promote the formation of open human body surfaces:

[0106]

[0107] During training, the predicted human prediction image and the predicted mask are supervised according to the above five loss functions to obtain the model.

[0108] Step four: generate the human body surface according to the obtained model.

[0109] Input a 2D human photo into the trained model, and the model generates a predicted human image and a predicted mask for the input picture. According to the predicted human image and the predicted mask, the validity probability V(p(t)) of the signed distance function (SDF) is verified, and the signed distance value of the 3D sampling point with V(p(t)) less than 0.5 is set to NAN. The human body surface effectiveness processing process is as shown in Figure 2 , and the processed signed distance value is fed into the Marching Cubes algorithm to produce the final human body surface model.

[0110] Figure 2(a) is the signed distance function (SDF); (b) is the validity probability function V; (c) is the watertight surface extracted from (a) SDF; (d) is the open surface extracted from (a) SDF and (b) validity probability. In our mesh extraction process, we set the SDF of 3D query points with low validity (here V < 0.5) to NAN and extract the open surface using the Marching Cubes algorithm.

[0111] As shown in Figure 3 , where Ground Truth is the original human body model. OURS is the model modeled by the method of the application. Ani-NeRF is the model modeled by the Ani-NeRF method. It can be seen that in the method of the application, the human body model is smoother and closer to the original human body model.

Claims

1. An implicit surface human body modeling method based on weight mask strategy, characterized in that: The steps are as follows: Step 1: Generate implicit representation based on 2D human body photos; Step 2: Input the generated implicit expression of the spatial point into the renderer to render the human body prediction image and prediction mask; the specific method is as follows: Pass the three implicit representations of the spatial points into the renderer to render the human body prediction image from the implicit representation and the predicted mask ; Prediction Mask The rendering weight of each spatial point Inferred from the image, while the human body is predicted Based on RGB values and render weights Calculation yields: According to the above equation, a key issue in the rendering process is to find a suitable weight function; this task can be divided into two steps: 1) Construct a probability density function based on the signed distance value Estimate bulk density; 2) Obtaining a suitable weight function based on volume density and effectiveness probability estimation; The probability density function is constructed as follows: The signed distance value is modified by flipping its sign in the region where the signed distance value increases along the camera ray; the probability density function is defined as: in is the unit direction vector of the ray, is the gradient of the signed distance function; Represents the logical density distribution function, and the formula is as follows: The estimated weight function is as follows: The weight function has two properties: unbiased and occlusion-aware; The unbiased rendering weight function is defined using the following first formula, and the occlusion-aware weight function is defined based on the opacity density ρ(t) in the following second formula; Solving the above two equations, we can get the opacity : In order to obtain the predicted human body image, the classical discretization scheme in differentiable volume rendering is used for the opacity and weight functions; for a set of spatial points along the ray , the rendered human body prediction image can be expressed as: in That is , is the estimated color of the i-th sampling point; is the implicit expression of the signed distance value for the discrete opacity value when rendering, which can be written as: This establishes an unbiased and occlusion-aware weight function that supports the use of signed distance function (SDF) to represent the surface model of the rendered human body. Step 3: Optimize the surface of special parts of the human body, supervise and train the model according to the loss function; Step 4: Generate the human body surface based on the obtained model.

2. The implicit surface human body modeling method based on weight mask strategy according to claim 1 is characterized in that: The specific method of step one is as follows: Given N human images , with resolution (W, H) and corresponding camera intrinsics, extrinsics, and human object masks ,The goal is to reconstruct the detailed human body surface; For a given sampling point on an input human image First, project it into 3D space and get the 3D point on the ray emitted from the pixel as ,in is the center of the camera, is the unit direction vector of the ray; then, three fully connected neural networks are used to predict the spatial points respectively The signed distance value of , validity probability and RGB values , that is, the spatial point Specifically, the three fully connected neural networks include: SDF-Net: Mapping Function To represent the signed distance field; Validity-Net: Mapping Function To express the probability of effectiveness; Color-Net: Mapping Function To predict the color of each point in 3D space.

3. The implicit surface human body modeling method based on weight mask strategy according to claim 2 is characterized in that: The specific method of step three is as follows: In order to render the special surface of the human body model, the validity probability of the 3D sampling point is Multiply them by their opacity values ​​during rendering; then the discrete opacity value of the i-th sampling point is for: So the final rendered pixel color of the human body model surface can be expressed as: During the training of the human body surface, five loss functions are used to supervise the rendered human body prediction image and prediction mask; the first three loss functions are RGB Loss, Mask Loss and Eikonal Loss; they are defined as: Where BCE is binary cross entropy; and Represents the center of the camera in the current training round and the unit direction vector of the ray , represents the original image of the human body, The original object mask representing the human body; The fourth is the rendering probability loss function: In the physical world, the existence of the human body surface is binary; therefore, the validity probability of a 3D sampling point is 0 or 1; therefore, the validity probability The binary cross entropy of serves as additional regularization: The fifth is the sparse loss function: For real-world human objects with open structures, the human body surface is sparsely distributed in 3D space; in order to prevent the network from predicting redundant human body surfaces, a sparse loss function is introduced to promote the formation of open human body surfaces: During the training process, the predicted human body images and predicted masks are supervised according to the above five loss functions to obtain the model.

4. The implicit surface human body modeling method based on weight mask strategy according to claim 3 is characterized in that: The specific method of step 4 is as follows: Input a 2D human body photo into the trained model. The model generates a predicted human body image and a predicted mask for the input image. The validity probability is verified by the signed distance function (SDF) based on the predicted human body image and the predicted mask. ,Will The signed distance values ​​of 3D sampling points less than 0.5 are set to NAN, and the processed signed distance values ​​are fed into the Marching Cubes algorithm to generate the final human body surface model.

Images