Implicit surface human body modeling method based on weight mask strategy

Abstract

The invention discloses an implicit surface human body modeling method based on a weight mask strategy, and the method comprises the steps: generating an implicit expression based on a 2D human body picture, inputting the generated implicit expression of a spatial point into a renderer, and rendering a human body prediction image and a prediction mask. Optimizing the surface of the special part of the human body, and supervising and training the model according to a loss function; and finally, generating a human body surface according to the obtained model. According to the method, adjustable weight masks are introduced into different areas of the model, so that details of all parts of the human body are described more accurately, and particularly, tiny contour and shape changes can be better captured in areas such as the face and joints with large curvature changes. Through the weight mask strategy, the complex area of the model can be optimized more meticulously, so that the distortion, distortion or stretching phenomenon during curved surface conversion is avoided.

Description

Technical Field

[0001] The present invention relates to the fields of computer image processing and three-dimensional reconstruction, and in particular to a method for generating a high-precision three-dimensional model of a human body by combining a weighted mask implicit surface model with computer vision technology and a deep learning algorithm. Background Art

[0002] Recent years have seen rapid development of neural implicit representations in 3D modeling and image synthesis. Neural implicit representations model scenes using neural networks, which are spatially continuous and show higher fidelity and flexibility compared to their classical discrete counterparts such as meshes, point clouds, and voxels. As the most popular implicit representation in neural rendering, Neural Radiance Fields (NeRF) has shown amazing results in various tasks since its first introduction. The original NeRF is overfitted on a static scene by design and therefore cannot model time-varying content.

[0003] Volume rendering in NeRF is a technique that generates images by integrating the color and opacity of all sampling points on the light path. However, when applied to human models, due to the complexity and diversity of human surface details, it is difficult to accurately capture every subtle surface feature by relying solely on volume rendering, because during volume rendering, light accumulates a large amount of information when passing through an object, and the accumulation of this information may cause the surface details to become blurred.

[0004] Implicit surface model is a widely used technology in computer graphics, computer vision and machine learning. It represents shapes or objects in three-dimensional space by defining a scalar field, where the value of the scalar field can be used to distinguish the space inside and outside the object. Specifically, for a given three-dimensional point, if the point is inside the object, the value of the scalar field here is negative; if it is outside the object, the value is positive; and when the value is zero, the point is exactly on the surface of the object. This representation method can not only accurately capture complex geometric structures, but also facilitate various geometric operations and transformations. In recent years, with the rise of deep learning technology, implicit surface models have been further developed. By combining neural networks with implicit functions, researchers have developed a variety of new models, such as NeRF (Neural Radiance Fields) and DeepSDF (Deep Signed Distance Function), which can achieve efficient modeling and rendering of three-dimensional scenes while maintaining high accuracy. In addition, these technologies are also widely used in many cutting-edge fields such as virtual reality, augmented reality and autonomous driving, greatly promoting the progress and development of related technologies. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an implicit surface human body modeling method based on a weight mask strategy. In order to solve the problem of blurred surface details, the present invention adopts a weight mask strategy to express the implicit surface model, which describes the shape of the object by defining a continuous scalar field, in which the change of the field value indicates the spatial transformation inside and outside the object, and can provide a clear surface definition, namely the zero-level set, which is a set of points that make the field function value zero, just corresponding to the surface of the object. The surface S of an object can be expressed by the zero-level set of its SDF as:

[0006]

[0007] During the rendering process, the intersection point between the light ray and the object surface can be accurately determined by solving this zero level set, thus avoiding the surface blur problem caused by inaccurate sampling.

[0008] An implicit surface human body modeling method based on a weight mask strategy comprises the following steps:

[0009] Step 1: Generate implicit representation based on 2D human body photos.

[0010] Step 2: Input the generated implicit expression of the spatial points into the renderer to render the human body prediction image and prediction mask.

[0011] Step 3: Optimize the surface of special parts of the human body, supervise and train the model according to the loss function;

[0012] Step 4: Generate the human body surface based on the obtained model.

[0013] Furthermore, the specific method of step one is as follows:

[0014] Given N human images Resolution (W, H) and corresponding camera intrinsics, extrinsics, and human object masks The goal is to recreate a detailed human surface.

[0015] For a given sampling point t on an input human image, it is first projected into 3D space, and the 3D point on the ray emitted from the pixel is {p(t)=o+tv|t≥0}, where o is the center of the camera and v is the unit direction vector of the ray. Then, three fully connected neural networks (SDF-Net, Validity-Net, Color-Net) are used to predict the signed distance value f(p(t)), validity probability V(p(t)) and RGB value c(p(t)) of the spatial point p(t), respectively, to obtain the implicit expression (including three values) of the spatial point p(t). Specifically, the three fully connected neural networks include:

[0016] SDF-Net: Mapping Function to represent the signed distance field.

[0017] Validity-Net: Mapping Function To represent the probability of effectiveness.

[0018] Color-Net: Mapping Function To predict the color of each point in 3D space.

[0019] Furthermore, the specific method of step 2 is as follows:

[0020] The three implicit representations of the spatial points are passed to the renderer to render the human body prediction image (I pred (o,v)) and the predicted mask (M pred (o,v)).

[0021] Prediction mask M pred (o,v) is inferred from the rendering weight w(p(t)) of each spatial point, and the human body prediction image I pred (o,v) is calculated based on the RGB value c(p(t)) and the rendering weight w(p(t)):

[0022]

[0023] According to the above equation, a key issue in the rendering process is to find a suitable weight function w(p(t)). This task is divided into two steps:

[0024] 1) Construct a probability density function to estimate the volume density based on the signed distance value (f(p(t))).

[0025] 2) Obtain a suitable weight function w(p(t)) based on volume density and validity probability estimation.

[0026] The probability density function is constructed as follows:

[0027] The signed distance value is modified by flipping its sign in the region where the signed distance value increases along the camera ray. The probability density function is defined as:

[0028]

[0029] Where v is the unit direction vector of the ray and n is the gradient of the signed distance function. Represents the logical density distribution function, and the formula is as follows:

[0030] φ s (x) = se -sx / (1+e -sx ) 2

[0031] The definition of the probability density function ensures the same rendering behavior when light enters the surface from any direction.

[0032] The estimated weight function is as follows:

[0033] The weight function w(p(t)) has two properties: unbiased and occlusion-aware. The unbiased rendering weight function w(p(t)) is defined using the first formula below, and the occlusion-aware weight function is defined based on the opacity density ρ(t) in the second formula below.

[0034]

[0035] Solving the above two equations, we can obtain the opacity ρ(t):

[0036]

[0037] In order to obtain the predicted human image, the classical discretization scheme in differentiable volume rendering is used for the opacity and weight functions. For a set of spatial points {p i =o+t i v|i=1,...m,t i <t i+1}, the rendered human prediction image can be expressed as:

[0038]

[0039] where c i That is c(p(t i )), is the estimated color of the i-th sampling point; α i is the implicit expression of the signed distance value for the discrete opacity value when rendering, which can be written as:

[0040]

[0041] This results in an unbiased and occlusion-aware weight function w(p(t)) that enables the rendering of a human surface model using a signed distance function (SDF) representation.

[0042] Furthermore, the specific method of step three is as follows:

[0043] In order to render the special surface of the human body model, the validity probability of the 3D sampling point (V(p(t i ))) multiplied by their opacity values ​​during rendering. Then the discrete opacity value of the i-th sampling point is β i for:

[0044] β i =α i ·V(p(t i ))

[0045] So the final rendered pixel color of the human body model surface can be expressed as:

[0046]

[0047] In the process of training the human body surface, five loss functions are used to supervise the rendered human body prediction image (I pred ) and the prediction mask (M pred ). The first three loss functions are RGB Loss, Mask Loss, and Eikonal Loss. They are defined as:

[0048]

[0049] Where BCE is the binary cross entropy. i and j represent the center o of the camera and the unit direction vector v of the ray in the current training round, I gt Represents the original image of the human body, M gt The original object mask representing the human body.

[0050] The fourth is the rendering probability loss function: In the physical world, the existence of the human body surface is binary (absent / absent). Therefore, the validity probability of a 3D sample point is 0 (no surface) or 1 (surface). Therefore, the binary cross entropy of the validity probability V(p(t)) is used as an additional regularization:

[0051]

[0052] The fifth is the sparse loss function: For real-world human objects with open structures, the human body surface is sparsely distributed in 3D space. In order to prevent the network from predicting redundant human body surfaces, a sparse loss function is introduced to promote the formation of open human body surfaces:

[0053]

[0054] During the training process, the predicted human body prediction image and prediction mask are supervised according to the above five loss functions to obtain the model.

[0055] Furthermore, the specific method of step 4 is as follows:

[0056] Input a 2D human body photo into the trained model. The model generates a predicted human body image and a predicted mask for the input image. The validity probability V(p(t)) is verified based on the signed distance function (SDF) of the predicted human body image and the predicted mask. The signed distance value of the 3D sampling point with V(p(t)) less than 0.5 is set to NAN, and the processed signed distance value is fed to the Marching Cubes algorithm to generate the final human body surface model.

[0057] The beneficial effects of the present invention are as follows:

[0058] The present invention proposes an implicit surface human body modeling method based on a weight masking strategy, aiming to improve the modeling accuracy and visual expression of the human body surface shape. Compared with traditional modeling techniques, this method introduces adjustable weight masks in different areas of the model, making the depiction of details of various parts of the human body more accurate, especially in areas with large curvature changes, such as the face, joints and other areas, which can better capture subtle contours and shape changes. This masking strategy applies different weight coefficients in different areas, so as to adaptively smooth the surface when building the model, and effectively reduce the edges, abruptness and noise interference that appear in the traditional modeling process, so that the generated surface transition is more natural and smooth.

[0059] In addition, the method not only achieves higher fidelity in visual performance, but also significantly improves the accuracy and stability of the human body model. Through the weight mask strategy, the method of the present invention can optimize the complex areas of the model in a more detailed manner, thereby avoiding distortion, distortion or stretching during surface conversion. At the same time, the present invention is flexible in weight adjustment and can control the level of detail of different parts according to actual needs, so that the model is not only suitable for general animation production or virtual reality scenes, but also can meet the application scenarios with high precision requirements such as medical simulation, motion capture and human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Example images of predicted images with validity probabilities rendered for different human body parts.

[0061] Figure 2 This is a diagram of the human body surface effectiveness processing process according to an embodiment of the present invention.

[0062] Figure 3 The following are experimental comparison results of the embodiments of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0064] An implicit surface human body modeling method based on a weight mask strategy comprises the following steps:

[0065] Step 1: Generate implicit representation based on 2D human body photos:

[0066] Given N human images (gt is the abbreviation of ground truth), the resolution is (W, H) and the corresponding camera intrinsics, extrinsics and human object masks The goal is to recreate a detailed human surface.

[0067] For a given sampling point t on an input human image, it is first projected into 3D space, and the 3D point on the ray emitted from the pixel is {p(t)=o+tv|t≥0}, where o is the center of the camera and v is the unit direction vector of the ray. Then, three fully connected neural networks (SDF-Net, Validity-Net, Color-Net) are used to predict the signed distance value f(p(t)), validity probability V(p(t)) and RGB value c(p(t)) of the spatial point p(t), respectively, to obtain the implicit expression (including three values) of the spatial point p(t). Specifically, the three fully connected neural networks include:

[0068] SDF-Net: Mapping Function to represent the signed distance field.

[0069] Validity-Net: Mapping Function To represent the probability of effectiveness.

[0070] Color-Net: Mapping Function To predict the color of each point in 3D space.

[0071] Step 2: Input the generated implicit expression of the spatial points into the renderer to render the human body prediction image and prediction mask.

[0072] The three implicit representations of the spatial points are passed to the renderer to render the human body prediction image (I pred (o,v)) and the predicted mask (M pred (o,v)).

[0073] Prediction mask M pred(o,v) is inferred from the rendering weight w(p(t)) of each spatial point, and the human body prediction image I pred (o,v) is calculated based on the RGB value c(p(t)) and the rendering weight w(p(t)):

[0074]

[0075] According to the above equation, a key issue in the rendering process is to find a suitable weight function w(p(t)). This task is divided into two steps:

[0076] 1) construct a probability density function to estimate the volume density based on the signed distance value (f(p(t)));

[0077] 2) Obtain a suitable weight function w(p(t)) based on volume density and validity probability estimation.

[0078] The probability density function is constructed as follows:

[0079] The signed distance value is modified by flipping its sign in the region where the signed distance value increases along the camera ray. The probability density function is defined as:

[0080]

[0081] Where v is the unit direction vector of the ray and n is the gradient of the signed distance function. Represents the logical density distribution function, and the formula is as follows:

[0082] φ s (x) = se -sx / (1+e -sx )2 2

[0083] The definition of the probability density function ensures the same rendering behavior when light enters the surface from any direction.

[0084] The estimated weight function is as follows:

[0085] The weight function w(p(t)) has two properties: unbiased and occlusion-aware. The unbiased rendering weight function w(p(t)) is defined using the first formula below, and the occlusion-aware weight function is defined based on the opacity density ρ(t) in the second formula below.

[0086]

[0087] Solving the above two equations, we can obtain the opacity ρ(t):

[0088]

[0089] In order to obtain the predicted human image, the classical discretization scheme in differentiable volume rendering is used for the opacity and weight functions. For a set of spatial points {p i =o+t i v|i=1,...m,t i <t i+1}, the rendered human prediction image can be expressed as:

[0090]

[0091] where c i That is c(p(t i )), is the estimated color of the i-th sampling point; α i is the implicit expression of the signed distance value for the discrete opacity value when rendering, which can be written as:

[0092]

[0093] This results in an unbiased and occlusion-aware weight function w(p(t)) that enables the rendering of a human surface model using a signed distance function (SDF) representation.

[0094] Step 3: Optimize the surface of special parts of the human body, supervise and train the model according to the loss function;

[0095] In order to render some special surfaces of the human body model, such as closed surfaces and open surfaces, the validity probability of the 3D sampling points (V(p(t i ))) are multiplied by their opacity values ​​during rendering, e.g. Figure 1 (To render a human body prediction image using validity probability, where Object1 and Object2 represent human body parts from different perspectives).

[0096] Then the discrete opacity value of the i-th sampling point is β i for:

[0097] β i =α i ·V(p(t i ))

[0098] So the final rendered pixel color of the human body model surface can be expressed as:

[0099]

[0100] In the process of training the human body surface, five loss functions are used to supervise the rendered human body prediction image (I pred ) and the prediction mask (M pred). The first three loss functions are RGB Loss, Mask Loss, and Eikonal Loss. They are defined as:

[0101]

[0102] Where BCE is the binary cross entropy. i and j represent the center o of the camera and the unit direction vector v of the ray in the current training round, I gt Represents the original image of the human body, M gt The original object mask representing the human body.

[0103] The fourth is the rendering probability loss function: In the physical world, the existence of the human body surface is binary (absent / absent). Therefore, the validity probability of a 3D sample point is 0 (no surface) or 1 (surface). Therefore, the binary cross entropy of the validity probability V(p(t)) is used as an additional regularization:

[0104]

[0105] The fifth is the sparse loss function: For real-world human objects with open structures, the human body surface is sparsely distributed in 3D space. In order to prevent the network from predicting redundant human body surfaces, a sparse loss function is introduced to promote the formation of open human body surfaces:

[0106]

[0107] During the training process, the predicted human body prediction image and prediction mask are supervised according to the above five loss functions to obtain the model.

[0108] Step 4: Generate the human body surface based on the obtained model.

[0109] Input a 2D human body photo into the trained model. The model generates a predicted human body image and a predicted mask for the input image. The validity probability V(p(t)) is verified based on the signed distance function (SDF) of the predicted human body image and the predicted mask. The signed distance value of the 3D sampling point with V(p(t)) less than 0.5 is set to NAN. The human body surface validity processing process is as follows: Figure 2 As shown, the processed signed distance value is fed into the Marching Cubes algorithm to generate the final human body surface model.

[0110] Figure 2(a) is the signed distance function (SDF); (b) is the validity probability function V; (c) is the watertight surface extracted from (a) SDF; (d) is the open surface extracted from (a) SDF and (b) validity probability. In our mesh extraction process, we set the SDF of 3D query points with low validity (here V < 0.5) to NAN and use the Marching Cubes algorithm to extract open surfaces.

[0111] like Figure 3 As shown in the figure, Ground Truth is the original human body model. OURS is the model modeled by the method of the present invention. Ani-NeRF is the model modeled by the Ani-NeRF method. It can be seen that in the method of the present invention, the human body model is smoother and closer to the original human body model.

Claims

1. An implicit surface human body modeling method based on weight mask strategy, characterized in that: The steps include: Step 1: Generate implicit expression based on 2D human body photos; Step 2: Input the generated implicit expression of the spatial point into the renderer to render the human body prediction image and prediction mask; Step 3: Optimize the surface of special parts of the human body, supervise and train the model according to the loss function; Step 4: Generate the human body surface based on the obtained model.

2. According to the implicit surface human body modeling method based on weight mask strategy according to claim 1, it is characterized in that: Step 1: Given N human images Resolution is (W, H) and the corresponding camera intrinsics, extrinsics and human object masks The goal is to recreate a detailed human surface; For a given sampling point t on an input human image, it is first projected into 3D space, and the 3D point on the ray emitted from the pixel is obtained as {p(t)=o+tv|t≥0}, where o is the center of the camera and v is the unit direction vector of the ray; then, three fully connected neural networks (SDF-Net, Validity-Net, Color-Net) are used to predict the signed distance value f(p(t)), validity probability V(p(t)) and RGB value c(p(t)) of the spatial point p(t), respectively, that is, the implicit expression of the spatial point p(t) is obtained; specifically, the three fully connected neural networks include: SDF-Net: Mapping function f(·): To represent the signed distance field; Validity-Net: Mapping function V(·): To express the probability of effectiveness; Color-Net: Mapping function c(·): To predict the color of each point in 3D space.

3. The implicit surface human body modeling method based on weight mask strategy according to claim 2 is characterized in that: The specific method of step 2 is as follows: The three implicit representations of the spatial points are passed to the renderer to render the human prediction image I from the implicit representation pred (o, v) and the prediction mask M pred (o, v); Prediction mask M pred (o, v) is inferred from the rendering weight w(p(t)) of each spatial point, and the human body prediction image I pred (o, v) is calculated based on the RGB value c(p(t)) and the rendering weight w(p(t)): According to the above equation, a key issue in the rendering process is to find a suitable weight function w(p(t)); this task can be divided into two steps: 1) Construct a probability density function to estimate the volume density based on the signed distance value f(p(t)); 2) Obtain a suitable weight function w(p(t)) based on volume density and validity probability estimation; The probability density function is constructed as follows: The signed distance value is modified by flipping its sign in the region where the signed distance value increases along the camera ray; the probability density function is defined as: Where v is the unit direction vector of the ray, and n is the gradient of the signed distance function; Represents the logical density distribution function, and the formula is as follows: φ s (x)=se -sx / (1+e -sx ) 2 The definition of the probability density function ensures the same rendering behavior when light enters the surface from any direction; The estimated weight function is as follows: The weight function w(p(t)) has two properties: unbiased and occlusion-aware; The unbiased rendering weight function w(p(t)) is defined using the following first formula, and the occlusion-aware weight function is defined according to the opacity density ρ(t) in the following second formula; Solving the above two equations, we can obtain the opacity ρ(t): In order to obtain the predicted human body image, the classical discretization scheme in differentiable volume rendering is used for the opacity and weight functions; for a set of spatial points {p i =o+t i v|i=1,...m,t i <t i+1 }, the rendered human prediction image can be expressed as: where c i That is c(p(t i )), is the estimated color of the i-th sampling point; α i is the implicit expression of the signed distance value for the discrete opacity value when rendering, which can be written as: This establishes an unbiased and occlusion-aware weight function w(p(t)) that supports rendering the human body surface model using a signed distance function (SDF).

4. The implicit surface human body modeling method based on weight mask strategy according to claim 3 is characterized in that: Step 3: In order to render the special surface of the human body model, the validity probability V(p(t i )) multiplied by their opacity values ​​during rendering; then the discrete opacity value of the i-th sampling point is β i for: b i =a i ·V(p(t i )) So the final rendered pixel color of the human body model surface can be expressed as: During the training of the human body surface, five loss functions are used to supervise the rendered human body prediction images and prediction masks; the first three loss functions are RGB Loss, Mask Loss, and Eikonal Loss; they are defined as: Where BCE is the binary cross entropy; i and j represent the center o of the camera and the unit direction vector v of the ray in the current training round, I gt Represents the original image of the human body, M gt The original object mask representing the human body; The fourth is the rendering probability loss function: in the physical world, the existence of the human body surface is binary; therefore, the validity probability of a 3D sampling point is 0 or 1; therefore, the binary cross entropy of the validity probability V(p(t)) is used as an additional regularization: The fifth is a sparse loss function: For real-world human objects with open structures, the human body surfaces are sparsely distributed in 3D space; in order to prevent the network from predicting redundant human body surfaces, a sparse loss function is introduced to promote the formation of open human body surfaces: During the training process, the predicted human body prediction image and prediction mask are supervised according to the above five loss functions to obtain the model.

5. The implicit surface human body modeling method based on weight mask strategy according to claim 4 is characterized in that: Step 4: The specific method is as follows: Input a 2D human body photo into the trained model. The model generates a predicted human body image and a predicted mask for the input image. The validity probability V(p(t)) is verified based on the signed distance function SDF of the predicted human body image and the predicted mask. The signed distance value of the 3D sampling point with V(p(t)) less than 0.5 is set to NAN, and the processed signed distance value is fed to the Marching Cubes algorithm to generate the final human body surface model.

Images