Method, device, electronic device and storage medium for three-dimensional human body reconstruction

Abstract

The present invention discloses a method, device, electronic device and storage medium for three-dimensional human reconstruction, including: obtaining a human body image; inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained after training by rendering the human body scanning model after scanning the human body into different background images. The present invention provides semantic information and strong constraints for the point cloud by extracting the global features and local features of the point cloud, so that the three-dimensional human body reconstruction model can learn the global posture information and local detail information, obtain the three-dimensional semantic features by parameterizing the human body model, provide strong constraints for the model, improve the accuracy and authenticity of the details of the three-dimensional human body reconstruction model, and at the same time improve the robustness to the human body posture.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic equipment and storage medium for three-dimensional human body reconstruction. Background Art

[0002] At present, three-dimensional human body reconstruction is widely used in virtual reality, gaming experience, virtual try-on and other fields.

[0003] In the 3D human reconstruction model, the implicit expression method based on pixel alignment is dominant. This method implicitly represents the 3D human body as an occupation field in the 3D space, and aligns the sampled points in the 3D space with the 2D image to determine whether they are inside the human body occupation field. On the one hand, this method ignores the spatial geometric dependency between points, resulting in local reconstruction details such as clothing folds and facial wrinkles that are not realistic enough. On the other hand, due to factors such as scene diversity and the complexity of human posture, the existing methods have poor robustness to human posture.

[0004] In summary, there is an urgent need for a three-dimensional human body reconstruction technology to solve the problems existing in the above-mentioned existing technologies. Summary of the invention

[0005] Since the existing methods have the above problems, the present invention proposes a method, device, electronic device and storage medium for three-dimensional human body reconstruction.

[0006] In a first aspect, the present invention provides a method for three-dimensional human body reconstruction, comprising:

[0007] Acquire human body images;

[0008] Inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image;

[0009] The trained three-dimensional human body reconstruction model is obtained by training an image generated by rendering a human body scanning model after scanning the human body into different background images.

[0010] Furthermore, before inputting the human body image into the trained three-dimensional human body reconstruction model to obtain the three-dimensional reconstruction result of the human body image, it also includes:

[0011] Acquire a background image, a human body scan model, and camera projection parameters corresponding to the human body scan model;

[0012] Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, the camera projection parameters, and the background image; the image training set is obtained by rendering the human body scanning model into the background image;

[0013] Sampling a point cloud in the three-dimensional space where the human body scan model is located and generating a label corresponding to each point in the point cloud; the label is used to indicate the spatial relationship between each point in the point cloud and the human body scan model;

[0014] Extracting image features, global features and local features corresponding to the point cloud;

[0015] Determine a probability set of the point cloud being inside a human body according to the image features, the global features, and the local features;

[0016] Determine a first loss and a second loss according to a probability set of the point cloud inside the human body;

[0017] The first loss and the second loss are weightedly calculated to update the parameters of the three-dimensional human body reconstruction model to obtain a trained three-dimensional human body reconstruction model.

[0018] Furthermore, sampling in the three-dimensional space where the human body scan model is located to obtain a point cloud and generating a label corresponding to each point in the point cloud includes:

[0019] Sampling N points on the human body scan model and adding noise to the coordinates of each point;

[0020] Sampling M points in the three-dimensional space where the human body scanning model is located;

[0021] The set of the N points and the set of the M points are taken as a point cloud; wherein N and M are both positive integers, and N is greater than M.

[0022] Furthermore, extracting image features, global features and local features corresponding to the point cloud includes:

[0023] Determine the posture information and shape information of the human body scan model according to the image training set;

[0024] Obtaining a parameterized human body model according to the posture information and the body information;

[0025] determining global features of the point cloud based on the parameterized human body model;

[0026] Local features of the point cloud are determined based on the parameterized human body model and the point cloud.

[0027] Further, determining the local features of the point cloud according to the parameterized human body model and the point cloud includes:

[0028] For each point in the point cloud, calculating the distance from each point to a point on the surface of the parameterized human body model;

[0029] Connecting points on the surface of the parameterized human body model whose distance is less than a preset threshold;

[0030] The local features of the point cloud are extracted by a local feature extraction model.

[0031] Further, determining the first loss and the second loss according to the probability set of the point cloud inside the human body includes:

[0032] Determine a first loss according to a probability set of the point cloud inside the human body and a label corresponding to each point in the point cloud;

[0033] Calculate a first silhouette value, a first depth value, and a first normal vector value respectively according to the silhouette image, the depth image, and the normal vector image;

[0034] Determine a second silhouette value, a second depth value, and a second normal vector value according to a probability set of the point cloud inside the human body;

[0035] A second loss is determined according to the first silhouette value, the first depth value, the first normal vector value, the second silhouette value, the second depth value, and the second normal vector value.

[0036] Furthermore, the updating of the parameters of the three-dimensional human body reconstruction model after weighted calculation of the first loss and the second loss to obtain a trained three-dimensional human body reconstruction model includes:

[0037] The first loss and the second loss are weighted to obtain a training loss;

[0038] When the training loss converges, the training is stopped to obtain the trained three-dimensional human body reconstruction model.

[0039] In a second aspect, the present invention provides a device for three-dimensional human body reconstruction, comprising:

[0040] An acquisition module, used for acquiring a human body image;

[0041] A processing module is used to input the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained after training using an image generated by rendering a human body scanning model after scanning the human body into different background images.

[0042] Furthermore, the processing module is also used for:

[0043] Before inputting the human body image into the trained three-dimensional human body reconstruction model to obtain the three-dimensional reconstruction result of the human body image, obtaining a background image, a human body scanning model, and camera projection parameters corresponding to the human body scanning model;

[0044] Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, the camera projection parameters, and the background image; the image training set is obtained by rendering the human body scanning model into the background image;

[0045] Sampling a point cloud in the three-dimensional space where the human body scan model is located and generating a label corresponding to each point in the point cloud; the label is used to indicate the spatial relationship between each point in the point cloud and the human body scan model;

[0046] Extracting image features, global features and local features corresponding to the point cloud;

[0047] Determine a probability set of the point cloud being inside a human body according to the image features, the global features, and the local features;

[0048] Determine a first loss and a second loss according to a probability set of the point cloud inside the human body;

[0049] The first loss and the second loss are weightedly calculated to update the parameters of the three-dimensional human body reconstruction model to obtain a trained three-dimensional human body reconstruction model.

[0050] Furthermore, the processing module is specifically used for:

[0051] Sampling N points on the human body scan model and adding noise to the coordinates of each point;

[0052] Sampling M points in the three-dimensional space where the human body scanning model is located;

[0053] The set of the N points and the set of the M points are taken as a point cloud; wherein N and M are both positive integers, and N is greater than M.

[0054] Furthermore, the processing module is specifically used for:

[0055] Determine the posture information and shape information of the human body scan model according to the image training set;

[0056] Obtaining a parameterized human body model according to the posture information and the body information;

[0057] determining global features of the point cloud based on the parameterized human body model;

[0058] Local features of the point cloud are determined based on the parameterized human body model and the point cloud.

[0059] Furthermore, the processing module is specifically used for:

[0060] For each point in the point cloud, calculating the distance from each point to a point on the surface of the parameterized human body model;

[0061] Connecting points on the surface of the parameterized human body model whose distance is less than a preset threshold;

[0062] The local features of the point cloud are extracted by a local feature extraction model.

[0063] Furthermore, the processing module is specifically used for:

[0064] Determine a first loss according to a probability set of the point cloud inside the human body and a label corresponding to each point in the point cloud;

[0065] Calculate a first silhouette value, a first depth value, and a first normal vector value respectively according to the silhouette image, the depth image, and the normal vector image;

[0066] Determine a second silhouette value, a second depth value, and a second normal vector value according to a probability set of the point cloud inside the human body;

[0067] A second loss is determined according to the first silhouette value, the first depth value, the first normal vector value, the second silhouette value, the second depth value, and the second normal vector value.

[0068] Furthermore, the processing module is specifically used for:

[0069] The first loss and the second loss are weighted to obtain a training loss;

[0070] When the training loss converges, the training is stopped to obtain the trained three-dimensional human body reconstruction model.

[0071] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for three-dimensional human body reconstruction as described in the first aspect is implemented.

[0072] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for three-dimensional human body reconstruction as described in the first aspect.

[0073] It can be seen from the above technical scheme that the method, device, electronic device and storage medium for three-dimensional human body reconstruction provided by the present invention provide semantic information and strong constraints for point clouds by extracting global features and local features of point clouds, so that the three-dimensional human body reconstruction model can learn global posture information and local detail information, obtain three-dimensional semantic features by parameterizing the human body model, provide strong constraints for the model, improve the accuracy and authenticity of the details of the three-dimensional human body reconstruction model, and at the same time improve the robustness to human body posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A system framework for the method of three-dimensional human body reconstruction provided by the present invention;

[0076] Figure 2 A schematic diagram of a flow chart of a method for three-dimensional human body reconstruction provided by the present invention;

[0077] Figure 3 A schematic diagram of a flow chart of a method for three-dimensional human body reconstruction provided by the present invention;

[0078] Figure 4 A schematic diagram of a flow chart of a method for three-dimensional human body reconstruction provided by the present invention;

[0079] Figure 5 A schematic diagram of a flow chart of a method for three-dimensional human body reconstruction provided by the present invention;

[0080] Figure 6 A schematic diagram of a flow chart of a method for three-dimensional human body reconstruction provided by the present invention;

[0081] Figure 7 A schematic diagram of the structure of a device for three-dimensional human body reconstruction provided by the present invention;

[0082] Figure 8 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0083] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0084] The method for 3D human body reconstruction provided by the embodiment of the present invention can be applied to Figure 1 In the system architecture shown, the system architecture includes a camera 100 and a three-dimensional human body reconstruction model 200 .

[0085] Specifically, the camera 100 is used to obtain human body images.

[0086] The three-dimensional human body reconstruction model 200 is used to obtain a three-dimensional reconstruction result of the human body image after the human body image sent by the camera 100 is input.

[0087] It should be noted that Figure 1 This is only an example of the system architecture of the embodiment of the present invention, and the present invention does not make any specific limitation to this.

[0088] Based on the system architecture shown above, Figure 2 A flowchart corresponding to a method for 3D human body reconstruction provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0089] Step 201, obtaining a human body image.

[0090] Step 202: input the human body image into the trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image.

[0091] It should be noted that the trained three-dimensional human body reconstruction model is obtained by training with images generated by rendering the human body scanning model after scanning the human body into different background images.

[0092] The above scheme inputs the human body image to be reconstructed into the trained three-dimensional human body reconstruction model, performs network forward calculation, and obtains the probability set of the point cloud in the three-dimensional space being inside the human body. The three-dimensional semantic features are obtained by parameterizing the human body model, which provides strong constraints for the model, improves the accuracy and realism of the details of the three-dimensional human body reconstruction model, and improves the robustness to human body posture.

[0093] In the embodiment of the present invention, before step 202, the steps are as follows: Figure 3 As shown, the details are as follows:

[0094] Step 301, obtaining a background image, a human body scan model, and camera projection parameters corresponding to the human body scan model.

[0095] Specifically, the human body scanning model is obtained by scanning a three-dimensional human body with a scanner.

[0096] It should be noted that the coordinate system of the human body scan model is the world coordinate system. The camera projection parameters are generated in the process of rendering the human body scan model to the background image, and one image corresponds to one camera projection parameter.

[0097] In the embodiment of the present invention, real images in different scenes are obtained as background images. For example, real images of amusement facilities in an amusement park scene are obtained as background images.

[0098] It should be noted that the human body scan model obtained must be airtight and without holes.

[0099] Furthermore, in order to make the three-dimensional human body reconstruction model more generalizable, the acquired background images must meet the requirements of authenticity and diversity to be applicable to common scenes in life.

[0100] Step 302: Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, camera projection parameters, and background image.

[0101] It should be noted that the image training set is obtained by rendering the human body scan model into the background image.

[0102] In one possible implementation, the image training set is generated in the following manner:

[0103] After the human body scan model is normalized, it is randomly rotated, translated, and scaled along a coordinate axis perpendicular to the ground plane, such as the y-axis, to obtain its corresponding camera projection parameters.

[0104] Specifically, a background image is randomly selected in the image library, and the transformed human body scan model is rendered into the selected background image as a training sample through orthogonal projection.

[0105] Furthermore, a silhouette image, a depth image and a normal vector image corresponding to the human body scan model are obtained by rendering the transformed human body scan model.

[0106] In the embodiment of the present invention, the silhouette image highlights the subject and shows the body's appearance, posture and outline. The depth image refers to an image that uses the depth from the image collector to each point in the scene as a pixel value, which reflects the geometric shape of the visible surface of the human body.

[0107] It should be noted that the embodiment of the present invention may also use projection methods such as perspective projection, which is not specifically limited in the embodiment of the present invention.

[0108] For example, the process of generating an image training set is as follows:

[0109] First, a background image is selected, and the background image is cropped and scaled to a predefined size, such as a predefined size of 512×512.

[0110] Furthermore, the human body scan model is normalized.

[0111] Specifically, the human body scanning model is translated to a position centered on the zero coordinate, and then the human body scanning model is scaled to the range of the coordinate axis [-1,1], that is, the three-dimensional human body coordinates are divided by the maximum value of the coordinate range of the coordinate axis in the direction of the human body height, and then the human body scanning model is randomly translated, scaled, and rotated to any viewing angle, and finally the human body scanning model is rendered into the background image through orthogonal projection.

[0112] Step 303: sampling in the three-dimensional space where the human body scan model is located to obtain a point cloud and generate a label corresponding to each point in the point cloud.

[0113] It should be noted that the labels are used to indicate the spatial relationship between each point in the point cloud and the human body scan model.

[0114] In a possible implementation manner, the embodiment of the present invention marks points inside the human body as 1, and marks points outside the human body as 0.

[0115] It should be noted that in the above marking method, a point label of 1 indicates that the probability of the point being inside the human body is 1, that is, the point is on the human body scanning model, and a point label of 0 indicates that the probability of the point being inside the human body is 0, that is, the point is outside the human body scanning model.

[0116] In step 303 of the embodiment of the present invention, the method flow is as follows: Figure 4 As shown, the specific steps are as follows:

[0117] Step 401: sample N points on the human body scan model and add noise to the coordinates of each point.

[0118] It should be noted that the embodiment of the present invention samples N points on the surface of the human body scanning model, and after adding noise to the coordinates of each point, a portion of the sampled N points becomes points inside the human body scanning model or points outside the human body scanning model.

[0119] In a possible implementation, a normal distribution noise is added to the three-dimensional coordinates of each point.

[0120] For example, a normal distribution noise perturbation with a mean of 0 and a variance of 0.05 is added to the three-dimensional coordinates of each point.

[0121] Step 402: Sample M points in the three-dimensional space where the human body scan model is located.

[0122] Step 403: Take the set of N points and M points as a point cloud.

[0123] It should be noted that, N and M are both positive integers.

[0124] In the embodiment of the present invention, N is greater than M.

[0125] In the above scheme, since infinite precision sampling can be achieved in three-dimensional space, simple uniform sampling is very inefficient. However, sampling within a certain distance range on the human body surface can enable the three-dimensional human body reconstruction model to better find the decision boundary.

[0126] In a possible implementation, sampling is performed uniformly near the surface of the human body scan model and in three-dimensional space in proportion.

[0127] For example, the ratio of N to M is 16:1.

[0128] The above scheme can make the three-dimensional human body reconstruction model converge quickly while having better generalization ability.

[0129] Step 304: extract image features, global features, and local features corresponding to the point cloud.

[0130] In a possible implementation, the point cloud, the image training set, and the camera projection parameters are input into an image feature extraction model to obtain image features corresponding to the point cloud.

[0131] For example, a feature map is obtained through an image feature extraction model built based on the Hourglass network. According to the camera projection parameters, the sampled point cloud is projected onto the feature map through orthogonal projection to extract the image features corresponding to the point cloud.

[0132] In step 304 of the embodiment of the present invention, the method flow is as follows Figure 5 As shown, the specific steps are as follows:

[0133] Step 501, determining the posture information and shape information of the human body scanning model according to the image training set.

[0134] Step 502, obtaining a parameterized human body model according to the posture information and the body information.

[0135] In a possible implementation, a parameterized human body model is obtained by using a three-dimensional human body reconstruction model constructed based on a SPIN network.

[0136] It should be noted that the parameterized human body model may also be obtained through a three-dimensional human body reconstruction model constructed based on an SMPL (Skinned Multi-Person Linear) network, and this embodiment of the present invention does not specifically limit this.

[0137] The above scheme provides semantic information for unconstrained points in three-dimensional space by constructing a parametric human body model and using it as prior knowledge.

[0138] Step 503: determine the global features of the point cloud according to the parameterized human body model.

[0139] In a possible implementation, the global features of the point cloud are acquired through a global feature extraction model built based on the Pointnet network.

[0140] The above scheme extracts global features through a global feature extraction model. The global features contain the posture information of the three-dimensional human body, thereby improving the robustness of the three-dimensional human body reconstruction model to the posture.

[0141] Furthermore, the point cloud of the parameterized human body model is subjected to feature extraction through a multi-layer perception network to obtain a feature set of the point cloud.

[0142] In a possible implementation, the global features of the point cloud are obtained using the maximum pooling function, as follows:

[0143]

[0144] Among them, f g represents the global feature extraction model, V represents the point cloud of the parameterized three-dimensional human body model, h represents the multi-layer perception network, Represents the maximum pooling function.

[0145] In the above scheme, the symmetry of the maximum pooling function makes it invariant to the disorder, data missing, noise and disturbance of the point cloud.

[0146] Step 504 : determining local features of the point cloud according to the parameterized human body model and the point cloud.

[0147] Specifically, the point cloud and the parameterized human body model are input into a local feature extraction model to obtain local features of the point cloud.

[0148] In a possible implementation, local features of the point cloud are acquired by using a local feature extraction model built based on a graph convolutional neural network.

[0149] In the embodiment of the present invention, for each point in the point cloud, the distance from each point to a point on the surface of the parameterized human body model is calculated;

[0150] Connecting points on the surface of the parameterized human body model whose distance is less than a preset threshold;

[0151] The local features of the point cloud are extracted through the local feature extraction model.

[0152] In a possible implementation, the points on the surface of the parameterized human body model that have the smallest distances to the first K sampling points in the point cloud are connected.

[0153] Specifically, for each point in the point cloud, the distance from the point to the surface of the parametric human body model, i.e., the P2S (point-to-surface distance) distance, is calculated, and the point is connected to the first K points closest to the parametric human body model. The three-dimensional local features are extracted by the local feature extraction model based on the graph convolutional network (GCN). The specific method is as follows:

[0154]

[0155] Where p is any point in the point cloud, V is the point cloud of the parameterized human body model, and V p represents the set of K points closest to p in V, and h represents the local feature extraction model.

[0156] The above scheme extracts local features through a local feature extraction model. The local features contain detailed information, such as fine wrinkles of clothes, etc., which improves the accuracy of the three-dimensional human body reconstruction model.

[0157] Step 305 , determining a probability set of the point cloud being inside the human body according to the image features, global features, and local features.

[0158] In a possible implementation, the image features, global features, and local features of the point cloud are input into a point cloud classification model to obtain a probability set of the point cloud being inside the human body.

[0159] In a possible implementation, the probability of each point in the point cloud being inside the human body is predicted by a point cloud classification model built based on a multilayer perceptron (MLP).

[0160] Specifically, the probability set of the point cloud inside the human body is calculated as follows;

[0161] F(p,I,V)=H(Φ(p,I),f g (V), f l (p,V)

[0162] It should be noted that, p represents any point in the point cloud, I represents the image training set, V represents the point cloud of the parameterized human body model; Φ represents the image feature extraction model, f g represents the global feature extraction model, f l represents the local feature extraction model, and H represents the point cloud classification model.

[0163] From the above formula, it can be seen that the three sets of features are combined in series and input into the point cloud classification model to obtain the probability of each point being inside the human body.

[0164] In the above scheme, global features and local features provide semantic information and strong constraints for the three-dimensional point cloud, so that the three-dimensional human body reconstruction model can simultaneously learn global posture information and local detail information.

[0165] Step 306 , determining the first loss and the second loss according to the probability set of the point cloud inside the human body.

[0166] In step 306 of the embodiment of the present invention, the method flow is as follows: Figure 6 As shown, the specific steps are as follows:

[0167] Step 601, determining a first loss based on a probability set of a point cloud inside a human body and a label corresponding to each point in the point cloud.

[0168] Specifically, the specific calculation formula for the first loss is as follows:

[0169]

[0170] The labels corresponding to each point in the cloud.

[0171] Step 602, respectively calculate a first silhouette value, a first depth value, and a first normal vector value according to the silhouette image, the depth image, and the normal vector image.

[0172] Specifically, the specific calculation formula of the first silhouette value is as follows:

[0173]

[0174] It should be noted that, among them, π k is the camera projection parameter, x is the two-dimensional coordinate on the image, c represents the camera center position, r(π k , x) represents the direction of the ray passing through the point on the image along the camera position, d j represents the distance from the camera center to each sampling point, F represents the classification network, Ψ represents the maximum pooling function, N p Indicates the number of sampling points.

[0175] In the embodiment of the present invention, the specific calculation formula of the first depth value is as follows:

[0176]

[0177] Where σ represents the decision threshold, represents the jth point on ray r, represents the j+1th point on ray r, It means that the j+1th point on ray r is input into the model to obtain the probability that it is inside the human body.

[0178] In a possible implementation, the value of σ is 0.5. The human body model is segmented using the Marching Cubic algorithm according to the decision threshold of 0.5 to obtain a three-dimensional reconstruction result.

[0179] In the embodiment of the present invention, the depth value of the human body surface is in the interval (d j ,d j+1 ), and then the second depth value is accurately calculated using the binary search method.

[0180] Specifically, the specific calculation formula for the first normal vector value is as follows:

[0181]

[0182] Among them, π k is the camera projection parameter, x is the two-dimensional coordinate on the image, p k is the coordinate of the 3D point closest to the 3D human body surface along the ray at viewing angle k.

[0183] Step 603, determining a second silhouette value, a second depth value, and a second normal vector value according to a probability set of the point cloud inside the human body.

[0184] In the embodiment of the present invention, the silhouette value, depth value and normal vector value of the point cloud at different viewing angles are calculated based on the probability set of the point cloud inside the human body.

[0185] For example, four silhouette values, four depth values, and four normal vector values ​​are calculated at viewing angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively, and the four silhouette values, four depth values, and four normal vector values ​​are averaged to obtain a second silhouette value, a second depth value, and a second normal vector value, respectively.

[0186] Step 604, determining a second loss according to the first silhouette value, the first depth value, the first normal vector value, the second silhouette value, the second depth value, and the second normal vector value.

[0187] It should be noted that in the embodiment of the present invention, the first silhouette value, the first depth value, and the first normal vector value are obtained from the real silhouette image, depth map, and normal vector map, and the second silhouette value, the second depth value, and the second normal vector value are obtained from the predicted silhouette image, depth map, and normal vector map.

[0188] Specifically, in the embodiment of the present invention, the specific calculation formula of the second loss is as follows:

[0189] L 2D =L sillhouette +L depth +L normal (Formula 1)

[0190]

[0191]

[0192]

[0193] It should be noted that, among them, L 2D represents the second loss, L sillhouette represents the silhouette loss, L depth represents the depth loss, L normal represents the normal vector loss; K represents K pre-set viewing angles, π k represents the camera projection parameters of the viewing angle k, x represents the image coordinates; S, D, N represent the predicted silhouette image, depth map and normal vector map respectively, S * , D * , N * Represent the true silhouette image, depth image and normal vector image respectively.

[0194] The above scheme renders the 3D reconstruction results into depth maps, silhouette maps and normal vector maps through multi-view differentiable rendering, and uses the second loss to perform more refined optimization of the 3D reconstruction results.

[0195] Step 307 , performing weighted calculation on the first loss and the second loss to update the parameters of the 3D human body reconstruction model, and obtaining a trained 3D human body reconstruction model.

[0196] Specifically, the weighted sum of the first loss and the second loss is calculated as follows:

[0197]

[0198] Among them, γ 1 and γ 2 represents the weight parameter, L 3D represents the first loss, that is, the mean square error loss between the predicted probability value of each point inside the 3D human body and its label; L 2D represents the second loss, i.e., the loss of rendering the 3D human body reconstruction result into a 2D image calculation, and θ represents the parameters of the 3D human body reconstruction model.

[0199] The above scheme renders the network output results to multiple perspectives through a differentiable rendering method based on ray tracing, obtains the predicted silhouette image, depth map and normal vector map, and then calculates the loss with the corresponding real silhouette image, depth map and normal vector map, so as to reversely conduct and update the parameters of the three-dimensional human body reconstruction model.

[0200] Furthermore, in an embodiment of the present invention, the learnable model parameter θ is optimized according to the weighted loss of the first loss and the second loss, and it is determined whether the training loss converges. If it converges, the training is terminated to obtain the model parameter θ; otherwise, the above method is repeated until a trained three-dimensional human body reconstruction model is obtained.

[0201] The above scheme provides semantic information and strong constraints for the point cloud by extracting the global and local features of the point cloud, so that the 3D human body reconstruction model can learn the global posture information and local detail information. The 3D semantic features are obtained by parameterizing the human body model, which provides strong constraints for the model, improves the accuracy and realism of the details of the 3D human body reconstruction model, and at the same time improves the robustness to the human body posture.

[0202] Based on the same inventive concept, Figure 7 An apparatus for three-dimensional human body reconstruction provided by an embodiment of the present invention is exemplarily shown, and the apparatus may be a process of a three-dimensional human body reconstruction method.

[0203] The device comprises:

[0204] An acquisition module 701 is used to acquire a human body image;

[0205] Processing module 702 is used to input the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained after training by rendering an image generated by scanning a human body model into different background images.

[0206] Furthermore, the processing module 702 is also used for:

[0207] Before inputting the human body image into the trained three-dimensional human body reconstruction model to obtain the three-dimensional reconstruction result of the human body image, obtaining a background image, a human body scanning model, and camera projection parameters corresponding to the human body scanning model;

[0208] Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, the camera projection parameters, and the background image; the image training set is obtained by rendering the human body scanning model into the background image;

[0209] Sampling a point cloud in the three-dimensional space where the human body scan model is located and generating a label corresponding to each point in the point cloud; the label is used to indicate the spatial relationship between each point in the point cloud and the human body scan model;

[0210] Extracting image features, global features and local features corresponding to the point cloud;

[0211] Determine a probability set of the point cloud being inside a human body according to the image features, the global features, and the local features;

[0212] Determine a first loss and a second loss according to a probability set of the point cloud inside the human body;

[0213] The first loss and the second loss are weightedly calculated to update the parameters of the three-dimensional human body reconstruction model to obtain a trained three-dimensional human body reconstruction model.

[0214] Furthermore, the processing module 702 is specifically used for:

[0215] Sampling N points on the human body scan model and adding noise to the coordinates of each point;

[0216] Sampling M points in the three-dimensional space where the human body scanning model is located;

[0217] The set of the N points and the set of the M points are taken as a point cloud; wherein N and M are both positive integers, and N is greater than M.

[0218] Furthermore, the processing module 702 is specifically used for:

[0219] Determine the posture information and shape information of the human body scan model according to the image training set;

[0220] Obtaining a parameterized human body model according to the posture information and the body information;

[0221] determining global features of the point cloud based on the parameterized human body model;

[0222] Local features of the point cloud are determined based on the parameterized human body model and the point cloud.

[0223] Furthermore, the processing module 702 is specifically used for:

[0224] For each point in the point cloud, calculating the distance from each point to a point on the surface of the parameterized human body model;

[0225] Connecting points on the surface of the parameterized human body model whose distance is less than a preset threshold;

[0226] The local features of the point cloud are extracted by a local feature extraction model.

[0227] Furthermore, the processing module 702 is specifically used for:

[0228] Determine a first loss according to a probability set of the point cloud inside the human body and a label corresponding to each point in the point cloud;

[0229] Calculate a first silhouette value, a first depth value, and a first normal vector value respectively according to the silhouette image, the depth image, and the normal vector image;

[0230] Determine a second silhouette value, a second depth value, and a second normal vector value according to a probability set of the point cloud inside the human body;

[0231] A second loss is determined according to the first silhouette value, the first depth value, the first normal vector value, the second silhouette value, the second depth value, and the second normal vector value.

[0232] Furthermore, the processing module 702 is specifically used for:

[0233] The first loss and the second loss are weighted to obtain a training loss;

[0234] When the training loss converges, the training is stopped to obtain the trained three-dimensional human body reconstruction model.

[0235] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, see Figure 8 , the electronic device specifically includes the following contents: a processor 801, a memory 802, a communication interface 803 and a communication bus 804;

[0236] The processor 801, the memory 802, and the communication interface 803 communicate with each other through the communication bus 804; the communication interface 803 is used to realize information transmission between various devices;

[0237] The processor 801 is used to call the computer program in the memory 802. When the processor executes the computer program, all steps of the above-mentioned three-dimensional human body reconstruction method are implemented. For example, when the processor executes the computer program, the following steps are implemented: obtaining a human body image; inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained after training by rendering the human body scanning model after scanning the human body into an image generated by different background images.

[0238] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, all the steps of the above-mentioned three-dimensional human body reconstruction method are implemented. For example, when the processor executes the computer program, the following steps are implemented: obtaining a human body image; inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained after training by rendering the human body scanning model after scanning the human body into an image generated by different background images.

[0239] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a user life pattern prediction device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0240] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. Those of ordinary skill in the art may understand and implement them without creative effort.

[0241] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a user life pattern prediction device, or a network device, etc.) to execute the user life pattern prediction method described in each embodiment or some parts of the embodiment.

[0242] In addition, in the present invention, terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0243] In addition, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0244] In addition, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-dimensional human body reconstruction, characterized in that: include: Acquire human body images; Inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; The trained three-dimensional human body reconstruction model is obtained by training with images generated by rendering the human body scanning model after scanning the human body into different background images; Before inputting the human body image into the trained three-dimensional human body reconstruction model to obtain the three-dimensional reconstruction result of the human body image, the method further includes: Acquire a background image, a human body scan model, and camera projection parameters corresponding to the human body scan model; Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, the camera projection parameters, and the background image; the image training set is obtained by rendering the human body scanning model into the background image; Sampling a point cloud in the three-dimensional space where the human body scan model is located and generating a label corresponding to each point in the point cloud; the label is used to indicate the spatial relationship between each point in the point cloud and the human body scan model; Extracting image features, global features and local features corresponding to the point cloud; Determine a probability set of the point cloud being inside a human body according to the image features, the global features, and the local features; Determine a first loss and a second loss according to a probability set of the point cloud inside the human body; The first loss and the second loss are weightedly calculated to update the parameters of the three-dimensional human body reconstruction model to obtain a trained three-dimensional human body reconstruction model.

2. The method for 3D human body reconstruction according to claim 1, characterized in that: The step of sampling a point cloud in the three-dimensional space where the human body scanning model is located and generating a label corresponding to each point in the point cloud includes: Sampling N points on the human body scan model and adding noise to the coordinates of each point; Sampling M points in the three-dimensional space where the human body scanning model is located; The set of the N points and the set of the M points are taken as a point cloud; wherein N and M are both positive integers, and N is greater than M.

3. The method for 3D human body reconstruction according to claim 1, characterized in that: The extracting of image features, global features and local features corresponding to the point cloud includes: Determine the posture information and shape information of the human body scan model according to the image training set; Obtaining a parameterized human body model according to the posture information and the body information; determining global features of the point cloud based on the parameterized human body model; Local features of the point cloud are determined based on the parameterized human body model and the point cloud.

4. The method for 3D human body reconstruction according to claim 3, characterized in that: The determining the local features of the point cloud according to the parameterized human body model and the point cloud comprises: For each point in the point cloud, calculating the distance from each point to a point on the surface of the parameterized human body model; Connecting points on the surface of the parameterized human body model whose distance is less than a preset threshold; The local features of the point cloud are extracted by a local feature extraction model.

5. The method for 3D human body reconstruction according to claim 1, characterized in that: The determining the first loss and the second loss according to the probability set of the point cloud inside the human body includes: Determine a first loss according to a probability set of the point cloud inside the human body and a label corresponding to each point in the point cloud; Calculate a first silhouette value, a first depth value, and a first normal vector value respectively according to the silhouette image, the depth image, and the normal vector image; Determine a second silhouette value, a second depth value, and a second normal vector value according to a probability set of the point cloud inside the human body; A second loss is determined according to the first silhouette value, the first depth value, the first normal vector value, the second silhouette value, the second depth value, and the second normal vector value.

6. The method for 3D human body reconstruction according to claim 1, characterized in that: The method of updating the parameters of the three-dimensional human body reconstruction model after weighted calculation of the first loss and the second loss to obtain a trained three-dimensional human body reconstruction model comprises: The first loss and the second loss are weighted to obtain a training loss; When the training loss converges, the training is stopped to obtain the trained three-dimensional human body reconstruction model.

7. A device for three-dimensional human body reconstruction, characterized in that: include: An acquisition module, used for acquiring a human body image; A processing module, used for inputting the human body image into a trained three-dimensional human body reconstruction model to obtain a three-dimensional reconstruction result of the human body image; wherein the trained three-dimensional human body reconstruction model is obtained by training an image generated by rendering a human body scanning model after scanning a human body into different background images; The processing module is also used for: Before inputting the human body image into the trained three-dimensional human body reconstruction model to obtain the three-dimensional reconstruction result of the human body image, obtaining a background image, a human body scanning model, and camera projection parameters corresponding to the human body scanning model; Generate an image training set, a silhouette image, a depth image, and a normal vector image according to the human body scanning model, the camera projection parameters, and the background image; the image training set is obtained by rendering the human body scanning model into the background image; Sampling a point cloud in the three-dimensional space where the human body scan model is located and generating a label corresponding to each point in the point cloud; the label is used to indicate the spatial relationship between each point in the point cloud and the human body scan model; Extracting image features, global features and local features corresponding to the point cloud; Determine a probability set of the point cloud being inside a human body according to the image features, the global features, and the local features; Determine a first loss and a second loss according to a probability set of the point cloud inside the human body; The first loss and the second loss are weightedly calculated to update the parameters of the three-dimensional human body reconstruction model to obtain a trained three-dimensional human body reconstruction model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images