Method and apparatus for reconstructing a three-dimensional garment model
By acquiring monocular video and selecting keyframes for 3D reconstruction, the high complexity caused by complex camera systems in existing technologies is solved, achieving efficient 3D clothing model reconstruction and reducing information loss and occlusion problems.
Patent Information
- Application Number
- CN202310834542.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-07-07
AI Technical Summary
Existing technologies for reconstructing 3D clothing models from multiple perspectives require complex camera array systems, resulting in high complexity.
By acquiring monocular video of a stationary human body wearing the clothing to be reconstructed around its entire length, multiple target images are selected for 3D reconstruction. The complexity of 3D reconstruction is reduced by utilizing monocular video and keyframe selection mechanisms, combined with the SMPL model and renderer.
It enables efficient reconstruction of 3D clothing models without the need for complex camera systems and complex photography requirements, reducing the amount of image processing, alleviating occlusion and information loss problems, and reducing the complexity of multi-view reconstruction.
Smart Images

Figure CN116883592B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer vision, and in particular to methods and apparatus for reconstructing three-dimensional clothing models, and computer-storable media. Background Technology
[0002] Digital clothing models are a key element in fields such as e-commerce, design and manufacturing, virtual reality, digital humans, and animation production.
[0003] In related technologies, methods for reconstructing 3D models of clothing based on multiple perspectives require complex camera array systems or specific photography requirements. This involves capturing images of clothing from multiple viewpoints in a studio environment and then solving the surface geometry of the clothing using solid geometry algorithms. Summary of the Invention
[0004] Related technologies require complex camera array systems, and the complexity of multi-view 3D clothing reconstruction is relatively high.
[0005] To address the aforementioned technical issues, this disclosure proposes a solution that can reduce the complexity of reconstructing 3D clothing models from multiple perspectives.
[0006] According to a first aspect of this disclosure, a method for reconstructing a three-dimensional clothing model is provided, comprising: acquiring a monocular video of a stationary human body wearing clothing to be reconstructed, the monocular video comprising multiple frames of original images; selecting multiple target images from the multiple frames of original images, wherein the number of the multiple target images is less than or equal to a number threshold and the capture range of the multiple target images of the clothing to be reconstructed is greater than or equal to a range threshold, and the number of the multiple target images is less than the number of the multiple original images; and performing three-dimensional reconstruction of the clothing to be reconstructed based on the multiple target images to obtain a target three-dimensional clothing model corresponding to the clothing to be reconstructed.
[0007] In some embodiments, selecting multiple target images from the multiple original images includes: performing human pose estimation on each original image to obtain shape parameters, pose parameters, and camera parameters of a skinned multi-person linear SMPL model corresponding to each original image; determining an SMLP model based on the average of multiple shape parameters and the average of multiple pose parameters corresponding to the multiple original images; determining the camera-visible human surface area in each original image based on the determined SMLP model and the camera parameters corresponding to each original image, wherein the human surface area represents the capture range of the original image for the garment to be reconstructed; and selecting multiple target images from the multiple original images in descending order of the area of the human surface area, wherein the capture range of the target images for the garment to be reconstructed is the area of the union of the camera-visible human surface areas in the target images.
[0008] In some embodiments, performing three-dimensional reconstruction of the garment to be reconstructed based on the multiple target images to obtain a target three-dimensional garment model corresponding to the garment to be reconstructed includes: acquiring camera parameters corresponding to each target image frame and an initial three-dimensional garment model corresponding to the garment to be reconstructed; determining the contour mask and normal map of the garment to be reconstructed in each target image frame; performing deformation processing on the initial three-dimensional garment model based on latent variables sampled from a normal distribution to obtain a first intermediate three-dimensional garment model; and rendering the first intermediate three-dimensional garment model using a first renderer with the camera parameters corresponding to each target image frame as the viewpoint to obtain a target three-dimensional garment model corresponding to each target image frame. The contour mask and normal map of the first intermediate 3D clothing model corresponding to the target image of each frame are obtained; the latent variables are updated with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each target image of each frame and the contour mask of the clothing to be reconstructed, and the gap between the normal map of the first intermediate 3D clothing model corresponding to each target image of each frame and the normal map of the clothing to be reconstructed; the initial 3D clothing model is deformed according to the updated latent variables to obtain the second intermediate 3D clothing model; the target 3D clothing model corresponding to the clothing to be reconstructed is determined according to the second intermediate 3D clothing model.
[0009] In some embodiments, updating the latent variables with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, and the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, includes updating the latent variables with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, and the regularization term of the latent variables.
[0010] In some embodiments, updating the latent variables with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, and the regularization term of the latent variables, includes: determining the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, as a first gap; determining a first loss value based on the average of multiple first gaps corresponding to multiple frames of the target image; determining the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, as a second gap; determining a second loss value based on the average of multiple second gaps corresponding to multiple frames of the target image; determining a third loss value based on the regularization term of the latent variables; and updating the latent variables with the goal of reducing the first loss value, the second loss value, and the third loss value.
[0011] In some embodiments, the first loss value, the second loss value, and the third loss value have different weights. In some embodiments, determining the target 3D clothing model corresponding to the clothing to be reconstructed based on the second intermediate 3D clothing model includes: rendering the second intermediate 3D clothing model using a second renderer with the camera parameters corresponding to each frame of the target image as the viewpoint to obtain the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image; determining the difference between the Laplacian coordinates of each vertex of the second intermediate 3D clothing model and the Laplacian coordinates of each vertex of the first intermediate 3D clothing model as the Laplacian loss; for each edge, determining the mean square error between the length of each edge of the second intermediate 3D clothing model and the average length of all edges; updating the second intermediate 3D clothing model with the goal of reducing the difference between the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, the Laplacian loss corresponding to each vertex, and the mean square error corresponding to each edge, to obtain the target 3D clothing model corresponding to the clothing to be reconstructed.
[0012] In some embodiments, updating the second intermediate 3D clothing model to obtain a target 3D clothing model corresponding to the clothing to be reconstructed, with the goal of reducing the difference between the normal graph of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal graph of the clothing to be reconstructed, the Laplacian loss corresponding to each vertex, and the mean square error corresponding to each edge, includes: determining the difference between the normal graph of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal graph of the clothing to be reconstructed, as a third difference; determining a fourth loss value based on the average of multiple third differences corresponding to multiple frames of the target image; determining a fifth loss value based on the average of multiple Laplacian losses corresponding to multiple vertices of the second intermediate 3D clothing model; determining a sixth loss value based on the average of the mean square errors corresponding to multiple edges of the second intermediate 3D clothing model; and updating the second intermediate 3D clothing model with the goal of reducing the fourth loss value, the fifth loss value, and the sixth loss value, to obtain a target 3D clothing model corresponding to the clothing to be reconstructed.
[0013] In some embodiments, the fourth loss value, the fifth loss value, and the sixth loss value have different weights.
[0014] In some embodiments, deforming the initial 3D clothing model based on latent variables sampled from a normal distribution to obtain a first intermediate 3D clothing model includes: obtaining initial deformation information of the initial 3D clothing model using a decoder based on the latent variables sampled from a normal distribution; and deforming the initial 3D clothing model using the initial deformation information to obtain a first intermediate 3D clothing model; and / or deforming the initial 3D clothing model based on updated latent variables to obtain a second intermediate 3D clothing model includes: obtaining target deformation information of the initial 3D clothing model using the decoder based on the updated latent variables; and deforming the initial 3D clothing model using the target deformation information to obtain a second intermediate 3D clothing model.
[0015] According to a second aspect of this disclosure, an apparatus for reconstructing a three-dimensional clothing model is provided, comprising: an acquisition module configured to acquire a monocular video of a stationary human body wearing clothing to be reconstructed, the monocular video including multiple frames of original images; a selection module configured to select multiple target images from the multiple frames of original images, wherein the number of the multiple target images is less than or equal to a number threshold and the capture range of the multiple target images of the clothing to be reconstructed is greater than or equal to a range threshold, and the number of the multiple target images is less than the number of the multiple original images; and a three-dimensional reconstruction module configured to perform three-dimensional reconstruction of the clothing to be reconstructed based on the multiple target images to obtain a target three-dimensional clothing model corresponding to the clothing to be reconstructed.
[0016] According to a third aspect of this disclosure, an apparatus for reconstructing a three-dimensional clothing model is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the method for reconstructing a three-dimensional clothing model as described in any of the above embodiments based on instructions stored in the memory.
[0017] According to a fourth aspect of this disclosure, a computer-storeable medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method for reconstructing a three-dimensional clothing model as described in any of the above embodiments.
[0018] In the above embodiments, the complexity of reconstructing 3D clothing models based on multiple perspectives can be reduced. Attached Figure Description
[0019] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.
[0020] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:
[0021] Figure 1 This is a flowchart illustrating a method for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure;
[0022] Figure 2 This is a flowchart illustrating a method for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure;
[0023] Figure 3 This is a block diagram illustrating an apparatus for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure;
[0024] Figure 4 This is a block diagram illustrating an apparatus for reconstructing a three-dimensional clothing model according to other embodiments of the present disclosure;
[0025] Figure 5 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation
[0026] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0027] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0028] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0030] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0031] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0032] The following will combine Figure 1 , Figure 2 The present disclosure describes in detail a method for reconstructing a three-dimensional clothing model in some embodiments.
[0033] Figure 1 This is a flowchart illustrating a method for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure.
[0034] Figure 2 This is a flowchart illustrating a method for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure.
[0035] like Figure 1 As shown, the method for reconstructing a 3D clothing model includes: step S110, acquiring a monocular video of a stationary human body wearing the clothing to be reconstructed around its perimeter, the monocular video including multiple frames of original images; step S120, selecting multiple target images from the multiple frames of original images, wherein the number of the multiple target images is less than or equal to a number threshold and the capture range of the multiple target images of the clothing to be reconstructed is greater than or equal to a range threshold, and the number of the multiple target images is less than the number of the multiple original images; and step S130, performing 3D reconstruction of the clothing to be reconstructed based on the multiple target images to obtain a target 3D clothing model corresponding to the clothing to be reconstructed. In some embodiments, the method for reconstructing a 3D clothing model is performed by an apparatus for reconstructing a 3D clothing model.
[0036] In the above embodiments, monocular video is used in conjunction with a keyframe selection mechanism to automatically select multiple target images that can reflect complementary geometric information. This enables the 3D reconstruction of the garment to be reconstructed. This not only eliminates the need for complex camera systems or demanding photographic requirements but also reduces the amount of image processing required during reconstruction, thus lowering the complexity of multi-view 3D garment model reconstruction. Complementary geometric information refers to the fact that a single image cannot capture all the shape information of the garment, while image frames from different viewpoints can capture different shape information, exhibiting complementarity. The keyframes here are the selected multiple target images.
[0037] Furthermore, compared to single-viewpoint clothing reconstruction methods and single-camera-based clothing reconstruction methods, the method for reconstructing 3D clothing models in this disclosure can alleviate the occlusion and information loss problems existing in single-viewpoint and single-camera-based clothing reconstruction methods. Single-viewpoint and single-camera-based clothing reconstruction methods use a single RGB image for clothing reconstruction.
[0038] In step S110, a monocular video is acquired that surrounds a stationary human body wearing the clothing to be reconstructed, the monocular video comprising multiple frames of original images. In some embodiments, the monocular video is captured using a single RGB camera that surrounds the stationary human body wearing the clothing to be reconstructed. Figure 2 As shown, monocular video is represented by V.
[0039] In step S120, multiple target images are selected from the multiple original images. The number of the multiple target images is less than or equal to a number threshold, and the capture range of the multiple target images of the garment to be reconstructed is greater than or equal to a range threshold. The number of the multiple target images is less than the number of the multiple original images.
[0040] In some embodiments, selecting multiple target images from the multiple original images includes the following steps 1)-4).
[0041] In step 1), human pose estimation is performed on each original image frame to obtain the shape parameters, pose parameters, and camera parameters of the SMPL (Skinned Multi-Person Linear) model corresponding to each original image frame. In some embodiments, such as Figure 2 As shown, in the parameter estimation stage, the human body 3D estimation module M is used. P The module estimates the camera parameters K, SMPL human shape parameters β, and pose parameters θ for each frame of the original image. For example, the human 3D estimation module uses the PyMAF (Pyramidal Mesh Alignment Feedback) method to estimate the shape parameters of the human SMPL model in the image. Attitude parameters And camera parameter K.
[0042] In some embodiments, a weak perspective camera model is used, and the camera parameters K include global rotation parameters. Translation parameters and scaling parameters
[0043] In step 2), the SMLP model is determined based on the average values of multiple shape parameters and multiple pose parameters corresponding to the multiple original images. In some embodiments, the SMLP parameters of different frames are averaged to obtain the human body 3D SMLP model H(β,θ). Since the subject is stationary, the average value of the prediction results for different frames is taken as the shape parameter β and pose parameter θ of the human body captured in the video.
[0044] In step 3), based on the determined SMLP model and the camera parameters corresponding to each original image frame, the human body surface area visible to the camera in each original image frame is determined, and the human body surface area represents the capture range of the clothing to be reconstructed in each original image frame.
[0045] In step 4), multiple target images are selected from the multiple original images in descending order of the area of the human body surface region. The capture range of the clothing to be reconstructed by the multiple target images is the area of the union of the human body surface regions visible to the camera in the multiple target images.
[0046] refer to Figure 2 In the keyframe extraction stage, the keyframe extraction module M is used. K The human body model H selects X key image frames (i.e., multiple target images) with complementary geometric information, resulting in {I1, I2, ..., I...} X}, as a set of keyframes.
[0047] The keyframe selection process is performed iteratively, for example, as follows: for each frame of the original image I i Using the SMPL human model H(β,θ) corresponding to the human pose parameters θ and shape parameters β, and the camera parameters K... i Determine the visible area of the human body surface. Since the template for the SMPL human body model is known, areas that typically do not cover clothing, such as the head and hands, can be removed beforehand from the SMPL template. Next, keyframes are selected iteratively. In the first iteration, the original image with the largest visible human body area is chosen as the first keyframe (target image) I1, and the human body surface area corresponding to the first keyframe is used as the initial coverage area F. *In subsequent iterations, the remaining original images are traversed to select the one that makes F... i ∪F * The frame with the largest coverage area is selected as the keyframe (target image) for the i-th iteration, and the coverage area F is updated. * For F i ∪F * The iteration stops when the preset maximum number of iterations is reached (i.e., the number of selected keyframes reaches a threshold), or F... * It stops when the coverage area reaches a preset percentage.
[0048] In the above embodiments, by keyframe selection, a small number of image frames with complementary geometric information are extracted from the numerous image frames contained in the input video V captured around a stationary dressed human body. Using as few image frames as possible, the image capture range can cover the surface of the clothing to the maximum extent, minimizing information loss in 3D clothing reconstruction and reducing the complexity of subsequent multi-view 3D clothing model optimization. To effectively estimate the capture range of the clothing in each captured image, since there is no 3D ground truth, this embodiment uses the SMPL 3D human body parameter model as a proxy. For each image, the visible range of the SMPL 3D human body model is calculated, and an iterative mechanism selects N image frames {I1, I2, ..., I...} with complementary capture ranges. X This set of keyframes covers as much surface information as possible while reducing the computational cost of subsequent multi-view geometry optimization. 3D ground truth refers to the 3D model of the reconstructed object (e.g., an .obj file).
[0049] In step S130, the garment to be reconstructed is reconstructed in three dimensions based on the multi-frame target images to obtain a target three-dimensional garment model corresponding to the garment to be reconstructed.
[0050] In some embodiments, the three-dimensional reconstruction of the garment to be reconstructed based on the multi-frame target images can be achieved through the following steps (1)-(7) to obtain a target three-dimensional garment model corresponding to the garment to be reconstructed.
[0051] In step (1), the camera parameters corresponding to each frame of the target image and the initial 3D clothing model corresponding to the clothing to be reconstructed are obtained. For example, in the previous embodiment, the camera parameters corresponding to each frame of the original image have been determined, so the camera parameters corresponding to each frame of the target image can be obtained directly here.
[0052] In step (2), the contour mask and normal map of the garment to be reconstructed in each frame of the target image are determined.
[0053] In some embodiments, reference Figure 2In the image feature extraction stage, for the extracted i-th target image (keyframe) I i Using the clothing segmentation module M S Segmenting it yields the clothing segment S in the image. i This refers to the contour mask of the clothing to be reconstructed in the i-th target image. For example, the Self-Correction for Human Parsing (SCHP) network can be used for clothing segmentation. Clothing segmentation refers to outlining the overall outer contour of the object to be reconstructed in the image. The clothing segmentation image only needs to distinguish the foreground pixels belonging to the clothing, and does not need to segment different parts of the clothing.
[0054] In some embodiments, reference Figure 2 In the image feature extraction stage, the normal estimation module M is used. N Estimate the normal map N of human body parts in the image i For example, the normal estimation module in the PIFuHD (Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization) method can be used for normal estimation. The role of the normal map is to identify details of clothing, such as wrinkles and folds, in the image.
[0055] In the above embodiments, for keyframe I i After the image feature extraction stage, a set of camera parameters, clothing segmentation (contour mask), and normal maps {K} is obtained. i ,S i N i That is, for the keyframe set {I1, I2, ..., I...} X Using the clothing segmentation module M S Obtain the set of mask images {S1,S2,…,S} for the clothing region in the image. X}, using the normal estimation module M N We obtain the normal graph set {N1, N2, ..., N} X Different target images represent different perspectives, such as... Figure 2 Viewpoint 1, viewpoint 2, and viewpoint 3, etc.
[0056] In step (3), the initial three-dimensional clothing model is deformed based on the latent variables sampled from the normal distribution to obtain the first intermediate three-dimensional clothing model.
[0057] In some embodiments, deforming the initial three-dimensional clothing model based on latent variables sampled from a normal distribution to obtain a first intermediate three-dimensional clothing model includes: obtaining initial deformation information of the initial three-dimensional clothing model using a decoder based on latent variables sampled from a normal distribution (e.g., randomly sampled); and deforming the initial three-dimensional clothing model using the initial deformation information to obtain the first intermediate three-dimensional clothing model.
[0058] For example, refer to Figure 2 In the geometric optimization stage of the multi-view clothing model, latent variables Z are sampled from a normal distribution and input into the decoder Dec to generate initial deformation information. This initial deformation information is denoted as Dec(Z). Based on the initial deformation information Dec(Z), the initial 3D clothing model T is deformed to obtain the first intermediate 3D clothing model Dec(Z)+T.
[0059] In step (4), the first intermediate 3D clothing model is rendered using the camera parameters corresponding to each frame of the target image as the viewpoint, and the first renderer is used to obtain the contour mask and normal map of the first intermediate 3D clothing model corresponding to each frame of the target image.
[0060] For example, using camera parameter K i From the perspective of the first intermediate 3D clothing model, the contour mask is represented as Rd. mask (Dec(Z)+T,K i ), where i is the sequence number of the key image frame, and Rd mask The rendering function uses the differentiable rendering algorithm Kaolin, and its input includes the camera parameters K of keyframe i. i And the 3D clothing model Dec(Z)+T corresponding to Z, where T is the template model, Rd mask The output is a contour mask of a 3D clothing model projected onto a 2D image.
[0061] For example, using camera parameter K i From the perspective of the first intermediate 3D clothing model, the normal diagram is represented as Rd. normal (Dec(Z)+T,K i ), where i is the sequence number of the key image frame, and Rd normal The rendering function uses the differentiable rendering algorithm Kaolin, and its input includes the camera parameters K of keyframe i. i And the 3D clothing model Dec(Z)+T corresponding to Z, where T is the template model, Rd normal The output is a normal map of a 3D clothing model.
[0062] In some embodiments, reference Figure 2 In the shape fitting module MF In this paper, given a clothing template model T, a variational autoencoder (VAE) from PERGAMO (Personalized 3D garments from monocular video) is used to generate the deformation of T from the statistical latent variable space. Specifically, the VAE is trained on a synthetic clothing dataset to learn a generative model. The VAE consists of an encoder (Enc) and a decoder (Dec), both of which are multi-layer fully connected networks. The VAE learns the 3D shape distribution of the clothing and uses low-dimensional latent variables Z to represent the vertex deformation of the high-dimensional clothing template model T. After training the VAE on the synthetic clothing dataset, the parameters of the neural network are fixed. The decoder (Dec) takes latent variables Z sampled from a normal distribution as input and outputs the vertex deformation of the clothing template model T. This VAE learns the statistical shape prior of clothing deformation using the clothing dataset, avoiding unreasonable clothing deformation. Shape fitting module M F Includes the first renderer.
[0063] In step (5), the latent variables are updated with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed in each frame of the target image, and the gap between the normal map of the first intermediate 3D clothing model and the normal map of the clothing to be reconstructed in each frame of the target image.
[0064] The difference between the contour mask of the first intermediate 3D clothing model and the contour mask of the clothing to be reconstructed in each frame of the target image is the difference between the contour mask of the first intermediate 3D clothing model with the camera parameters corresponding to each frame of the target image as the viewpoint and the contour mask of the clothing to be reconstructed in each frame of the target image. It can also be described as the difference between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed.
[0065] In some embodiments, the latent variables are updated with the aim of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed in each frame of the target image, the gap between the normal map of the first intermediate 3D clothing model and the normal map of the clothing to be reconstructed in each frame of the target image, and the regularization term of the latent variables.
[0066] In some embodiments, the latent variables can be updated with the aim of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed in each frame of the target image, the gap between the normal map of the first intermediate 3D clothing model and the normal map of the clothing to be reconstructed in each frame of the target image, and the regularization term of the latent variables.
[0067] First, the difference between the contour mask of the first intermediate 3D clothing model and the contour mask of the clothing to be reconstructed in each frame of the target image is determined as the first difference. For example, for the i-th frame of the target image, the first difference, i.e., the mask error loss function, is expressed as: S i For module M s Output clothing segments.
[0068] Secondly, a first loss value is determined based on the average of multiple first differences corresponding to multiple target images. For example, the first loss value is expressed as... X represents the number of target images in multiple frames.
[0069] Next, the difference between the normal map of the first intermediate 3D clothing model and the normal map of the clothing to be reconstructed in each frame of the target image is determined as the second difference. For example, for the i-th frame of the target image, the first difference, i.e., the normal error loss function, is expressed as: N i For module M N The output normal graph. During optimization, only N is considered. i The clothing section.
[0070] Then, a second loss value is determined based on the average of multiple second gaps corresponding to multiple target images. For example, the second loss value is expressed as...
[0071] Then, based on the regularization term of the latent variables, the third loss value is determined. For example, the third loss value is represented as l reg =|Z| 2 .
[0072] Finally, the latent variables are updated with the goal of reducing the first loss value, the second loss value, and the third loss value.
[0073] In some embodiments, the first loss value, the second loss value, and the third loss value have different weights. For example, the weight of the first loss value is λ1, the weight of the second loss value is λ2, and the weight of the third loss value is λ3. In this case, the weighted sum of the first loss value, the second loss value, and the third loss value is represented as the shape fitting loss, denoted as... For example, by minimizing the loss function L fit (Z) obtains the updated Z.
[0074] In the above embodiments, the loss function L in the shape fitting stage... fit The optimization target of (Z) is the input Z, L of the decoder Dec. fit Includes and L reg Three items. Among them Used to measure the deformed T in camera parameter K i The normal map rendered from the perspective of [viewpoint], and the normal map N. i The differences. Used to measure the deformed T in camera parameter K i The mask image rendered from the perspective of clothing segmentation S i The difference. L reg λ1, λ2, and λ3 are regularization terms to prevent Z from deviating from the standard distribution. X represents the number of key image frames, i is the index of the key image frame, and λ1, λ2, and λ3 are the weights of each loss function.
[0075] In step (6), the initial 3D clothing model is deformed based on the updated latent variables to obtain a second intermediate 3D clothing model. For example, the initial 3D clothing model T is deformed based on the updated Z to obtain the second intermediate 3D clothing model, and the corresponding initial clothing reconstruction result is G. * .
[0076] In some embodiments, deforming the initial three-dimensional clothing model according to the updated latent variables to obtain a second intermediate three-dimensional clothing model includes: obtaining target deformation information of the initial three-dimensional clothing model using the decoder according to the updated latent variables; and deforming the initial three-dimensional clothing model using the target deformation information to obtain a second intermediate three-dimensional clothing model.
[0077] In step (7), a target three-dimensional clothing model corresponding to the clothing to be reconstructed is determined based on the second intermediate three-dimensional clothing model.
[0078] In some embodiments, it can be achieved by utilizing Figure 2 The detail extraction module M shown D Perform the following steps (a)-(d) to determine the target three-dimensional clothing model corresponding to the clothing to be reconstructed based on the second intermediate three-dimensional clothing model.
[0079] In step (a), the second intermediate 3D clothing model is rendered using the camera parameters corresponding to each frame of the target image as the viewpoint, and the second renderer is used to obtain the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image.
[0080] For example, using camera parameter K i From the perspective of the second intermediate 3D clothing model, the normal diagram is represented as Rd. normal (G * ,K i Rd normal (G * ,K i () is based on camera parameters K i From the perspective of the 3D clothing model G * Perform rendering.
[0081] In step (b), the difference between the Laplacian coordinates of each vertex of the second intermediate 3D clothing model and the Laplacian coordinates of each vertex of the first intermediate 3D clothing model is determined as the Laplacian loss.
[0082] For example, the Laplace loss corresponding to vertex position x is represented as L. lap =‖δ′ x -δ x || 2 , where δ′ x and δ x For the three-dimensional model G * Let x be the Laplacian coordinates of a vertex position after deformation (in the second intermediate 3D clothing model) and before deformation (in the first intermediate 3D clothing model). The definition of Laplacian coordinates is... Where '∈N(x) represents the nearest neighbor of vertex x, and ||N(x)|| represents the number of nearest neighbor vertices. L lap To define in the three-dimensional model G * The Laplace loss on the vertex is defined as the difference between the displacement of the vertex and the average displacement of the nearest neighboring vertices. This loss term serves to keep the deformation smooth.
[0083] In step (c), for each side, the root mean square deviation of the length of each side of the second intermediate 3D clothing model from the average length of all sides is determined. For example, the root mean square deviation for each side is expressed as... L edge G was calculated * The length of each side and the average side length The mean squared error, N e G represents * The number of edges. L edgeThe edge loss term is defined as the deviation of each edge length from the average edge length in the 3D mesh model, and its purpose is to avoid unreasonable vertex displacement.
[0084] In step (d), the second intermediate 3D clothing model is updated with the goal of reducing the difference between the normal graph of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal graph of the clothing to be reconstructed in each frame of the target image, the Laplacian loss corresponding to each vertex, and the root mean square error corresponding to each edge, so as to obtain the target 3D clothing model corresponding to the clothing to be reconstructed.
[0085] In some embodiments, the second intermediate 3D clothing model can be updated with the goal of reducing the difference between the normal graph of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal graph of the clothing to be reconstructed in each frame of the target image, the Laplacian loss corresponding to each vertex, and the mean square error corresponding to each edge, so as to obtain the target 3D clothing model corresponding to the clothing to be reconstructed.
[0086] First, the difference between the normal map of the second intermediate 3D clothing model and the normal map of the clothing to be reconstructed in each frame of the target image is determined as the third difference. For example, for the i-th frame of the target image, the third difference is represented as...
[0087] Secondly, a fourth loss value is determined based on the average of multiple third gaps corresponding to multiple target images across multiple frames. For example, the fourth loss value is represented as...
[0088] Next, a fifth loss value is determined based on the average of multiple Laplacian losses corresponding to multiple vertices of the second intermediate 3D clothing model. For example, the fifth loss value is expressed as... Where Q is the total number of vertices.
[0089] Then, a sixth loss value is determined based on the average of the root mean square errors corresponding to the multiple edges of the second intermediate 3D clothing model. For example, the sixth loss value is expressed as... Where P is the total number of edges.
[0090] Finally, with the goal of reducing the fourth, fifth, and sixth loss values, the second intermediate 3D clothing model is updated to obtain the target 3D clothing model corresponding to the clothing to be reconstructed.
[0091] In some embodiments, the fourth, fifth, and sixth loss values have different weights. For example, the weight of the fourth loss value is λ4, the weight of the fifth loss value is λ5, and the weight of the sixth loss value is λ6. In this case, the weighted sum of the fourth, fifth, and sixth loss values is represented as the detail extraction loss, as follows: For example, by minimizing the loss function L detail (G * Update the second intermediate three-dimensional clothing model to obtain the target three-dimensional clothing model G corresponding to the clothing to be reconstructed.
[0092] In the above embodiments, after the shape fitting stage, a generative model based on clothing templates is used to optimize the template deformation in the statistical deformation space, ensuring that the clothing contour and normal in the multi-view images are consistent. This avoids unreasonable deformation and improves the robustness of the fitting. However, since the degrees of freedom of clothing deformation in the statistical space are limited, in the detail extraction stage, the deformation or displacement of each vertex of the 3D clothing is directly optimized, so that the 3D clothing reconstruction result can have deformation details such as folds consistent with the input image.
[0093] exist Figure 2 In the geometric optimization stage of the multi-view clothing model, the clothing template T is fitted to the multi-view segmentation map and normal map through two steps: shape fitting and detail extraction. In the shape fitting module M... F In the process, the decoder Dec of the statistical generative model is used to optimize the deformation of the clothing template, and the initial reconstruction result G of the clothing is obtained. * In the detail extraction module M D In the middle, for G * The vertex displacement is directly optimized to further fit the geometric details on the normal map, and finally the 3D clothing reconstruction result G is obtained.
[0094] In the geometric optimization stage of the multi-view clothing model, a 3D clothing template model is used. Through two steps of template-based deformation, the template model T is fitted to key image frames from multiple perspectives. In the first step, shape fitting, this disclosure utilizes a PERGAMO-based variational autoencoder to generate a model. By optimizing the latent variables of the variational autoencoder input, the deformation of the clothing template is optimized. The deformed clothing template is then compared with the clothing segmentation {S1, S2, ..., S...} of the multi-view keyframes. X The outline of the garment is consistent with the outline of the normal map. Furthermore, the optimization objective also includes consistency with the multi-view normal map. In the second step of detail extraction, the vertex displacement of the 3D garment template model is directly optimized to ensure that the multi-view rendering results of the garment reconstruction are consistent with the normal map. Figure 1 The result of the 3D clothing reconstruction can reflect the wrinkles in the RGB photo, and the final optimized result is the 3D clothing reconstruction result.
[0095] In contrast, multi-view clothing reconstruction methods in related technologies either rely on complex camera arrays or have high requirements for multi-view shooting. The embodiments of this disclosure only require a video taken around a static reconstruction object. Through a key frame selection mechanism, key image frames that can reflect complementary geometric information are automatically extracted for multi-view 3D clothing model reconstruction, thereby reducing the complexity of multi-view reconstruction.
[0096] Furthermore, related multi-view clothing reconstruction methods may generate distorted clothing structures due to the lack of prior geometric structure knowledge. This disclosure utilizes a 3D clothing template and a generative model based on a variational autoencoder to learn prior geometric structure from a clothing dataset, and uses its decoder for fitting and reconstructing a multi-view 3D clothing model. This disclosure employs two steps—shape fitting and detail extraction—to deform the 3D clothing template and fit it with the geometric information from the multi-view images to obtain the final reconstruction result.
[0097] This disclosure utilizes a statistical clothing template model to optimize geometric information from multiple perspectives. It leverages the prior structural information of the statistical template to enhance the robustness of clothing reconstruction, resulting in reconstructed images with contours and folds that match the geometric information from multiple perspectives. Unlike regression-based clothing reconstruction methods that require a dataset containing two-dimensional images and corresponding ground truth three-dimensional models for training, this disclosure does not require ground truth three-dimensional models of clothing.
[0098] Figure 3 This is a block diagram illustrating an apparatus for reconstructing a three-dimensional clothing model according to some embodiments of the present disclosure.
[0099] like Figure 3 As shown, the device 3 for reconstructing a three-dimensional clothing model includes an acquisition module 31, a selection module 32, and a three-dimensional reconstruction module 33.
[0100] The acquisition module 31 is configured to acquire a monocular video that surrounds a stationary human body wearing the clothing to be reconstructed, the monocular video comprising multiple frames of raw images, for example, by performing... Figure 1 The step S110 is shown.
[0101] Selection module 32 is configured to select multiple target images from the multiple original images, wherein the number of target images is less than or equal to a number threshold and the capture range of the target images for the garment to be reconstructed is greater than or equal to a range threshold, and the number of target images is less than the number of original images, for example, performing the following... Figure 1 The step S120 shown.
[0102] The 3D reconstruction module 33 is configured to perform 3D reconstruction of the garment to be reconstructed based on the multi-frame target images, to obtain a target 3D garment model corresponding to the garment to be reconstructed, for example, by performing... Figure 1 The step S130 shown.
[0103] Figure 4 This is a block diagram illustrating an apparatus for reconstructing a three-dimensional clothing model according to other embodiments of the present disclosure.
[0104] like Figure 4 As shown, the apparatus 4 for reconstructing a three-dimensional clothing model includes a memory 41 and a processor 42 coupled to the memory 41. The memory 41 is used to store instructions for performing a method for reconstructing a three-dimensional clothing model corresponding to an embodiment. The processor 42 is configured to perform a method for reconstructing a three-dimensional clothing model in any of the embodiments of this disclosure based on the instructions stored in the memory 41.
[0105] Figure 5 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
[0106] like Figure 5 As shown, the computer system 50 can be represented in the form of a general computing device. The computer system 50 includes a memory 510, a processor 520, and a bus 500 connecting different system components.
[0107] The memory 510 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for performing at least one of the methods for reconstructing a 3D clothing model in a corresponding embodiment. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.
[0108] The processor 520 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by executing instructions in the central processing unit (CPU) memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.
[0109] Bus 500 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.
[0110] The computer system 50 may also include an input / output interface 530, a network interface 540, and a storage interface 550. These interfaces 530, 540, and 550, as well as the memory 510 and processor 520, can be connected via a bus 500. The input / output interface 530 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 540 provides a connection interface for various networked devices. The storage interface 550 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.
[0111] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.
[0112] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.
[0113] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.
[0114] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.
[0115] The method and apparatus for reconstructing three-dimensional clothing models, as well as the computer-storable medium described in the above embodiments, can reduce the complexity of reconstructing three-dimensional clothing models from multiple perspectives.
[0116] This concludes the detailed description of the method and apparatus for reconstructing three-dimensional clothing models according to this disclosure, as well as the computer-storable medium. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.
Claims
1. A method for reconstructing a three-dimensional clothing model, comprising: Acquire a monocular video that surrounds a stationary human body wearing the clothing to be reconstructed, the monocular video comprising multiple frames of raw images; From the multiple original images, select multiple target images, wherein the number of the multiple target images is less than or equal to a number threshold and the capture range of the multiple target images of the garment to be reconstructed is greater than or equal to a range threshold, and the number of the multiple target images is less than the number of the multiple original images; Based on the multiple target images, the garment to be reconstructed is reconstructed in three dimensions to obtain a target three-dimensional garment model corresponding to the garment to be reconstructed, including: Obtain the camera parameters corresponding to each frame of the target image and the initial 3D clothing model corresponding to the clothing to be reconstructed; Determine the contour mask and normal map of the garment to be reconstructed in each frame of the target image; Based on the latent variables sampled from the normal distribution, the initial three-dimensional clothing model is deformed to obtain the first intermediate three-dimensional clothing model; Using the camera parameters corresponding to each frame of the target image as the viewpoint, the first renderer is used to render the first intermediate 3D clothing model to obtain the contour mask and normal map of the first intermediate 3D clothing model corresponding to each frame of the target image. The latent variables are updated with the goal of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, and the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed. Based on the updated latent variables, the initial three-dimensional clothing model is deformed to obtain a second intermediate three-dimensional clothing model. Based on the second intermediate 3D clothing model, a target 3D clothing model corresponding to the clothing to be reconstructed is determined, including: using the camera parameters corresponding to each frame of the target image as the viewpoint, rendering the second intermediate 3D clothing model using a second renderer to obtain the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image; determining the difference between the Laplacian coordinates of each vertex of the second intermediate 3D clothing model and the Laplacian coordinates of each vertex of the first intermediate 3D clothing model as the Laplacian loss; for each edge, determining the mean square error between the length of each edge of the second intermediate 3D clothing model and the average length of all edges, with the goal of reducing the difference between the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, the Laplacian loss corresponding to each vertex, and the mean square error corresponding to each edge; updating the second intermediate 3D clothing model to obtain the target 3D clothing model corresponding to the clothing to be reconstructed.
2. The method for reconstructing a three-dimensional clothing model according to claim 1, wherein, Selecting multiple target images from the multiple original images includes: Human pose estimation is performed on each original image frame to obtain the shape parameters, pose parameters, and camera parameters of the skinned multi-person linear SMPL model corresponding to each original image frame; The SMLP model is determined based on the average value of multiple shape parameters and the average value of multiple pose parameters corresponding to the multiple original images. Based on the determined SMLP model and the camera parameters corresponding to each original image frame, the human body surface area visible to the camera in each original image frame is determined, and the human body surface area represents the capture range of the clothing to be reconstructed in each original image frame. According to the order of the area of the human body surface region from largest to smallest, multiple target images are selected from the multiple original images. The capture range of the clothing to be reconstructed by the multiple target images is the area of the union of the human body surface regions visible to the camera in the multiple target images.
3. The method for reconstructing a three-dimensional clothing model according to claim 1, wherein, The latent variables are updated with the aim of reducing the difference between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, and the difference between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed. The latent variables are updated with the aim of reducing the gap between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, the gap between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, and the regularization term of the latent variables.
4. The method for reconstructing a three-dimensional clothing model according to claim 3, wherein, The latent variables are updated with the aim of reducing the difference between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed, the difference between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed, and the regularization term of the latent variables. The first gap is defined as the difference between the contour mask of the first intermediate 3D clothing model corresponding to each frame of the target image and the contour mask of the clothing to be reconstructed. The first loss value is determined based on the average of multiple first gaps corresponding to multiple target images; The difference between the normal map of the first intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed is determined as the second difference; The second loss value is determined based on the average of multiple second gaps corresponding to multiple target images; The third loss value is determined based on the regularization term of the latent variables; The latent variables are updated with the goal of reducing the first loss value, the second loss value, and the third loss value.
5. The method for reconstructing a three-dimensional clothing model according to claim 4, wherein, The first loss value, the second loss value, and the third loss value have different weights.
6. The method for reconstructing a three-dimensional clothing model according to claim 1, wherein, With the objectives of reducing the difference between the normal graph of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal graph of the clothing to be reconstructed, the Laplacian loss corresponding to each vertex, and the root mean square error corresponding to each edge, the second intermediate 3D clothing model is updated to obtain the target 3D clothing model corresponding to the clothing to be reconstructed, including: The difference between the normal map of the second intermediate 3D clothing model corresponding to each frame of the target image and the normal map of the clothing to be reconstructed is determined as the third difference; The fourth loss value is determined based on the average of multiple third gaps corresponding to multiple target images; The fifth loss value is determined based on the average of multiple Laplacian losses corresponding to multiple vertices of the second intermediate 3D clothing model; The sixth loss value is determined based on the average of the root mean square errors corresponding to the multiple edges of the second intermediate three-dimensional clothing model; With the goal of reducing the fourth, fifth, and sixth loss values, the second intermediate three-dimensional clothing model is updated to obtain the target three-dimensional clothing model corresponding to the clothing to be reconstructed.
7. The method for reconstructing a three-dimensional clothing model according to claim 6, wherein, The fourth loss value, the fifth loss value, and the sixth loss value have different weights.
8. The method for reconstructing a three-dimensional clothing model according to claim 1, wherein, The process of deforming the initial 3D clothing model based on latent variables sampled from a normal distribution to obtain a first intermediate 3D clothing model includes: obtaining initial deformation information of the initial 3D clothing model using a decoder based on the latent variables sampled from the normal distribution; and deforming the initial 3D clothing model using the initial deformation information to obtain the first intermediate 3D clothing model; and / or The process of deforming the initial 3D clothing model based on the updated latent variables to obtain a second intermediate 3D clothing model includes: obtaining target deformation information of the initial 3D clothing model using the decoder based on the updated latent variables; and deforming the initial 3D clothing model using the target deformation information to obtain a second intermediate 3D clothing model.
9. An apparatus for reconstructing a three-dimensional clothing model, used to implement the method for reconstructing a three-dimensional clothing model as described in claim 1.
10. An apparatus for reconstructing a three-dimensional clothing model, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method for reconstructing a three-dimensional clothing model as described in any one of claims 1 to 8, based on instructions stored in the memory.
11. A computer-storeable medium having stored thereon computer program instructions that, when executed by a processor, implement the method for reconstructing a three-dimensional clothing model as described in any one of claims 1 to 8.
