Linearly differentiable parametric clothing model making method and parameter optimization method

Through linearly differentiable parameterized clothing model production method and gradient descent optimization technology, the high precision and high realistic reconstruction of clothing models in dynamic scenarios are solved, efficient parameterization and optimization of clothing and pants are realized, and dynamic reconstruction is improved. accuracy and continuity.

CN112102470BActive Publication Date: 2024-11-08YIYUN ZHIXING (SHENZHEN) TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202010828692.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-18
Publication Date
2024-11-08
Estimated Expiration
2040-08-18

AI Technical Summary

Technical Problem

Existing human body recognition and three-dimensional reconstruction algorithms lack accuracy and realism when dealing with challenging scenes, especially in dynamic scenes, the parameterization and optimization of clothing models are difficult to achieve high-precision and high-reality reconstruction.

Method used

Using a linearly differentiable parametric clothing model production method, the geometric dimensions and node positions of the clothes are determined through 11 parameters, the triangle face sheet is filled with Delaunay triangulation algorithm, and combined with the gradient descent optimization method, the loss function is designed to optimize the three-dimensional clothing. and two-dimensional geometric parameters, detect collisions and build loss functions to realize dynamic reconstruction and optimization of clothing and human bodies.

Benefits of technology

Achieve high precision and high reality dual-layer model reconstruction, make full use of the robustness of static reconstruction, complete parameterization and semantics of dynamic clothing and pants, optimize their geometric parameters in real time, and improve the accuracy and continuous dynamic reconstruction. sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure GDA0005066296000000081
    Figure GDA0005066296000000081
Patent Text Reader

Abstract

The present invention provides a method for making a linearly differentiable parametric clothing model. A rough model of an object is reconstructed by using a video sequence containing a person. On this basis, a human body part and a clothing part are segmented by image segmentation technology. The clothing and the human body are modeled separately, thereby obtaining a double-layer model with higher precision and higher realism. At the same time, the present invention proposes an optimization method based on the above scheme, and a parameterization scheme for clothes and trousers, which can realize real-time dynamic optimization of geometric parameters of clothes and trousers based on three-dimensional data and two-dimensional images. The scheme of the present invention makes full use of the robustness provided by the existing deep learning scheme of static reconstruction, and completes parameterization, semanticization and other tasks in later improvements and optimizations to complete dynamic reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to computer vision, three-dimensional reconstruction and computer graphics. Background Art

[0002] At present, there are relatively mature algorithms in the fields of human body recognition and segmentation. For example, the currently best-performing human image segmentation algorithms CIHP_PGN and LIP_JPPNet can separate the human body from the background through two-dimensional images or videos, and divide the human body into several parts such as hair, face, clothes, arms, pants, feet, etc. They can have a high accuracy rate for most general scenes, but for a few challenging scenes, the algorithm will fail, resulting in low precision and low realism. Therefore, it can only be used as a universal application to a certain extent.

[0003] At the same time, there are already many highly robust algorithms for human static 3D reconstruction, such as PiFu and PAIR. The former completes a single RBG image as input and reconstructs a 3D model as output by training a deep network. The latter improves on the former and adds the estimation of SMPL human posture and shape to make the algorithm more suitable for 3D reconstruction of the human body.

[0004] This project is the research result based on the basic maturity of the above-mentioned related algorithms. Summary of the invention

[0005] The first object of the present invention is to provide a method for making a linearly differentiable parameterized clothing model with higher precision and higher realism;

[0006] The second object of the present invention is to provide a method for optimizing parameters of a linearly differentiable parametric clothing model making method for optimizing three-dimensional geometry and corresponding two-dimensional geometric parameters.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] The method for making a linearly differentiable parametric clothing model includes the following steps:

[0009] S1. The pattern of the clothing is completely determined by 11 parameters, which in turn completely determine 16 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 16 nodes divide the entire pattern into 8 sub-patterns, that is, a total of 11 parameters, completely determining the geometric size and node position of the clothing;

[0010] S2. The pattern of the pants is completely determined by 7 parameters, which in turn completely determine 10 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 10 nodes divide the entire pattern into 4 sub-patterns, that is, a total of 7 parameters, completely determining the geometric size and node position of the pants;

[0011] S3. Use the Delaunay triangulation algorithm to fill the encrypted outline with triangular faces to obtain the final clothing pattern;

[0012] S4. The corresponding (u, v) coordinates and shape function values ​​(N1, N2, N3, N4) are calculated for each vertex of the patch.

[0013] The parameter optimization method of the linearly differentiable parametric clothing model making method includes the following steps:

[0014] S1. Based on the reasonable inner human body model, a loss function is designed to optimize the coordinates of all vertices of the three-dimensional geometry of the clothing through gradient descent;

[0015] S2. While optimizing the 3D geometry of the garment, a loss function is designed to optimize the 2D geometric parameters of the garment by gradient descent;

[0016] S3. During the optimization process, quickly detect whether the clothing and the inner human body collide, and use the normal distance to construct a loss function to optimize the collision part.

[0017] Furthermore, in the above S3 collision detection, the minimum normal distance between the point to be processed and the inner layer of the human body is calculated. When it is less than a given threshold, a set gradient is added to it, with the direction being the corresponding normal direction, and acts in the final gradient descent.

[0018] The beneficial effects of the present invention are as follows: the present invention utilizes a video sequence containing characters to reconstruct a rough model of an object, and on this basis, uses image segmentation technology to segment the human body part and the clothing part, and separately models the clothing and the human body, thereby obtaining a double-layer model with higher precision and higher realism, thereby achieving the first purpose of the present invention; at the same time, the present invention proposes a new optimization method and a parameterization scheme for clothes and trousers, making full use of the robustness provided by the existing deep learning scheme for static reconstruction, and in the later improvement and optimization, completing parameterization, semanticization and other tasks to complete dynamic reconstruction, and realizing real-time dynamic optimization of the geometric parameters of clothes and trousers based on three-dimensional data and two-dimensional images, thereby achieving the second purpose of the present invention. DETAILED DESCRIPTION

[0019] Embodiment 1

[0020] The method for making a linearly differentiable parametric clothing model includes the following steps:

[0021] S1. The pattern of the clothing is completely determined by 11 parameters, which in turn completely determine 16 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 16 nodes divide the entire pattern into 8 sub-patterns, that is, a total of 11 parameters, completely determining the geometric size and node position of the clothing;

[0022] S2. The pattern of the pants is completely determined by 7 parameters, which in turn completely determine 10 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 10 nodes divide the entire pattern into 4 sub-patterns, that is, a total of 7 parameters, completely determining the geometric size and node position of the pants;

[0023] S3. Use the Delaunay triangulation algorithm to fill the encrypted outline with triangular faces to obtain the final clothing pattern;

[0024] S4. The corresponding (u, v) coordinates and shape function values ​​(N1, N2, N3, N4) are calculated for each vertex of the patch.

[0025] The parameter optimization method according to the above-mentioned linear differentiable parametric clothing model making method includes the following steps:

[0026] S1. Based on the reasonable inner human body model, a loss function is designed to optimize the coordinates of all vertices of the three-dimensional geometry of the clothing through gradient descent;

[0027] S2. While optimizing the 3D geometry of the garment, a loss function is designed to optimize the 2D geometric parameters of the garment by gradient descent;

[0028] S3. During the optimization process, quickly detect whether the clothing and the inner human body collide, and use the normal distance to construct a loss function to optimize the collision part.

[0029] Furthermore, in the above S3 collision detection, the minimum normal distance between the point to be processed and the inner layer of the human body is calculated. When it is less than a given threshold, a set gradient is added to it, the direction of which is the corresponding normal direction, and it acts in the final gradient descent.

[0030] Embodiment 2

[0031] First, the input is a video sequence of the same person, with a total of P frames. As the input image, the video needs to include the whole body of the person, which can be the front, side or back. The character's actions can be various actions in a simple standing posture, but do not include complex actions, such as somersaults, falling, etc. For each frame, through PAIR, a corresponding rough model is output through the neural network, and the parameters of the corresponding SMPL model are output at the same time. That is, there are a total of P models M_i (i=1,2,3,...,P). P groups of parameters (β,θ,r,t,s)_i (i=1,2,3,...,P), where β,θ correspond to the shape and pose parameters in the SMPL parameters respectively, r,t correspond to the rotation and translation of the model respectively, and s corresponds to the scale.

[0032] Then optimize the SMPL model and fit the model Mi (i = 1, 2, 3, ..., P). First, based on the SMPL parameters output by PAIR, optimize (β, θ, r, t, s)_i (i = 1, 2, 3, ..., P) to obtain a SMPL model with a roughly appropriate posture and body shape. Then define an offset b (6890 × 3) for each point on the SMPL model, that is, the offset of each vertex relative to the original position, optimize b separately, and obtain the fitted SMPL model. Thus, for all P models, a topologically consistent expression m_i (i = 1, 2, 3, ..., P) is obtained, and all P models are expressed by SMPL parameters and offset b.

[0033] At the same time, for each frame of the input video, CIHP_PGN and LIP_JPPNet are used to segment the corresponding clothes / pants and human body parts. Combining the previously obtained P models Mi (i = 1, 2, 3, ..., P) and the corresponding SMPL expression m (i = 1, 2, 3, ..., P), the corresponding segmentation is first performed on the corresponding SMPL model, and the model vertices are marked by the image, and the final vertex classification is counted by the mark "vote", and the SMPL expression of the clothes part mc (i = 1, 2, 3, ..., P) and the human body part ms (i = 1, 2, 3, ..., P) are obtained. Then, by finding the nearest point between the fitting SMPL model and the rough model, the classification label is directly transferred to obtain the model expression of the clothes part Mc (i = 1, 2, 3, ..., P) and the human body part Ms (i = 1, 2, 3, ..., P).

[0034] Then, the SMPL expression is optimized again through the segmented model to describe the inner human body part S_i (i=1,2,3,...,P) of the model. The clothing and pants parameterized model proposed by the present invention is used to fit the corresponding clothing and pants parts, and the collision and contact problems are handled at the same time, and the corresponding parameters of clothing and pants are optimized, and then the final clothing and pants model is optimized, and the clothing and pants parts are parameterized and expressed C_i (i=1,2,3,...,P). The details are as follows:

[0035] First of all, in order to ensure that clothes and pants can be correctly and reasonably optimized, "worn" on the human body model, and correctly "sewn", appropriate initialization work must be done. In the present invention, the human body model is first kept in a T-pose posture and a default body shape, and a front and back clothing or pants plate is placed on the front and back of the human body model respectively. In this way, a piece of clothing will be composed of two front and back plates with the same parameters, and the parts that should be "sewn" will be connected during the optimization process.

[0036] Then we start to optimize the vertex coordinates of the four plates of clothes and pants, and optimize the corresponding geometric parameters of the plates, and gradually change the posture and body shape of the human body model during the optimization process. In addition, since the geometric parameters of clothes and pants can only determine the vertex coordinates of the plates in two dimensions, while the model is three-dimensional during the actual optimization process, in order to solve this problem, the present invention proposes a new optimization scheme that allows the gradient to be transferred back to the plate geometric parameters. First, we need to define an additional two-dimensional grid, record the vertex coordinates V c_2d , V p_2d , which correspond to the results of the clothes and pants models being flattened on a two-dimensional plane. At the same time, an additional item is added to the loss function:

[0037] L d_edge =∑(||edge(V c )-edge(V c_2d )||2)+∑(||edge(V p )-edge(V p_2d )||2) (3.16)

[0039] This item makes the 3D model as similar as possible to the corresponding 2D grid after tiling. The specific operation is to take the edges of all the facets of the 3D clothes and pants model in order, subtract them one by one from the edges of the corresponding 2D grid, calculate the modulus and then sum them up. This item can ensure that as the clothes and pants models in the 3D space continue to deform, their tangential expansion and contraction deformation is constrained by the shape of the corresponding 2D plate, while the normal deformation is not constrained. The final result is that all tangential expansion and contraction deformations of the clothes and pants models in the 3D space are fully reflected on the 2D plate, and wrinkles and other phenomena may occur.

[0040] So to optimize the clothes and pants models, the total loss function is defined as follows:

[0041] Loss = w cd L cd +w constr L constr +w d_edge L d_edge +w ls L ls +w e L e +w n L n (3.17)

[0043] L cd =CD(clothes(m i ),V c )+CD(pants(m i ),V p )(3.18)

[0044] L constr =∑(||constrait_front(V c ,V p )-constrait_back(V c ,V p )||2)(3.19)

[0045] L cd is the chamfer distance between the clothes and pants model and the clothes and pants part of the corresponding SMPL model, L constr It is a constraint on the points where the front and rear panels should be stitched together.

[0046] Next, minimize the above loss function using stochastic gradient descent. At this time, the target to be optimized includes the vertex coordinates V of the clothes and pants models. c , V p , and also includes the corresponding geometric parameters param c ,param p In the actual process, these two parameters should be optimized according to different gradient sizes. That is, assuming that when optimizing the model vertex coordinates, the gradient descent ratio is 1 each time, and when optimizing the corresponding geometric parameters, the gradient descent ratio is lr each time. g .

[0047] Meanwhile, since there will inevitably be contact between the clothes and the human body, collision detection and handling are necessary. Since the optimization is based on gradient descent to minimize the loss function, the present invention proposes a simple and fast collision detection and handling that can be integrated into the gradient descent process.

[0048] The present invention assumes that there is a collision thickness r on the surface of the human body model. During each optimization, first, judge the projection t of the distance between the point on the clothing to be optimized and the nearest point on the human body model along the normal direction. When t < r, it is considered that a collision occurs, and an additional gradient d along this normal direction is required. Therefore, during the optimization process, in addition to the gradient propagated back by the loss function for each point coordinate to be optimized, there will also be an additional gradient. The expression is as follows:

[0049]

[0050] λ(X) = if(NN_normal(X)·(X - NN(X))) < r

[0051] where NN_normal(·) represents finding the normal vector corresponding to the nearest point of this point, and X represents a point to be optimized in the clothing or pants model.

[0052] Then, optimize according to the gradient after collision detection and handling, and the final results of the clothing and pants models can be obtained. At the same time, through the corresponding two-dimensional grid, the optimized geometric dimensions of the clothing can be obtained.

[0053] Optimizing the clothing model separately for each frame will consume a lot of time. Therefore, the solution of the present invention is to select one frame, generally the first frame, as the reference frame. The process of putting on clothes and pants and the process of optimizing the geometric parameters of clothes and pants are only completed on the reference frame. For other frames, only need to initialize the optimization result of the reference frame as a prior, and the optimization of the remaining frames can be quickly completed. Since the human pose in the video sequence changes gradually with time (frame number), the best solution is to optimize frame by frame in the order of the video sequence. In order to ensure the continuity between two consecutive frames, only need to add the following item to the total loss function:

[0054] L nn = ||m i (NN_index(m i-1 , V c )) - V c ||2 + ||m i (NN_index(m i-1 , V p )) - V p ||2 (3.21)

[0056] Where NN_index(X,Y) represents the index set of the nearest point to each point in the point set Y in X. This item can quickly make the clothing model change quickly and consistently with the posture change of the human body model.

[0057] Finally, the geometric representation of the reconstructed model is obtained by combining the parameterized inner human body part and the parameterized clothing part. The input video is then used to supervise the image obtained by directly rendering the model to estimate the texture of the model. When it is rendered again, a realistic reconstruction effect can be achieved, and the input viewpoint can be reproduced more accurately.

[0058] The above description does not limit the technical scope of the present invention in any way. Any modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. Linearly differentiable parametric clothing model making method, Features: It includes the following steps: S1. The pattern of the clothing is completely determined by 11 parameters, which in turn completely determine 16 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 16 nodes divide the entire pattern into 8 sub-patterns, that is, a total of 11 parameters, completely determining the geometric size and node position of the clothing; S2. The pattern of the pants is completely determined by 7 parameters, which in turn completely determine 10 nodes, satisfying symmetry and ensuring symmetry about the vertical axis. These 10 nodes divide the entire pattern into 4 sub-patterns, that is, a total of 7 parameters, completely determining the geometric size and node position of the pants; S3. Use the Delaunay triangulation algorithm to fill the encrypted outline with triangular faces to obtain the final clothing pattern; S4. The corresponding (u, v) coordinates and shape function values ​​(N1, N2, N3, N4) are calculated for each patch vertex.

2. The method for optimizing parameters of the linearly differentiable parametric clothing model making method according to claim 1, Features: It includes the following steps: S1. Based on the reasonable inner human body model, a loss function is designed to optimize the coordinates of all vertices of the three-dimensional geometry of the clothing through gradient descent; S2. While optimizing the 3D geometry of the garment, a loss function is designed to optimize the 2D geometry parameters of the garment by gradient descent; S3. During the optimization process, quickly detect whether the clothing and the inner human body collide, and use the normal distance to construct a loss function to optimize the collision part.

3. The method for optimizing parameters of the linearly differentiable parametric clothing model making method according to claim 2, Features: In S3 collision detection, the minimum normal distance between the point to be processed and the inner layer of the human body is calculated. When it is less than a given threshold, a set gradient is added to it with the direction of the corresponding normal direction, and it acts in the final gradient descent.

Citation Information

Patent Citations

  • Rapid posture-synchronizing method of three-dimensional virtual garment

    CN101630417A

  • Estimation method for dressed body 3D model in single character image

    CN102982578A

  • Garment deformation method based on human body Laplace deformation

    CN110176063A

  • Three-dimensional clothes model reconstruction method based on deep neural network

    CN111311751A

  • System and method for establishing correspondence, matching and repairing three dimensional surfaces of arbitrary genus and arbitrary topology in two dimensions using global parameterization

    US20100328311A1