Real-time three-dimensional virtual dressing method based on graph network, storage medium and equipment

Abstract

The invention provides a real-time three-dimensional virtual dressing method based on a graph network. The real-time three-dimensional virtual dressing method comprises the steps that a video containing a human body, low-resolution clothes and a relation matrix are acquired; the relation matrix is a matrix between low-resolution clothes and high-resolution clothes; inputting the low-resolution clothes into a coarse network, and in the coarse network, extracting clothes features and outputting rough deformation clothes; and performing up-sampling on the rough deformation clothes by using the relation matrix, taking the sampled rough deformation clothes as input of a fine network, taking the clothes features as feature priori of the fine network, and outputting final deformation clothes through the fine network. According to the method, the generation efficiency of the clothes driver is improved, a real-time effect can be achieved, multi-card parallel computing can be supported, and the invention further provides a computer readable storage medium and electronic equipment which have the above beneficial effects.

Description

Technical Field

[0001] The present application relates to the field of video technology processing, and in particular to a real-time three-dimensional virtual dressing method, storage medium and device based on a graph network. Background Art

[0002] Existing technologies for 3D clothing driving tasks are primarily divided into two categories: those based on physical modeling and simulation, and those based on deep learning. Physical simulation methods are often used for offline dataset creation, resulting in low efficiency and poor real-time and practical performance. Algorithms based on parametric models offer poor driving effects and lack of realism, as well as poor model generalization. Typically, a separate model must be trained for each piece of clothing, resulting in limited practicality. Therefore, improving the practicality of clothing driving is a pressing technical issue facing those skilled in the art. Summary of the Invention

[0003] The purpose of this application is to provide a real-time three-dimensional virtual dressing method, system, storage medium and device based on a graph network, which can improve the generation efficiency and generation effect of clothing drive.

[0004] To solve the above technical problems, this application provides a real-time three-dimensional virtual dressing method based on a graph network. The specific technical solution is as follows:

[0005] Obtaining a video containing a human body, low-resolution clothing, and a relationship matrix; the relationship matrix is ​​a matrix between the low-resolution clothing and the high-resolution clothing;

[0006] Inputting the low-resolution clothing into a coarse network, extracting clothing features in the coarse network and outputting a coarse deformed clothing;

[0007] The rough deformable clothing is upsampled by the relationship matrix and used as the input of a fine network. The clothing features are used as feature priors of the fine network, and the final deformable clothing is output through the fine network.

[0008] Optionally, after upsampling the rough deformable clothing and before using it as input to the fine network, the method further includes:

[0009] Calculate the relationship matrix between low-resolution clothes and high-resolution clothes;

[0010] Calculating the relationship matrix between low-resolution clothes and high-resolution clothes includes:

[0011] Calculating the nearest distance face index value and center of gravity weight from the vertex of the high-resolution clothing to the low-resolution clothing;

[0012] Calculate the feature value of the closest surface from low resolution to high resolution according to the closest distance surface index value;

[0013] The coordinates of the high-resolution clothing vertices are obtained by weighting the eigenvalues ​​with the center of gravity weights to determine the relationship matrix.

[0014] Optionally, inputting the low-resolution clothing into the coarse network includes:

[0015] Initializing the low-resolution clothing and the human body to obtain clothing parameterization weights; the clothing parameterization weights are obtained based on the human body parameterization weights and an association matrix, and the association matrix is ​​related to the distance between the template human body and the clothing;

[0016] performing network initialization on the coarse network based on the clothing parameterized weights;

[0017] Utilizing the graph network corresponding to the coarse network, the node features and edge features of the low-resolution clothing are extracted and updated; the node features include vertex velocity vectors, normal vectors, material parameters, node types, and node levels; the edge features include edges of the current deformed clothing, edges of the template clothing, and new edges constructed between the current deformed clothing and the human body;

[0018] Calculating a deformation vertex according to the node feature, the edge feature and the acceleration of the regression vertex;

[0019] A rough deformable garment is determined based on the deformable vertices.

[0020] Optionally, the process of constructing the new edge of the current deformed clothing and the human body includes:

[0021] Determine the first vertex on the clothing and the second vertex on the human body;

[0022] If the first vertex and the second vertex belong to the same category, determine a new edge constructed by the human body and the clothing; the category is a category divided based on the human body structure; wherein, if the second vertex is located in a category boundary area, the nearest first vertex sought by the second vertex belongs to a different category corresponding to the category boundary area.

[0023] Optionally, the loss functions of the coarse network and the fine network include collision loss between clothing and the human body, self-collision loss, gravity loss, clothing material-related loss, friction loss between clothing and the human body, external force loss, and inertia loss;

[0024] The collision loss is calculated based on the signed distance from the vertex of the clothing to the human body, a threshold constant, and the number of vertices of the clothing;

[0025] The self-collision loss is used to constrain two points that have no connection and whose distance is less than a set distance;

[0026] The friction loss between the clothing and the human body is calculated by establishing a smooth mapping between static friction and dynamic friction through the relative displacement of the collision pair.

[0027] Optionally, also include:

[0028] For the inner layer of clothing, calculate the nearest distance from the vertex of the inner layer of clothing to the outer layer of clothing and the normal vector of the projection point;

[0029] If the product of the nearest neighbor distance and the normal vector is less than a set threshold constant, it is determined that there is a clothing collision, and the inner clothing vertex is offset;

[0030] Rendering the inner layer of clothing, and the classification map and depth map of the inner layer of clothing and the outer layer of clothing; the classification map is used to distinguish pixel ownership, and the depth map is used to perform depth culling during rendering;

[0031] When rendering inner clothes, if the current pixel is determined to belong to outer clothes according to the classification map, the current pixel is discarded;

[0032] When rendering outer clothing, if it is determined according to the classification map that the current pixel belongs to inner clothing, pixel culling is performed according to the depth map.

[0033] Optionally, obtaining a video containing a human body includes:

[0034] Obtaining a human body parameter sequence output by a three-dimensional grid; the human body parameter sequence includes posture parameters and shape parameters;

[0035] Accordingly, performing a real-time three-dimensional virtual dressing operation on the human body in the video based on the driving sequence includes:

[0036] A three-dimensional virtual dressing operation is performed based on the driving sequence and the human body parameter sequence.

[0037] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-described method when executed by a processor.

[0038] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0039] The present application provides a real-time three-dimensional virtual dressing method based on a graph network, comprising: obtaining a video containing a human body, low-resolution clothes, and a relationship matrix; the relationship matrix is ​​a matrix between low-resolution clothes and high-resolution clothes; the low-resolution clothes are input into a coarse network, in which the clothes features are extracted and a coarse deformed clothes is output; the coarse deformed clothes are up-sampled using the relationship matrix and used as the input of a fine network, the clothes features are used as the feature priors of the fine network, and the final deformed clothes are output through the fine network.

[0040] This application uses both coarse and fine networks to process low-resolution clothing, significantly improving the speed and efficiency of clothing driver generation, enabling real-time results. Compared to existing serial architectures, this application supports multi-GPU parallel computing. It also mitigates the interactive collision issues present in existing graph network solutions, such as the chain-like collapse effect caused by simple contact between the human body and clothing.

[0041] The present application also provides a computer-readable storage medium and an electronic device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0043] Figure 1 A flowchart of a real-time three-dimensional virtual dressing method based on a graph network provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a cross collision provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the classification of human bodies and clothes provided in the embodiments of the present application;

[0046] Figure 4 Schematic diagram of input with different resolutions provided in the embodiment of the present application;

[0047] Figure 5 A schematic diagram of the network architecture provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a driving algorithm test architecture provided in an embodiment of the present application;

[0049] Figure 7A schematic diagram of the upper and lower garment processing process based on rendering provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram comparing the interactive collision effects provided in the embodiments of the present application;

[0051] Figure 9 A schematic diagram comparing the overlapping collision effects provided in the embodiments of the present application;

[0052] Figure 10 A schematic diagram comparing the collision effects between the human body and clothing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] Please refer to Figure 1 , Figure 1 A flowchart of a real-time three-dimensional virtual dressing method based on a graph network provided in an embodiment of the present application, the method comprising:

[0055] S101: Acquire a video containing a human body, low-resolution clothes, and a relationship matrix; the relationship matrix is ​​a matrix between the low-resolution clothes and the high-resolution clothes;

[0056] S102: Inputting the low-resolution clothing into a coarse network, extracting clothing features in the coarse network and outputting a coarse deformed clothing;

[0057] S103: The rough deformable clothing is upsampled using the relationship matrix and used as the input of a fine network, the clothing features are used as feature priors of the fine network, and the final deformable clothing is output through the fine network.

[0058] First, the acquisition of low-resolution clothes (referred to as low-score clothes) is performed, and the relationship matrix between low-score and high-score clothes can also be pre-calculated.

[0059] In a feasible implementation, when calculating the relationship matrix, the nearest surface index value and the center of gravity weight from the vertex of the high-resolution clothing to the low-resolution clothing can be calculated, and the eigenvalue of the nearest surface from the low resolution to the high-resolution can be calculated based on the nearest surface index value. Finally, the coordinates of the vertices of the high-resolution clothing can be obtained by weighting the eigenvalues ​​with the center of gravity weight to determine the relationship matrix.

[0060] It should be emphasized that part of the preprocessing stage can be calculated and stored in advance, and no additional time is required in the training and testing stages.

[0061] When obtaining low-resolution clothing, the template clothing can be reduced to different polygon counts to obtain low-resolution clothing of varying resolutions. For example, the template clothing can be reduced to 1 / 2, 1 / 4, 1 / 8 of its original polygon count, etc. This operation can be performed using a rendering engine such as Houdini. During training, clothing of one random resolution is input to the low-resolution network. During testing, different low-resolution clothing can be selected as input to the coarse network based on actual efficiency requirements.

[0062] The relationship matrix between low- and high-scoring clothes is pre-calculated for upsampling. This has two main effects: 1. The features extracted by the coarse network are used as feature priors for the fine network through upsampling. 2. The deformed clothes output by the coarse network are used as initialization input for the fine network through upsampling. This application provides a simple and efficient upsampling algorithm as follows:

[0063] First, calculate the nearest face index value from the vertex of the high-scoring clothes to the low-scoring clothes , and the center of gravity weight .

[0064] Then, for the coordinates or eigenvalues ​​of the low-scoring clothing vertices, input , calculate the coordinates or eigenvalues ​​of the nearest surface of the high-scoring clothing .

[0065] Finally, the above obtained , through the center of gravity weight Weighted to get the coordinates or eigenvalues ​​of high-scoring clothing vertices , the formula is as follows:

[0066] , (1);

[0067] At this point, the low-resolution clothing and the human body are initialized to obtain clothing parameterized weights, and the network of the coarse network is initialized based on the clothing parameterized weights. The clothing parameterized weights are obtained based on the human body parameterized weights and an association matrix, and the association matrix is ​​related to the distance between the template human body and the clothing.

[0068] Initialization is used to obtain the initialization clothing and human body drive sequence in the posture space. No network learning is required in the initialization stage, and the purpose is to obtain the clothing parameterized weights , which can be expressed by the following formula:

[0069] , (2);

[0070] The human body parameterization weight The priors derived from known parameterized human bodies usually include skinning factors and distortion factors (such as pose-blend-shape, etc.). The correlation matrix R is usually related to the distance between the template human body and the clothing, such as a nearest neighbor-based weighting algorithm, etc. This application does not limit the specific scheme.

[0071] The above process is only used to initialize the coarse network. For the fine network, the initialization of its past frame sequence is obtained by upsampling the output of the coarse network.

[0072] In the coarse network, when extracting clothing features and outputting coarse deformed clothing, it is necessary to extract and update the features of vertex v and edge e based on the graph network.

[0073] First, obtain the node features and edge features , the formula is as follows:

[0074] , (3);

[0075] , (4);

[0076] The node features , mainly including: vertex velocity vector , normal vector , material parameters (such as Lame parameters , and the bending parameters ), the type of node (such as whether the point belongs to clothing or body, etc.) and the level of the node (such as points from the original model or model points that are downsampled by half, etc.).

[0077] Edge Features It mainly includes three parts: the edge length of the current deformed clothing (in posture space) , the side length of the template clothes (in standard space) , and the new edge lengths of clothing and human body in the pose space The specific construction process of the new edge is described below. It is mainly used to constrain the human body and clothes in motion.

[0078] Then, the vertices and edges are updated through the network, and the formula is expressed as follows:

[0079] , (5);

[0080] , (6);

[0081] Where 𝑓 represents a feature extraction network such as a multi-layer perceptron, and vertex updates depend on the features of the vertex in the current frame. , the characteristics of the edge connected to the vertex , and edge features that are not directly connected but are close enough to have self-collision potential The update of the edge depends on the current edge features , and the vertex features at both ends .

[0082] Features extracted from graph networks , the acceleration of the regression vertex The final deformed vertex is obtained, and the edge features are also involved in the calculation of the deformed vertex because they contain vertex features. The process is as follows:

[0083] , (7);

[0084] , (8);

[0085] , (9);

[0086] , (10);

[0087] in Represent low-score and high-score feature regression networks such as multi-layer perceptrons, etc. Represents the upsampling process, high score regression network It depends on two parts: low-score clothing vertex features and high-score clothing vertex features. Indicates initialization of vertex coordinate input, Represents the speed of the initialization vertex, v is the coordinate value of the final deformation vertex, for the fine network, the output of the coarse network As initialization input.

[0088] In addition, the SOTA algorithm in the existing technology has the phenomenon of interactive collision. The core problem lies in constructing the edge features of clothes and human body. The existing algorithm, for the vertices of clothes , find the closest point on the human body , and then connect the two points to form a new edge. This construction process has serious problems in many human interaction scenarios.

[0089] See also Figure 2 , Figure 2 This is a schematic diagram of the cross collision provided in the embodiment of the present application, for the clothes vertex on the "right calf" , in the current posture, the closest point found by the closest point is the body point on the "left foot" , the new edge constructed by this will lead to the next moment of the clothes vertex Will follow the left foot body point The movement will cause unreasonable phenomena such as clothing deformation.

[0090] To this end, this application proposes a category-based constraint that divides the human body into multiple categories, thereby constructing new edges between the current deformable clothing and the human body. The construction process includes the following steps:

[0091] Step 1: Determine the first vertex on the clothes and the second vertex on the body;

[0092] Step 2: If the first vertex and the second vertex belong to the same category, determine the new edge constructed by the human body and the clothes; the category is a category divided based on the human body structure; wherein, if the second vertex is located in the category boundary area, the nearest first vertex sought by the second vertex belongs to a different category corresponding to the category boundary area.

[0093] See also Figure 3 , Figure 3 A schematic diagram of the classification of human bodies and clothes provided in an embodiment of the present application. Figure 3 In [1], the human body is divided into five categories (left leg, right leg, left arm, right arm, and main body), with different colors corresponding to different categories. Clothes are also categorized accordingly. When constructing new edges between people and clothing, only vertices of the same category can form an edge. This effectively mitigates the problem of human interaction and collision during movement. This can be expressed using a simple formula as follows:

[0094] , (11);

[0095] in Represents the new edges constructed by the human body and clothing, and Represent points on clothes and human body respectively, and these points are all of the same category .

[0096] It should be noted that the above classification cannot handle the demarcation of boundary areas (such as the junction of the left arm and the main body) well. To make the classification more reasonable, the boundary area is additionally considered. For body points in the left arm boundary area, when constructing new edges, the label of the nearest clothing point can be "main body" in addition to "left arm".

[0097] At the same time, in order to consider the self-collision problem of the same category area, this application introduces additional new edges , where the two points are the closest points on the clothing vertex that belong to the same category area but are not connected. The introduction of additional new edges is mainly for the subsequent construction of self-collision approximation.

[0098] The classification of human bodies and clothing categories can be pre-processed in advance, so the strategy basically does not take up additional time and can effectively alleviate the problem of interactive collisions.

[0099] This application adopts a self-supervised learning model, which is convenient and fast to train without expensive big data production. In order to produce realistic and natural effects, the physical constraints cited in this application are as follows:

[0100] , (12);

[0101] in Represents the weight coefficient, loss function Including collision damage (between clothing and the human body) , self-collision loss , gravity loss , clothing material related losses such as bending loss , stretch loss , friction loss between clothes and human body , external force loss (such as random wind, etc.) and inertia loss .

[0102] The gravity constraint simulates the physical gravity field, and the material constraint simulates different materials. This application provides a commonly used StVK (Saint-Venant-Kirchhoff) elastic constraint.

[0103] The collision loss mainly constrains the penetration of human body and clothing. The formula is as follows:

[0104] , (13);

[0105] in Indicates the sdf value from the vertex of the clothing to the human body, c represents the threshold constant, such as 2mm, etc. Indicates the number of vertices of the clothing. For example, you can use a cubic exponent to strengthen the collision constraint. The sdf value indicates the shortest distance from the clothing vertex to the human body surface, with a symbol to distinguish the vertex position:

[0106] Positive (+): The apex is outside the body (not penetrated).

[0107] Negative (-): The vertex penetrates the body (through the mold).

[0108] For example, if the human body model is defined as a closed surface, the sdf value is calculated by calculating the perpendicular distance from the vertex to the surface and assigning a sign to reflect its relative position.

[0109] The self-collision constraint is used to constrain the self-collision problem of the clothing itself. The above description has considered the constraint problems of different types of regions. For the self-collision problem of the same region, this application provides a simple and effective constraint based on repulsion. It is mainly used to constrain two points that are not connected and are relatively close to each other. The formula is as follows:

[0110] , (14);

[0111] in It is required to be within a certain distance range. For two points that are far away, this constraint is meaningless.

[0112] The friction constraint is used to simulate the friction between the clothing and the body. For any pair of collision pairs, this application constructs a continuous static and dynamic friction formula based on their relative displacement. The friction formula is defined as follows:

[0113] , (15);

[0114] in is the local friction factor, is the local contact normal force, represents the local relative displacement of the collision pair, Represents the sliding projection matrix, which is used to project the relative displacement in space onto the plane, that is, . is a A continuous function that creates a smooth mapping between static and kinetic friction using the relative displacement of the collision pair. This formula ensures that static friction appears when the relative displacement is small, and transitions to kinetic friction when the relative displacement is large, thus achieving continuous and smooth friction.

[0115] External force constraints are mainly used to simulate external forces such as wind. To some extent, gravity can also be considered as an external force. The formula can be simplified as follows:

[0116] , (15);

[0117] in represents the external force vector, Represents the direction vector of the force.

[0118] In order to converge faster, this application adopts a step-by-step training strategy: first train the coarse network, then fix the coarse network parameters and train the fine network part.

[0119] In addition, due to the serial architecture of existing SOTA algorithms such as HOOD, it is difficult to achieve real-time effects for more complex clothing. Therefore, this application introduces corresponding strategies in both training and testing.

[0120] Adopting a network structure from coarse network to fine network, see Figure 4 , Figure 4 This is a schematic diagram of inputs with different resolutions provided in the embodiment of this application. From left to right, the number of input vertices is: 12K, 5.5K, 3.2K, and 1.8K. It can be seen that HOOD can also obtain good results for low-resolution inputs.

[0121] This result shows that the HOOD network structure has a lot of redundancy, and the original complex network can be decomposed into two smaller parts. The coarse network mainly processes low-score inputs, so there is no need for a complex network. This application only uses 1 / 2 of the depth of the original network. The fine network relies on the feature priors and initialization values ​​provided by the coarse network, and only needs to learn additional detail offsets. Therefore, there is no need for a complex network structure. This application only uses 1 / 3 of the depth of the original network. Overall, the relatively complex coarse network has low-score clothes as input; while for high-score inputs, the parameters of the fine network are greatly reduced. At the same time, the network structure from the coarse network to the fine network can better enable the network to learn high-frequency detail information.

[0122] This embodiment uses a coarse network and a fine network to process low-resolution clothing, significantly improving the running speed and the efficiency of clothing driver generation, achieving real-time effects. Compared to the existing serial architecture, this application can support multi-GPU parallel computing. It also alleviates the interactive collision issues existing in existing graph network solutions, such as the chain collapse effect caused by simple contact between the human body and clothing.

[0123] This application also proposes a serial and parallel testing architecture that can be adopted in the actual application of the real-time three-dimensional virtual dressing method based on the graph network.

[0124] See also Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of the network architecture provided by an embodiment of the present application. It can be seen that the present application is a network architecture system that combines a coarse network and a fine network. The coarse network is mainly based on low-resolution clothing, quickly obtaining a relatively rough deformed clothing. The fine network is based on the feature priors provided by the coarse network, and is initialized based on the results of the previous frame of the coarse network, learning more details and other information, and outputting the final deformed clothing.

[0125] Figure 6 Schematic diagram of the driving algorithm test architecture provided in the embodiment of the present application, that is, a 3D dressing human driving algorithm based on a graph network. Figure 5 The specific application of the network architecture shown can support multi-card parallel computing. The main reason why HOOD does not support parallel computing is that it relies on the results of the previous frame as input. In order to obtain a good initialization result and enable the network to be parallelized, the test architecture provided by this application is as follows:

[0126] The input sequence is divided into n segments, where the initial and final frames of each segment are defined as keyframe parameters. First, low-resolution clothing is input, the driving parameters of the keyframes are selected, and a coarse network architecture is used to obtain the initialization results of the keyframes. Then, high-resolution deformable clothing is obtained through upsamping operations.

[0127] For n input segments, the high-resolution deformable clothing of the keyframes above is used as the initial input for each segment. Then, based on the coarse-to-fine network architecture presented in this article, the results for each segment are obtained. This stage can be parallelized, for example, using multiple processors with multiple cards.

[0128] For key frames (the last few frames of the previous segment and the first few frames of the current segment), simple Laplacian geometry processing is applied to make the results between segments smoother. Finally, the results of each segment are combined into the final output.

[0129] Existing SOTA algorithms such as HOOD only consider the collision between clothes and the human body and cannot handle the problem of overlapping. In order to solve this problem, most of the existing methods are processed at the geometric or network level. Commonly used methods such as SENC and ContourCraft introduce self-collision pairs as input in the network, so that self-collision constraints can be added during training, but this process greatly increases the computational time and cannot meet real-time requirements. Therefore, it is difficult to ensure timing smoothness and time-consuming to perform overlapping processing purely on the geometric end. Taking into account that the real-time drive is ultimately displayed as a rendered image on the screen, this application proposes an overlapping strategy that combines geometry and rendering, decomposing the overlapping collision into two parts, geometry and rendering, to efficiently alleviate the overlapping problem. The main steps are as follows:

[0130] For the inner layer of clothing, calculate the nearest distance from the vertex of the inner layer of clothing to the outer layer of clothing and the normal vector of the projection point;

[0131] If the product of the nearest neighbor distance and the normal vector is less than a set threshold constant, it is determined that there is a clothing collision, and the inner clothing vertex is offset;

[0132] Rendering the inner layer of clothing, and the classification map and depth map of the inner layer of clothing and the outer layer of clothing; the classification map is used to distinguish pixel ownership, and the depth map is used to perform depth culling during rendering;

[0133] When rendering inner clothes, if the current pixel is determined to belong to outer clothes according to the classification map, the current pixel is discarded;

[0134] When rendering outer clothing, if it is determined according to the classification map that the current pixel belongs to inner clothing, pixel culling is performed according to the depth map.

[0135] The main purpose of simple geometry post-processing is to deal with large collision areas, in order to relieve the pressure on the rendering end. The main steps are as follows:

[0136] Collision detection. First calculate the nearest neighbor distance from the inner layer of clothing to the outer layer of clothing , and the normal vector of the projection point ; Then, calculate If the value is less than , it means there is a collision.

[0137] Collision response. For larger collision area points, such as If the value is less than a certain threshold constant, the inner clothing vertices are offset.

[0138] This operation mainly affects the inner layer of clothing, because the inner layer of clothing will be blocked by the outer layer of clothing during the rendering process. By offsetting the geometry of the inner layer of clothing, the visual effect will be minimally affected.

[0139] When performing rendering-based collision handling, the main method to mitigate overlapping collisions is depth culling on the rendering side (by comparing the depth value of the current rendered pixel with the depth value of the known depth map to determine whether the current pixel should be included in the rendering result). See Figure 7 , Figure 7 This is a schematic diagram of the upper and lower garment processing process based on rendering provided in an embodiment of the present application. The main steps are as follows:

[0140] Render the inner layer of clothes, inner layer + outer layer of clothes to generate classification maps and depth maps, such as Figure 6 ① and ② in the figure. The classification map also renders the human body. The purpose of the classification map is to distinguish each pixel's belonging to the inner layer of clothing, outer layer of clothing, body, or background. The depth map is used for depth culling during the final rendering. When rendering the inner layer of clothing, based on the classification result in ②, if the location is outer clothing, it is culled. Otherwise, the depth map in ② is used to determine whether the current pixel should be culled.

[0141] When rendering the outer layer of clothing, based on the classification result in ②, if the pixel is the inner layer of clothing, it is necessary to compare the depth value in ① to cull it (because there may be several layers of clothing falling on the same pixel). Otherwise, the depth map in ② is used to determine whether the current pixel should be culled.

[0142] Considering efficiency, modifications are made to the existing real-time rendering pipeline, and depth culling is performed in the custom rendering pipeline instead of leaving it to the rendering engine itself. In order to optimize the rendering process, the first two rendering steps in this solution are completed using CustomPass in Unity's HDRP. In this way, the rendering of these two steps only contains the required passes and is inserted into a series of passes of Camera Render, without any additional rendering overhead. The final process of rendering the top and bottom clothes is the original rendering process of Unity, but the solution is to make a slight modification to the original rendering shader of Unity, turn off the original depth occlusion culling, and pass ① and ② rendered in the previous step into the shader for custom occlusion culling.

[0143] Based on the graph network-based real-time 3D virtual dressing method provided by this application, this application can be extended to input 3D mesh (a data structure used to represent the surface shape of a 3D object). The main steps are as follows:

[0144] Fast estimation of parametric human body parameters such as posture parameters based on 3D Mesh And row parameters This application provides an algorithm based on NLF-FIT as a reference, which can quickly estimate the parameters of SMPL or SMPL-X, but does not limit the specific feasible algorithm.

[0145] Linear interpolation is used to obtain the driving parameter sequence from the standard pose (such as T-pose) to the current pose. ,in Indicates the initial standard posture, Indicates the pose estimated based on Mesh.

[0146] The obtained driving parameter sequence and template clothing are input into the dressing human body driving algorithm provided in this application, and the last frame of the sequence is the final output.

[0147] The adaptive dressing system based on mesh input provided in this application is applicable to mesh input of any shape and posture based on the SMPL topology. Furthermore, the source of the mesh can also come from the input image, such as using Pymaf to estimate the parameters of SMPL from the image, so the system can also be extended to image input.

[0148] Furthermore, this application further expands the input to video and provides a real-time 3D clothing driving system based on video input. For the video input, it first uses existing algorithms such as NLF+ optimization to obtain a smoothed driving parameter sequence. Then, the algorithm of this application is used to obtain the final 3D clothing sequence. To achieve the real-time driving effect, the specific strategy is as follows:

[0149] The video is segmented into 30-frame sequences per segment. First, the driving parameter sequence is obtained using the NLF algorithm;

[0150] For the first segment, we interpolate the first frame of the linear interpolation output, then input the interpolated driving parameter sequence into the network provided by this application, and finally output the final 30 frames of the 3D clothed human sequence. At the same time, the last 3 frames of each segment are also saved.

[0151] For the parameterized sequence input of the second and subsequent segments, the 3-frame sequence result saved from the previous segment is first superimposed during initialization, then input into the network, and finally the final 30-frame sequence result is output. The main purpose of superimposing the 3-frame sequence result from the previous segment is to smooth the segments and prevent jitter. The principle is to ensure that the current segment can well inherit the state information of the previous segment, such as the coordinates, vertex velocities, and accelerations of the vertex (acceleration requires at least 3 frames).

[0152] Figure 8 This is a schematic diagram comparing the interactive collision effects provided in the embodiment of the present application. Figure 8 The upper middle image has obvious unreasonable phenomena such as clothing distortion. After processing by this application, the human body is classified and the clothing is classified. Only human body and clothing vertices of the same category can form edges, which effectively alleviates the problem of human body interaction and collision during movement. Figure 8 The following figures are the effect diagrams after processing by this application, which significantly alleviate the phenomenon of clothing distortion.

[0153] Figure 9 The schematic diagram of the comparison of the overlapping collision effects provided in the embodiment of the present application clearly shows that Figure 9 In the four sets of effect comparison diagrams shown, the left side is the existing rendering result, and the right side is the rendering effect diagram after processing by this application, which can significantly alleviate the problem of overlapping collisions. In actual application, modifications are made in the existing real-time rendering pipeline, and depth culling is performed in the custom rendering pipeline, and it is no longer handled by the rendering engine itself. In order to optimize the rendering process, the early rendering in this application can be completed using CustomPass in Unity's HDRP. In this way, the rendering of these two steps only contains the required Pass, and is inserted into a series of Passes of Camera Render, without any additional rendering overhead.

[0154] Figure 10 This is a schematic diagram comparing the collision effects between the human body and clothing provided in the embodiment of the present application. Figure 10 The upper middle picture is a rendering diagram of the GAPS algorithm, and the lower picture is a rendering diagram after processing by this application. It can be clearly seen that this application can obtain more realistic simulation results and at the same time alleviate the collision problem between the human body.

[0155] This application also provides a computer-readable storage medium having a computer program stored thereon. When executed, the computer program can implement the steps provided in the above embodiments. The storage medium may include: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0156] The present application also provides an electronic device that may include a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the steps provided in the above embodiment can be implemented. Of course, the electronic device may also include various network interfaces, a power supply, and other components.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems provided in the embodiments, since they correspond to the methods provided in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0158] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core ideas of this application. It should be noted that for those skilled in the art, without departing from the principles of this application, various improvements and modifications can be made to this application, and such improvements and modifications also fall within the scope of protection of this application.

[0159] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A real-time three-dimensional virtual dressing method based on graph network, characterized in that: include: Get a video containing a human body, low-resolution clothing, and a relationship matrix; The relationship matrix is ​​a matrix between low-resolution clothes and high-resolution clothes; Inputting the low-resolution clothing into a coarse network, extracting clothing features in the coarse network and outputting a coarse deformed clothing; The rough deformable clothing is upsampled by the relationship matrix and used as the input of a fine network. The clothing features are used as feature priors of the fine network, and the final deformable clothing is output through the fine network.

2. The method according to claim 1, characterized in that After upsampling the rough deformable clothes and before using them as input to the fine network, the following steps are also included: Calculate the relationship matrix between low-resolution clothes and high-resolution clothes; Calculating the relationship matrix between low-resolution clothes and high-resolution clothes includes: Calculating the nearest distance face index value and center of gravity weight from the vertex of the high-resolution clothing to the low-resolution clothing; Calculate the feature value of the closest surface from low resolution to high resolution according to the closest distance surface index value; The coordinates of the high-resolution clothing vertices are obtained by weighting the eigenvalues ​​with the center of gravity weights to determine the relationship matrix.

3. The method according to claim 1, characterized in that Inputting the low-resolution clothing into the coarse network includes: Initializing the low-resolution clothing and the human body to obtain clothing parameterization weights; the clothing parameterization weights are obtained based on the human body parameterization weights and an association matrix, and the association matrix is ​​related to the distance between the template human body and the clothing; performing network initialization on the coarse network based on the clothing parameterized weights; Utilizing the graph network corresponding to the coarse network, the node features and edge features of the low-resolution clothing are extracted and updated; the node features include vertex velocity vectors, normal vectors, material parameters, node types, and node levels; the edge features include edges of the current deformed clothing, edges of the template clothing, and new edges constructed between the current deformed clothing and the human body; Calculating a deformation vertex according to the node feature, the edge feature and the acceleration of the regression vertex; A rough deformable garment is determined based on the deformable vertices.

4. The method according to claim 3, characterized in that The process of constructing the new edges of the current deformed clothing and human body includes: Determine the first vertex on the clothing and the second vertex on the human body; If the first vertex and the second vertex belong to the same category, determine a new edge constructed by the human body and the clothing; the category is a category divided based on the human body structure; wherein, if the second vertex is located in a category boundary area, the nearest first vertex sought by the second vertex belongs to a different category corresponding to the category boundary area.

5. The method according to claim 4, characterized in that The loss functions of the coarse network and the fine network include collision loss between clothing and human body, self-collision loss, gravity loss, clothing material-related loss, friction loss between clothing and human body, external force loss and inertia loss; The collision loss is calculated based on the signed distance from the vertex of the clothing to the human body, a threshold constant, and the number of vertices of the clothing; The self-collision loss is used to constrain two points that have no connection and whose distance is less than a set distance; The friction loss between the clothing and the human body is calculated by establishing a smooth mapping between static friction and dynamic friction through the relative displacement of the collision pair.

6. The method according to claim 1, characterized in that Also includes: For the inner layer of clothing, calculate the nearest distance from the vertex of the inner layer of clothing to the outer layer of clothing and the normal vector of the projection point; If the product of the nearest neighbor distance and the normal vector is less than a set threshold constant, it is determined that there is a clothing collision, and the inner clothing vertex is offset; Rendering the inner layer of clothing, and the classification map and depth map of the inner layer of clothing and the outer layer of clothing; the classification map is used to distinguish pixel ownership, and the depth map is used to perform depth culling during rendering; When rendering inner clothes, if the current pixel is determined to belong to outer clothes according to the classification map, the current pixel is discarded; When rendering outer clothing, if it is determined according to the classification map that the current pixel belongs to inner clothing, pixel culling is performed according to the depth map.

7. The method according to claim 1, characterized in that Obtaining videos containing human bodies involves: Obtaining a human body parameter sequence output by a three-dimensional grid; the human body parameter sequence includes posture parameters and shape parameters; Accordingly, performing a real-time three-dimensional virtual dressing operation on the human body in the video based on the driving sequence includes: A three-dimensional virtual dressing operation is performed based on the driving sequence and the human body parameter sequence.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed.

Images