Cloth penetration processing method based on self-attention mechanism

Abstract

The invention discloses a cloth penetration processing method based on a self-attention mechanism, and relates to the technical field of cloth penetration processing, and the method comprises the steps: firstly, taking a clothing vertex predicted by a clothing backbone network as input, and calculating an SDF value through a neural network; secondly, splicing the top points of the clothes and the characteristics related to the body and the clothes, executing sampling and aggregation operations, learning the relevance between the interiors of the blocks and the relevance between the blocks by utilizing a self-attention mechanism, and obtaining local geometric information and global geometric information of the clothes; then, an up-sampling process is performed, the block-level features are remapped back to the vertex level, a scaling factor for each vertex is calculated and then multiplied by the absolute value of the SDF to determine a shift offset, thereby performing a penetration process. Therefore, by the adoption of the cloth penetration processing method based on the self-attention mechanism, the real-time performance of the collision processing process and the high-efficiency Ethernet collision response precision are remarkably improved, the complexity of the problem is greatly simplified, and the method is suitable for any garment generation post-processing.

Description

Technical Field

[0001] The present invention relates to the technical field of fabric penetration processing, and particularly to a fabric penetration processing method based on a self-attention mechanism. Background Art

[0002] In the fields of computer graphics and virtual reality, fabric simulation and 3D clothing prediction technologies have important application values, such as virtual fitting, game development, animation production, etc.

[0003] Traditional clothing simulation technologies are based on physical simulation, and collision handling is a key component among them. Because even a small number of penetrating vertices will seriously reduce the realism. This physics-based simulation method relies on precise continuous collision detection to detect all penetrating vertices to achieve fine fabric animation production. However, this method is time-consuming and has a high computational cost, slowing down the simulation speed that already has a high computational complexity and making it difficult to achieve real-time interaction.

[0004] In recent years, with the development of machine learning and neural network technologies, learning-based methods have gradually become a research hotspot, aiming to improve simulation efficiency and prediction accuracy. For example, the approximate neural signed distance field (SDF) method uses the SDF value predicted by a neural network as the offset value, representing the signed distance from a point to the body surface. The penetration problem can be solved by moving the vertices along the gradient direction of the SDF, so approximate collision handling can be achieved. However, this method does not solve the penetration problem thoroughly enough. Especially, the error of the neural network prediction itself will cause the predicted SDF value to be biased. Although the vertices are moved out of the body surface, the edges connected between the vertices are still inside the body, thus still generating a visual effect of penetration. In addition, this method takes each vertex as the moving object, ignoring the correlation between vertices and reducing the efficiency of penetration processing.

[0005] Therefore, it is necessary to provide a clothing collision handling method based on the self-attention mechanism (Self-attention), and propose a fine clothing animation solution by combining a clothing generation backbone network (backbone) to solve the problems existing in the prior art. Summary of the Invention

[0006] The purpose of the present invention is to provide a fabric penetration processing method based on a self-attention mechanism, which can improve the time performance of the model's penetration processing, and at the same time make up for the moving error problem caused by using neural SDF, and propose a fine clothing animation solution.

[0007] To achieve the above purpose, the present invention provides a fabric penetration processing method based on a self-attention mechanism, including the following steps:

[0008] S1. Use the clothing vertices predicted by the clothing backbone network as input and calculate the SDF value using a neural network.

[0009] S2. Concatenate the clothing vertices with the features related to the body and clothing, perform sampling and aggregation operations, use the self-attention mechanism to learn the correlations within the blocks, obtain the local geometric information of the clothing, and use the attention mechanism to learn the correlations between the blocks to obtain the global geometric information of the clothing.

[0010] S3. After learning the local and global correlations, perform the upsampling process, remap the block-level features back to the vertex level, calculate the scaling factor for each vertex, then multiply it by the absolute value of the SDF to determine the movement offset, and use the obtained movement offset for penetration processing.

[0011] Preferably, step S2 includes: dividing the clothing into multiple blocks and capturing the correlations within and between the blocks.

[0012] Preferably, in step S2, performing the sampling and aggregation operations includes:

[0013] First, for a point set, obtain a subset of the point set using the farthest point sampling algorithm.

[0014] Second, for each vertex in the subset, find 3 neighbors for it through the K-nearest neighbor algorithm, take the sampled vertex and its neighbors as a block, and use the self-attention mechanism to learn the correlations between the vertex and its neighbor vertices to obtain the correlation features within the block.

[0015] Then, each block is regarded as a superpoint, and the blocks are further iteratively aggregated, dividing the clothing into multiple blocks, and using the self-attention mechanism to learn the correlations between the blocks.

[0016] Preferably, in step S3, the upsampling includes:

[0017] In each layer, given the position information of the points in the coarse-grained point set and the position information of the points in the fine-grained point set before downsampling, find K points in the coarse-grained point set around each point in the fine-grained point set through K-nearest neighbor search, and then use the reciprocal of the Euclidean distance as the weight to obtain the features of the points in the fine-grained point set based on the features of the points in the coarse-grained point set.

[0018] Preferably, in step S3, the scaling factor is predicted using a neural network through the downsampling and upsampling processes.

[0019] Preferably, the training of the neural network includes obtaining the data of each frame of the clothing animation using a physics-based simulation method and setting the total loss function by selecting the reconstruction term and the collision term.

[0020] Preferably, the reconstruction term supervision network restores the clothing to its original state at a specific animation frame as follows:

[0021]

[0022] where L r represents the reconstruction term loss, represents the vertex position predicted by the network, represents the position obtained in the physical simulation, and N is the number of vertices.

[0023] Preferably, the collision term restricts the vertices from penetrating the body surface, and the expression is as follows:

[0024] The collision term restricts the vertices from penetrating the body surface, and the expression is as follows:

[0025]

[0026] where L c represents the collision term loss, is the SDF value at the position .

[0027] Preferably, the expression of the total loss function is as follows:

[0028] L = λ r L r + λ c L c ;

[0029] In the formula, L is the total loss function, and λ r and λ c are balance factors.

[0030] Therefore, the present invention adopts the above-mentioned cloth penetration processing method based on the self-attention mechanism, and has the following technical effects:

[0031] (1) Real-time performance and high efficiency: Based on neural SDF, using a neural network for penetration processing significantly reduces the computational overhead of clothing animation generation and greatly improves the real-time effect of the collision processing process compared with the physical simulation method.

[0032] (2) Improvement in collision response accuracy: By using the self-attention mechanism to learn the intra-block and inter-block correlations, the local and global geometric information of the cloth is obtained, and an additional movement offset is predicted through this information, thereby compensating for the movement error problem generated by using neural SDF. Compared with the original SDF method for solving the penetration problem, the processing accuracy is significantly improved.

[0033] (3) Generalization and scalability: The post-collision processing network module can be combined with any clothing prediction backbone network to achieve efficient post-penetration processing, and the introduced additional computational overhead is relatively small compared to the backbone network, greatly enhancing the generalization and versatility of this method.

[0034] (4) Simplifying problem complexity: Physical-based collision handling in physical simulation often has high computational complexity and large overhead, while this method directly obtains the penetration correction distance through a simple network module, thus greatly simplifying the problem complexity of penetration processing.

[0035] The technical solution of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0036] Figure 1 is a flowchart of a cloth penetration processing method based on the self-attention mechanism;

[0037] Figure 2 is a schematic diagram of the downsampling and upsampling processes in an embodiment of a cloth penetration processing method based on the self-attention mechanism, where (a) is downsampling and (b) is upsampling;

[0038] Figure 3 is a schematic diagram of the SDF-based collision handling process in an embodiment of a cloth penetration processing method based on the self-attention mechanism, where (a) is the prediction result of the neural SDF and (b) is the prediction result combining the self-attention mechanism and the neural SDF. Detailed Embodiment

[0039] The present invention can be more detailedly explained through the following embodiments. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following embodiments.

[0040] Embodiment 1

[0041] As Figure 1 shown, the present invention provides a cloth penetration processing method based on the self-attention mechanism. The core idea is a collision handling module based on the self-attention mechanism and neural SDF, connected to the clothing prediction network. By dividing the clothing into multiple blocks and using the self-attention mechanism to capture the intra-block and inter-block correlations to obtain the local and global geometric information of the clothing, efficient collision handling is achieved. The specific steps are as follows:

[0042] S1. Use the clothing vertices predicted by the clothing backbone as input, and calculate the SDF value using a neural network. This value reflects the penetration state of each clothing vertex.

[0043] S2. Stitch the clothing vertices with the features related to the body and the clothing, perform sampling and aggregation operations, and use the self-attention mechanism to learn the correlations within the blocks, thereby obtaining the local geometric information of the clothing. Specifically, sample the clothing vertices and aggregate the sampled vertices with their surrounding vertices into blocks. To obtain the global geometric information of the clothing, use the attention mechanism to learn the correlations between the blocks after downsampling.

[0044] Among them, use the local and global geometric information to effectively handle collisions. The key to achieving this is to divide the clothing into multiple blocks and capture the correlations within and between the blocks. The specific steps include: input the coordinates x, signed distance field (SDF) value s, and the features g related to the body and the clothing of each vertex p on the clothing, and concatenate s and g to obtain the new feature f of the vertex. Then, the correlations can be learned based on the features of the vertices.

[0045] Considering the large number of vertices, in this embodiment, sampling and aggregation operations are adopted to reduce the number of vertex elements to reduce the storage burden and learn local information at the same time. As Figure 2 (a) shows, given a point set, use the farthest point sampling algorithm to obtain a well-distributed subset of this point set. The vertices in these subsets are called the "seed" vertices in the blocks. For each "seed" vertex, use K-nearest neighbors to find 3 neighbors for it. These neighbors and the sampled vertices together form a new block. Then, use the self-attention mechanism to learn the correlations between the "seed" vertices and the neighbor vertices, thereby obtaining the correlation features within the block. This process of dividing multiple vertices into multiple blocks can be regarded as a coarsening process. At the same time, each block can be regarded as a "superpoint", and its position is equal to the position of the "seed" vertex. In this way, the blocks can be aggregated through further iteration to expand the block coverage area.

[0046] After several sampling and aggregation iterations, the clothing is divided into multiple blocks. Take each block as the input and use the self-attention mechanism to learn the correlations between the blocks, thereby realizing the learning of the local geometric information and global geometric information of the clothing.

[0047] Since the ultimate goal is to predict the actual movement distance of each vertex, the learned correlation features must be mapped back to the vertex level. To complete this task, for each sampling and aggregation operation, an upsampling layer is paired with it, as Figure 3 (b) shows. In each layer, given the position information of the points in the coarse-grained point set and the position information of the points in the fine-grained point set before downsampling, find K points in the coarse-grained point set around each point in the fine-grained point set through K-nearest neighbors. Then, use the reciprocal of the Euclidean distance as the weight and obtain the features of the points in the fine-grained point set based on the features of the points in the coarse-grained point set, thereby realizing the upsampling process.

[0048] S3. After learning the local and global correlations, perform the upsampling process to remap the block-level features back to the vertex level, calculate the scaling factor for each vertex, then multiply it by the absolute value of the SDF to determine the movement offset, and use the obtained movement offset to move the penetrated vertices along the gradient direction of the SDF by the absolute value of the SDF as the movement distance to move the penetrated vertices to the body surface, thus achieving penetration processing.

[0049] However, moving the clothing vertices to the body surface does not guarantee that there are no body vertices penetrating the clothing, as shown in Figure 3 (a). For the method of using a neural network to approximate the SDF representation, due to the inaccuracy of the neural SDF, the situation may become worse. Based on this, this embodiment adopts the method of predicting an additional scaling factor α and multiplying it by the SDF value to obtain the final movement offset d, so as to achieve the collision handling process as shown in Figure 3 (b). Among them, the scaling factor α is predicted by the neural network through the downsampling and upsampling processes.

[0050] Among them, the neural network adopts a collision handling module based on the self-attention mechanism and neural SDF, which is connected to the clothing prediction backbone network to handle the penetrated vertices. This network can be fine-tuned in combination with the clothing generation backbone network or can be trained from scratch in an end-to-end manner.

[0051] To train the network, it is necessary to obtain the data of each frame of the clothing animation through a physics-based simulation method. For the setting of the loss function, the reconstruction term and the collision term are selected here.

[0052] The reconstruction term supervises the network to restore the clothing to the original state of a specific animation frame, and the formula is as follows:

[0053]

[0054] Among them, represents the vertex position predicted by the network, represents the position obtained in the physical simulation, and N is the number of vertices.

[0055] The collision term restricts the vertices from penetrating the body surface and is given by the following formula:

[0056]

[0057] Among them, is the SDF value at the position . To improve the training efficiency, a neural network approximating the SDF can be used to calculate the collision loss.

[0058] The final total loss function is the sum of the above two terms:

[0059] L = λ r L r + λ c L c ;

[0060] Wherein, λ r and λ c are balance factors.

[0061] Embodiment 2

[0062] In the field of digital humans and virtual try-on, efficient and real-time post-penetration processing is a crucial task. In some clothing animation generation schemes that combine motion generation models, fine clothing animation production can be achieved by introducing a clothing collision processing module, thereby improving the generalization of the clothing animation generation scheme for virtual fitting scenarios. Moreover, since the post-processing module uses a neural network, it can be coupled with a neural clothing simulator, thus improving the generation effect of neural cloth simulation and being applicable to post-processing of arbitrary clothing generation.

[0063] Therefore, the present invention adopts the above-mentioned cloth penetration processing method based on the self-attention mechanism, which can model the local and global geometric shape correlations of clothing and utilize these clothing correlation characteristics to combine with the signed distance field, thereby achieving efficient penetration processing.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cloth penetration processing method based on self-attention mechanism, characterized in that: The following steps are involved: S1, taking the clothing vertices predicted by the clothing backbone network as input, and using the neural network to calculate the SDF value; S2, splice clothing vertices with body and clothing-related features, perform sampling and aggregation operations, use the self-attention mechanism to learn the correlation within the block, obtain the local geometric information of the clothing, and use the attention mechanism to learn the correlation between blocks to obtain the global geometric information of the clothing; S3. After learning the local and global correlations, an upsampling process is performed to remap the block-level features back to the vertex level, calculate the scaling factor of each vertex, and then multiply it by the absolute value of SDF to determine the movement offset, and use the obtained movement offset for penetration processing.

2. According to claim 1, a cloth penetration processing method based on self-attention mechanism is characterized in that: Step S2 includes dividing the garment into multiple blocks and capturing the correlation within and between blocks.

3. According to the method for processing cloth penetration based on self-attention mechanism in claim 1, it is characterized in that: In step S2, performing sampling and aggregation operations includes: First, for a point set, a subset of the point set is obtained using the farthest point sampling algorithm; Secondly, for each vertex in the subset, three neighbors are found for it through the K nearest neighbor algorithm. The sampled vertices and neighbors are regarded as a block, and the self-attention mechanism is used to learn the correlation between the vertex and the neighbor vertices to obtain the correlation characteristics within the block; Then, each block is used as a super point to further iterate and aggregate the blocks, divide the clothing into multiple blocks, and use the self-attention mechanism to learn the correlation between blocks.

4. According to the method for processing cloth penetration based on self-attention mechanism in claim 1, it is characterized in that: In step S3, upsampling includes: In each layer, given the position information of the points in the coarse-grained point set and the position information of the points in the fine-grained point set before downsampling, the K nearest neighbors are used to find the K points in the coarse-grained point set around each point in the fine-grained point set, and then the inverse of the Euclidean distance is used as the weight to obtain the features of the points in the coarse-grained point set based on the features of the points in the coarse-grained point set.

5. The method for processing cloth penetration based on self-attention mechanism according to claim 1, characterized in that: In step S3, the scaling factor is obtained by downsampling and upsampling process using neural network prediction.

6. The method for processing cloth penetration based on self-attention mechanism according to claim 1, characterized in that: The training of the neural network includes obtaining the data of each frame of clothing animation through a physics-based simulation method, and selecting the reconstruction term and the collision term to set the total loss function.

7. A cloth penetration processing method based on self-attention mechanism according to claim 6, characterized in that: The reconstruction supervised network restores the garment to its original state at a specific animation frame as follows: Among them, l r represents the reconstruction loss, represents the vertex position predicted by the network, represents the position obtained in the physical simulation, and N is the number of vertices.

8. The method for processing cloth penetration based on self-attention mechanism according to claim 6, characterized in that: The collision term limits the penetration of vertices through the body surface and is expressed as follows: Among them, L c represents the collision loss, It's location The SDF value at .

9. A cloth penetration processing method based on self-attention mechanism according to claim 7 or claim 8, characterized in that: The expression of the total loss function is as follows: L=λ r L r +λ c L c ; Where L is the total loss function, λ r and λ c is the balancing factor.

Images