Attention mechanism-based model deformation generation method
By introducing attention mechanism and self-attention module into the deformation generation model, combining the implicit deformation field and smooth loss function, the problem of noise impact in the prior art is solved, and a higher quality three-dimensional model deformation result is achieved.
Patent Information
- Application Number
- CN202411116449.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-27
AI Technical Summary
The existing deformation technology ignores the noise problem when generating new models, resulting in flaws in the deformation results and it is difficult to achieve reasonable deformation without involving human-computer interaction.
The deformation generation model based on the attention mechanism is adopted, and the deformation vector generated during the deformation process is optimized through the shape information encoding module, deformation module and self-attention module, combined with the implicit deformation field and smooth loss function, and the deformation amplitude is restrained.
A more complete and reasonable deformation generation results are achieved, noise is eliminated, the stability and quality of the deformation process are ensured, and the overall performance of the generative model is improved.
Smart Images
Figure CN119206045A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a method for generating a deformation model based on an attention mechanism. Background Art
[0002] In recent years, with the rapid development of technologies such as virtual reality and augmented reality, content generation technology has received increasing attention, among which deformation plays an important role. Deformation in the field of content generation refers to the process of adjusting and changing the shape and structure of a given object. As the definition says, deformation in the field of content generation is different from deformation in real life. Deformation in real life generally refers to the deformation caused by an object under the influence of external forces, while deformation in content generation refers to the ability to change and adjust the shape and structure of a model through deep learning algorithms to generate a new model.
[0003] DeepSDF introduces a shape representation method based on signed distance function (SDF), which uses deep neural networks to learn continuous SDF to represent three-dimensional shapes, and can model different shapes with high precision, and then deform the shapes through interpolation. DIF also proposes a novel deformation implicit field representation method based on signed distance function, which is used to model 3D models, learn the latent space of a 3D model, and form dense correspondences between shapes, forming a deformation template field to generate deformation vectors for deformation.
[0004] Although the above methods can generate completely new models of the same type in some cases, they only focus on generating new model parts and ignore the noise generated during the deformation process, which makes the deformation results flawed. In addition, in order to obtain reasonable deformation without human-computer interaction, the deformation is constrained by smooth loss. Therefore, our company proposes to use a deformation generation network with a self-attention mechanism to adjust and change the structure of the 3D model, so as to generate a new model that is different from the original shape. Summary of the invention
[0005] The purpose of the present invention is to provide a method for a deformation generation model based on an attention mechanism in response to the defects and shortcomings of the prior art. It addresses the shortcomings of several current deformation technologies, focuses on the influence of noise generated during the deformation process on the deformation results, and constructs a deformation generation model architecture based on the attention mechanism.
[0006] The method of the deformation generation model based on the attention mechanism described in the present invention adopts the following steps:
[0007] S10: First, prepare a 3D model dataset containing any size and number, and the format of the 3D model dataset is .ply; perform necessary preprocessing on all models, and the corresponding preprocessing file implicitly represents the dataset through the signed distance function, and the format of the dataset is .mat.
[0008] S20: Design a deformation generation model using the following steps:
[0009] S2010: first, create a shape information encoding module, which assigns different latent codes according to different 3D models, and assigns network weights ω to subsequent models through the latent codes α;
[0010] S2020: Create a deformation module, which consists of a deformation field and a template field, and obtain the corresponding deformation vector V through the deformation model; finally, introduce a self-attention module to optimize the deformation vector generated by the deformation module;
[0011] S30: Training the deformation generation model, using the following steps:
[0012] S3010: assigning a corresponding latent code α to a point P of the input 3D model, and assigning a corresponding deformation network weight ω to the point P according to the latent code; then, inputting the point P and the corresponding network weight into the deformation module, thereby obtaining a corresponding implicit deformation vector V';
[0013] S3020: Input the implicit deformation vector V' into the deformation template field to form the corresponding implicit deformation template field. Then, use the implicit deformation vector V' to obtain the final deformation vector V through the mapping function, and finally add the deformation vector V and the original point P to obtain the final deformation result; and calculate the implicit loss function L SDF , regularization loss L reg And the smooth loss function L smooth , as the loss of the entire deformation network;
[0014] S3030: Combine three losses to back-propagate and update the weights of the deformation network;
[0015] S40: Repeat step S30 until a preset number of iterations is reached, which is 70 rounds by default; all three-dimensional models are traversed in each round; at the end of each round of iteration, the parameters of the model are saved.
[0016] Furthermore, the attention mechanism-based deformation generation model architecture in step S30 adopts the design of deformation implicit template field and attention mechanism.
[0017] Furthermore, the deformation generation model architecture based on the attention mechanism in step S30 constrains the deformation process by smoothing the loss to prevent the deformation amplitude from being too large.
[0018] Furthermore, in step S30, a deformation generation model architecture based on an attention mechanism and a signed distance field is introduced; during the training process, a self-attention module is introduced, which not only establishes the interdependence between deformation vector channels, but also eliminates the noise generated during the deformation process; the back-propagation mechanism is used to transmit the loss information back to the deformation generation model, and the optimization update is performed.
[0019] The beneficial effects of the present invention are as follows: the method of the deformation generation model based on the attention mechanism described in the present invention shows excellent generation effect on the three-dimensional model data set. By cleverly combining the deformation implicit field model architecture and the self-attention module, it fully utilizes the attention mechanism to capture global context information, so that the model can more comprehensively establish the connection between the model positions, thereby eliminating noise. This design makes the deformation generation results more complete and reasonable. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:
[0021] Figure 1 The attention mechanism-based deformation generation model architecture of the present invention;
[0022] Figure 2 This is a flow chart of model training of the present invention;
[0023] Figure 3 The self-attention module architecture of the present invention;
[0024] Figure 4 It is a comparison diagram of the experimental results of the present invention;
[0025] Figure 5 More test images of the present invention. DETAILED DESCRIPTION
[0026] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.
[0027] like Figure 1 As shown, the method for generating a deformation model based on an attention mechanism described in this specific embodiment adopts the following steps:
[0028] S10: First, prepare a 3D model dataset containing any size and number, and the format of the 3D model dataset is .ply; perform necessary preprocessing on all models, and the corresponding preprocessing file implicitly represents the dataset through the signed distance function, and the format of the dataset is .mat.
[0029] S20: Design a deformation generation model using the following steps:
[0030] S2010: first, create a shape information encoding module, which assigns different latent codes according to different 3D models, and assigns network weights ω to subsequent models through the latent codes α;
[0031] S2020: Create a deformation module, which consists of a deformation field and a template field, and obtain the corresponding deformation vector V through the deformation model; finally, introduce a self-attention module to optimize the deformation vector generated by the deformation module;
[0032] S30: Training the deformation generation model, using the following steps:
[0033] S3010: assigning a corresponding latent code α to a point P of the input 3D model, and assigning a corresponding deformation network weight ω to the point P according to the latent code; then, inputting the point P and the corresponding network weight into the deformation module, thereby obtaining a corresponding implicit deformation vector V';
[0034] S3020: Input the implicit deformation vector V' into the deformation template field to form the corresponding implicit deformation template field. Then, use the implicit deformation vector V' to obtain the final deformation vector V through the mapping function, and finally add the deformation vector V and the original point P to obtain the final deformation result; and calculate the implicit loss function L SDF , regularization loss L reg And the smooth loss function L smooth , as the loss of the entire deformation network;
[0035] S3030: Combine three losses to back-propagate and update the weights of the deformation network;
[0036] S40: Repeat step S30 until a preset number of iterations is reached, which is 70 rounds by default; all three-dimensional models are traversed in each round. At the end of each round of iteration, the parameters of the model are saved.
[0037] Furthermore, the attention mechanism-based deformation generation model architecture in step S30 adopts the design of deformation implicit template field and attention mechanism.
[0038] Furthermore, the deformation generation model architecture based on the attention mechanism in step S30 constrains the deformation process by smoothing the loss to prevent the deformation amplitude from being too large.
[0039] Furthermore, in step S30, a deformation generation model architecture based on an attention mechanism and a signed distance field is introduced; during the training process, a self-attention module is introduced, which not only establishes the interdependence between deformation vector channels, but also eliminates the noise generated during the deformation process; the back-propagation mechanism is used to transmit the loss information back to the deformation generation model, and the optimization update is performed.
[0040] The principle of the present invention is described as follows:
[0041] Step S10: First, prepare a 3D model dataset (.ply format) containing any size and quantity. In the initial step, the point information is classified into data that is not on the model surface and data that is on the model surface through a custom function, and the final implicit expression is obtained through the concatenate() function.
[0042] Step S20: In order to eliminate the noise generated during the deformation process and obtain a reasonable and complete deformation result, we propose to use the attention mechanism combined with the deformation implicit template field to control the deformation result different from the original model. Figure 1 As shown, our model consists of a shape information encoding module, a deformation module, and a self-attention module.
[0043] About the shape information encoding module: The module consists of 4 fully connected layers, each with a dimension of 128. Here, we hope to train a potential shape space S and a neural shape deformation model that can generate such models and provide dense corresponding shapes. Each shape can be encoded with a latent code α∈R in the latent space S. k Representation, the shape model can map the latent code to the corresponding 3D shape.
[0044] And the signed distance function (SDF) is introduced. SDF is mainly used to measure whether a point is inside or outside the 3D shape. Therefore, using SDF can be used as a field function to represent the geometric details of the surface through a neural network. It is worth noting that SDF is a continuous representation, so it is for each point p∈R 3 Assign a scalar value s∈R: SDF(p)=s, where the scalar value s represents the closest distance from the 3D point p to the surface of the shape, and the sign indicates whether the 3D point p is inside the shape (negative sign) or outside (positive sign). When SDF(p)=0, it means that the point is on the surface of the shape. After using SDF to represent the shape, the neural deformable shape model is represented as:
[0045] (α, p)∈R k+3 →s∈R;
[0046] The above representation combines the potential decoded shape information corresponding to each shape with the SDF prediction of the 3D point. The deformation module weight of each 3D model can be predicted by the shape information encoding module. The process of predicting the weight is: α∈R k →ω∈R m ;
[0047] Given a set of category shapes, the SDF loss is applied to learn the SDF value of this category. The specific SDF loss is:
[0048]
[0049] At the same time, in order to learn better potential coding, regularization loss is introduced to constrain it. The specific loss function formula is:
[0050]
[0051] About the deformation module: This module is mainly composed of deformation field and template field. In order to obtain a deformation vector of the 3D model category, we predict a deformation vector through the deformation field. The specific prediction process is as follows:
[0052] p∈R 3 →v′∈R 3 ; where v' is the deformation vector predicted by the deformation field. It should be noted that the weight of the deformation field is instance-specific and comes from the shape information encoding module.
[0053] The second is the template field, which is used to capture the common structure of the 3D model category. The formula for generating SDF from the template field is: v′∈R 3 →s∈R;
[0054] The template field maps the points of the 3D model to a scalar value s, which is mainly used to construct the SDF of the object through deformation. The network weights of the template field will be shared across the entire category, so it is necessary to enforce learning a common template across the category.
[0055] In summary, after the deformation field processes the 3D model, the final SDF value s of the deformed shape can be obtained, which can be expressed as: s = T (p + v) = T (p + D (p));
[0056] If no constraints are added during the deformation process, the deformation amplitude will be too large, resulting in some unreasonable results. Therefore, in order to promote smooth deformation and ensure that large deformation can be avoided during the deformation process, a simple smoothing loss is added to the deformation field. The smoothing loss formula is:
[0057]
[0058] This loss function penalizes the spatial gradients of the deformation field along the X, Y, and Z directions.
[0059] Then, in order to ensure the integrity of the final deformation result, the deformation vector obtained by the deformation template is passed through the self-attention module to capture the global information, so as to establish the dependency relationship between each other.
[0060] like Figure 2 As shown, given the deformation vector v'∈R N×3 In the case of , first v' passes through the convolutional layer to obtain three matrices Q, K and V, where {Q, K, V}∈R N×3 After obtaining the three matrices, perform matrix multiplication on the transpose of matrix Q and matrix K, and then use the softmax layer to process the operation results to obtain the corresponding self-attention matrix Z∈R N×N , where the specific calculation formula for the matrix Z is as follows:
[0061]
[0062] where z ij It represents the influence of the i-th position on the j-th position. The more similar the features of the two positions are, the greater the correlation between the two positions.
[0063] After obtaining the matrix Z, perform matrix multiplication on the matrix V and the transpose of the matrix Z, and finally multiply the result with the scale parameter α, and perform row-by-row element addition on the original input v' to obtain the final deformation vector v∈R N×3 , the specific calculation formula of the deformation vector is:
[0064]
[0065] α is initialized to 0, and gradually learns to assign more weights. From the formula, we can see that v captures global information, so that each position establishes a mutually dependent relationship.
[0066] Step 3: Model training process, such as Figure 3As shown. Blue represents the various modules of the present invention, and green represents the output of each module. Training deformation model framework: First, the potential code corresponding to each shape is input into the shape information encoding module to generate the deformation network weight, and the regularization loss is used to promote the network to learn better potential codes, and the SDF loss is used to learn a category of SDF values. Then, the weight and point information are input into the deformation field to obtain the implicit deformation vector, and then the smoothing loss is calculated to prevent the deformation amplitude from being too large. Then, the deformation template field is formed through the template field, and then the deformation vector is obtained through mapping, and finally the noise generated during the deformation process is eliminated through the self-attention module.
[0067] Step S40: Repeat step S30 until a preset number of iterations (70 rounds by default) is reached, and all three-dimensional model data sets are traversed in each round. At the end of each round of training, the parameters of the model are saved.
[0068] Specific experimental data and parameters: specific comparison results with the baseline model, such as Figure 4 The first row shows the input 3D model, the second row shows the deformation result of the baseline model, and the third row shows the deformation result of the present invention. The specific experimental details are as follows:
[0069] Step A: The training data includes a 3D model dataset. The dataset is a large-scale shape repository ShapeNet with rich annotations provided by the Stanford University team. The dataset contains 3D models from multiple semantic categories and organizes them according to WordNet classification. It is a dataset collection that provides many semantic annotations for each 3D model. The annotations are provided through a public web-based interface to enable dataset visualization of object properties and provide a large-scale quantitative benchmark for computer graphics and vision research. The ShapeNet dataset has indexed more than 3,000,000 models, of which 220,000 models are classified into 3,135 categories, and each model has original CAD files, mesh files, and rendered images. It took about 72 hours to train on two NVIDIA 3090 GPUs with 24GB of memory. The training batch size is Batch=2, the learning rate is 1e-4, the deformation loss gradient is 5, the iteration period is 70 epochs, the number of hidden layers is 128, and the potential encoding dimension is 128.
[0070] Step B: The present invention uses the training parameters of the 60th epoch to perform deformation testing. From the visualization results, it can be seen that the deformation results of the present invention are more complete and reasonable than the deformation results of the baseline model on the 3D ShapeNet dataset.
[0071] Step C: In addition, Figure 5More deformation results of the present invention are shown, and the research results show that the model did not overfit the training three-dimensional model during the training process. This proves the good generalization performance of the model when learning and deforming to generate three-dimensional models of different parts. This finding emphasizes the model's ability to handle the deformation of three-dimensional models, indicating that the network can learn the characteristics of the three-dimensional model and adjust its structure. This is crucial for the practical application of the generative model, especially the ability to freely deform when faced with different models. Therefore, the model not only retains the original structure of the current three-dimensional model, but also eliminates noise when adjusting or changing the original shape, making the final deformation result more complete and reasonable.
[0072] In the present invention, the given 3D ShapeNet dataset is first preprocessed, and the information of the points is implicitly expressed using a signed distance function. Secondly, a network framework based on a shape information encoding module, a deformation module and a self-attention module is designed. The shape information encoder assigns corresponding latent codes to each shape and assigns weights to the corresponding deformation modules according to the latent codes. The deformation module is mainly composed of a deformation field and a template field. The deformation field uses the information of the three-dimensional model points and the corresponding network weights to generate an implicit deformation vector for the model; the template field forms a corresponding template field through the implicit deformation vector, and then maps the implicit deformation vector back to the corresponding deformation vector through a mapping function. The self-attention module mainly captures the global information of the corresponding deformation vector and optimizes it to achieve the effect of eliminating noise.
[0073] In the present invention, during the model training and inference phases, the deformation model is optimized mainly using regularization loss, smoothing loss and SDF value loss. Ultimately, the technology realizes a generative model capable of deforming a three-dimensional model.
[0074] Based on the adoption of the 3D ShapeNet dataset, this paper establishes a more stable deformation network, which significantly improves the quality of the deformation results generated by the deformation network. The technical innovation lies in the integration of the attention mechanism and the implicit deformation network, which optimizes the problem of defects in the deformation results. By constraining the smooth loss function of the deformation, the deformation network can better control the deformation amplitude, providing an effective and stable solution for generating high-quality deformation results.
[0075] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the features and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.
Claims
1. A method for a deformation generation model based on an attention mechanism, characterized in that: Use the following steps: S10: First, prepare a 3D model dataset containing any size and number, the format of the 3D model dataset is .ply; perform necessary preprocessing on all models, the corresponding preprocessing file implicitly represents the dataset through the signed distance function, the format of the dataset is .mat; S20: Design a deformation generation model using the following steps: S2010: first, create a shape information encoding module, which assigns different latent codes according to different 3D models, and assigns network weights (ω) to subsequent models through the latent codes (α); S2020: Create a deformation module, which consists of a deformation field and a template field, and obtain the corresponding deformation vector (v) through the deformation model; finally, introduce a self-attention module to optimize the deformation vector generated by the deformation module; S30: Training the deformation generation model, using the following steps: S3010: assigning a corresponding latent code (α) to a point (p) of the input 3D model, and assigning a corresponding deformation network weight (ω) to the point according to the latent code; then, inputting the point (p) and the corresponding network weight into the deformation module, thereby obtaining a corresponding implicit deformation vector (v′); S3020: Input the implicit deformation vector (v′) into the deformation template field to form the corresponding implicit deformation template field. Then, use the implicit deformation vector (v′) to obtain the final deformation vector v through the mapping function, and finally add the deformation vector (v) and the original point (p) to obtain the final deformation result; and calculate the implicit loss function L SDF , regularization loss L reg And the smooth loss function L smooth , as the loss of the entire deformation network; S3030: Combine three losses to back-propagate and update the weights of the deformation network; S40: Repeat step S30 until a preset number of iterations is reached, which is 70 rounds by default; all three-dimensional models are traversed in each round; at the end of each round of iteration, the parameters of the model are saved.
2. The method for generating a deformation model based on an attention mechanism according to claim 1, characterized in that: The attention-based deformation generation model architecture in step S30 adopts the design of deformation implicit template field and attention mechanism.
3. The method for generating a deformation model based on an attention mechanism according to claim 1, characterized in that: The deformation generation model architecture based on the attention mechanism in step S30 constrains the deformation process by smoothing the loss to prevent the deformation amplitude from being too large.
4. The method for generating a deformation model based on an attention mechanism according to claim 1, characterized in that: The deformation generation model architecture based on the attention mechanism and the signed distance field in step S30; During the training process, a self-attention module is introduced, which not only establishes the interdependence between deformation vector channels, but also eliminates the noise generated during the deformation process; the back-propagation mechanism is used to pass the loss information back to the deformation generation model and perform optimization updates.