3D mesh generation method
By using structured latent space point representation and FlexiCubes mesh parameterization, the problems of insufficient detail, low computational efficiency, and training complexity in the generation of 3D shapes in existing technologies are solved, and high-quality, low-resource-consumption end-to-end 3D mesh generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING WAZIDA TECH CO LTD
- Filing Date
- 2025-04-25
- Publication Date
- 2026-06-30
AI Technical Summary
Existing 3D shape generation methods struggle to capture local details, have low computational efficiency, rely on multi-stage training processes, are difficult to train end-to-end, lack fine control over the generation process, and depend on complex preprocessing or post-processing.
A structured latent space point representation method is adopted to map disordered point clouds onto regular meshes. The latent space encoding is processed using a VAE encoder and a diffusion model. The FlexiCubes mesh is directly parameterized, and a 3D mesh is generated by aggregating feature vectors through a weighted radial basis function kernel.
It achieves high-fidelity and high-efficiency 3D mesh generation, reduces computational resource requirements, simplifies the training process, provides fine control capabilities, and avoids errors introduced by post-processing steps.
Smart Images
Figure CN120411376B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for generating three-dimensional meshes. Background Technology
[0002] 3D shape generation is an important research area in computer graphics, computer vision, and artificial intelligence. Its goal is to generate 3D shapes with specific characteristics based on given input (such as text, sketches, images, etc.) or random noise. With the rapid development of deep learning technology, generative models based on deep neural networks have made significant progress in the field of 3D shape generation.
[0003] The existing technology is as follows:
[0004] 3DShape2VecSet uses a VAE to learn a global latent vector to represent the entire 3D shape. The decoder outputs an implicit representation (SDF), which requires the Marching Cubes algorithm to extract the mesh. This method is simple and compact, but it struggles to capture local details and relies on post-processing steps.
[0005] XCube represents the latent space as a sparse VDB structure and processes it using sparse convolutions. The decoder outputs an implicit representation, which requires Marching Cubes to extract the mesh. This method improves efficiency to some extent, but still relies on a multi-stage training process and post-processing steps.
[0006] The TRELLIS method represents the latent space as a sparse grid and processes it using sparse convolution. The decoder outputs an implicit representation, which requires Marching Cubes to extract the grid.
[0007] The OctFusion method uses a hierarchical octree structure to represent the latent space and processes it using appropriate convolution operations. The decoder outputs an implicit representation, which requires Marching Cubes to extract the mesh.
[0008] In the process of implementing this technical solution, the inventors discovered the following defects in the prior art:
[0009] 1. Lack of attention to detail:
[0010] Methods based on global implicit vectors (such as 3DShape2VecSet, LaGeM, DORA) typically use a compact vector to represent the entire shape. This representation makes it difficult to capture the local details of the shape, resulting in a smoother mesh that lacks detail.
[0011] Existing techniques attempt to use structured latent spaces (such as 3DILG, XCube, TRELLIS, OctFusion), but they typically still rely on implicit representations (such as SDF) as output, requiring additional post-processing steps (such as MarchingCubes) to extract the mesh. This introduces additional computational overhead and may lead to artifacts in the generated results, such as uneven surfaces or holes.
[0012] 2. Low computational efficiency:
[0013] Voxel-based methods (such as early 3D-GANs) suffer from computational and memory overhead that increases cubically with resolution, severely limiting the resolution and level of detail of the generated models.
[0014] Many methods rely on multi-stage training processes, such as training a low-resolution model first and then gradually increasing the resolution, or requiring preprocessing data to build structured representations, which increases the complexity and time cost of training. Post-processing steps for implicit representations (such as Marching Cubes) also consume significant computational resources.
[0015] 3. Difficult to train end-to-end:
[0016] Many methods (such as 3DShape2VecSet, LaGeM, 3DILG, XCube, TRELLIS, OctFusion) have multiple stages in their generation process, such as first generating implicit representations and then extracting meshes. These stages are usually optimized independently, making end-to-end training difficult and limiting the optimization effect and generation efficiency of the model.
[0017] 4. Lack of fine-grained control over the generation process:
[0018] Global latent vectors make it difficult to finely control the generated shape; for example, it is difficult to modify the local details of the shape or adjust its pose.
[0019] 5. Relies on complex preprocessing or postprocessing:
[0020] Many methods require complex preprocessing before training, such as converting point clouds or meshes into voxels or SDF representations. Many methods require complex post-processing after generation, such as extracting meshes from SDF using Marching Cubes or performing Poisson reconstruction on the point cloud. Summary of the Invention
[0021] In view of the problems existing in the prior art, the present invention provides a three-dimensional mesh generation method, which at least partially solves the problem of insufficient detail in the prior art.
[0022] This disclosure provides a method for generating a three-dimensional mesh, including:
[0023] Obtain an unordered point cloud;
[0024] Unordered point clouds are mapped onto regular grids to obtain structured point clouds;
[0025] The structured point cloud is encoded into the latent space to obtain the first latent space encoding;
[0026] The second latent space code is obtained by processing the first latent space code using the trained diffusion model.
[0027] Structured latent space points are obtained based on the second latent space encoding;
[0028] FlexiCubes mesh is obtained based on structured latent space points;
[0029] Extracting a 3D mesh from a FlexiCubes mesh.
[0030] Optionally, mapping the disordered point cloud to a regular grid to obtain a structured point cloud includes:
[0031] Use optimal transport to map unordered point clouds onto regular grids.
[0032] Optionally, the points in the structured point cloud include assigned grid positions and offsets;
[0033] The structured point cloud includes position offset and feature vector. The position offset is used to represent the offset of a latent point relative to its grid position, and the feature vector is used to represent the local geometric features of the latent point.
[0034] The parameters stored in each grid cell of the FlexiCubes grid include:
[0035] Split weights are used to control the position of dual vertices;
[0036] Interpolation weights are used to control the division of the triangle;
[0037] And deformation vectors, used to control the spatial position of mesh vertices.
[0038] Optionally, FlexiCubes meshes are obtained based on structured latent space points, including:
[0039] The eigenvectors of latent space points are aggregated using a weighted radial basis function kernel;
[0040] Calculate the parameters of each voxel in the FlexiCubes mesh based on the feature vectors;
[0041] The method of aggregating structured latent space point feature vectors using weighted radial basis function kernels includes:
[0042] Aggregate the k structured latent space points that are closest to each FlexiCubes grid.
[0043] Optionally, FlexiCubes meshes are obtained based on structured latent space points, including:
[0044] Multiple structured latent space points are obtained by aggregating the initial structured latent space points;
[0045] Voxelize multiple structured latent space points onto a set mesh, the resolution of which is lower than that of the target FlexiCube mesh;
[0046] For each occupying voxel in the set grid, sample its corresponding 8 voxels in the target FlexiCube grid to obtain a voxel set;
[0047] Take the 26 neighborhoods from the voxel set to obtain the sampling set;
[0048] FlexiCubes meshes are calculated based on the sample set.
[0049] Optionally, encoding the structured point cloud into a latent space to obtain a first latent space encoding includes:
[0050] The structured point cloud is encoded into the latent space using a VAE encoder;
[0051] The process of using a trained diffusion model to process the first latent space code to obtain the second latent space code includes:
[0052] The second latent space encoding is obtained by performing T iterations using the trained diffusion model.
[0053] The process of obtaining structured latent space points based on the second latent space encoding includes:
[0054] The second latent space is encoded and decoded into structured latent space points using a VAE decoder.
[0055] Optionally, the VAE encoder includes:
[0056] 3D convolutional layers use convolutions with strides to downsample, gradually reducing the spatial resolution of the feature maps;
[0057] Transformer layers are applied to the downsampled feature maps to capture global dependencies;
[0058] The first fully connected layer is used to output the mean and log-variance of the latent space distribution;
[0059] Latent space coding is used to sample latent space distributions to obtain latent space codes.
[0060] Optionally, the VAE decoder includes:
[0061] The second fully connected layer maps the latent space encoding to the initial feature map;
[0062] The 3D deconvolution layer uses deconvolution with stride for upsampling, gradually increasing the spatial resolution of the feature map.
[0063] The output layer is used to predict the position offset and feature vector of each structured latent space point;
[0064] FlexiCubes parameterization uses structured latent space points to parameterize FlexiCubes meshes;
[0065] Mesh extraction: Extract the 3D mesh from the parameterized FlexiCubes mesh using isosurfaces. In mesh extraction, sparse convolutional layers are applied to upsample the FlexiCubes mesh.
[0066] Optionally, the trained diffusion model includes:
[0067] The input layer uses convolutional layers with strides to downsample the latent space encoding of the input;
[0068] The intermediate layer contains multiple stacked attention blocks, which are used to incorporate temporal and category-conditional embeddings.
[0069] The upsampling layer uses a transposed convolutional layer with stride to upsample the feature map.
[0070] Optionally, the forward pass of the diffusion model training process adds Gaussian noise to the latent space encoding over T time steps according to variance scheduling.
[0071] The three-dimensional mesh generation method provided by this invention combines the flexibility of point clouds with the processing efficiency of structured data, and directly generates a three-dimensional network from a parameterized differentiable FlexiCubes mesh, thereby improving the generation quality and preserving more details, and achieving the goal of generating a high-fidelity mesh with richer details. Attached Figure Description
[0072] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.
[0073] Figure 1 A flowchart of a three-dimensional mesh generation method provided in this embodiment of the disclosure;
[0074] Figure 2 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0075] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0076] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0077] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0078] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0079] The trade-off between high computational and memory requirements and detail preservation. High-fidelity 3D shape generation, especially mesh generation with fine geometric details, places extremely high demands on computational and memory resources. Traditional voxel-based methods, while leveraging the efficient processing power of convolutional neural networks (CNNs), suffer from memory requirements that increase cubically with resolution, severely limiting the resolution and level of detail of the generated models. Sparse voxel representations (such as octrees) can alleviate some of the problem, but they still struggle to handle extremely complex geometries.
[0080] Balancing efficiency and controllability in latent space representation. Existing 3D shape generation methods typically rely on latent space representations to learn and generate shapes. While global latent vectors are compact, they struggle to capture local details and provide fine-grained control. Methods based on structured latent spaces (such as arranging latent vectors on regular meshes or octrees) improve the representation of local details to some extent, but they often rely on multi-stage training processes, making end-to-end optimization difficult. Furthermore, they still require post-processing steps (such as Marching Cubes) to extract the final mesh, which introduces additional computational overhead and may lead to artifacts in the generated results.
[0081] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0082] The 3D mesh generation method in this embodiment adopts the "Structured Latent Points" representation for high-fidelity 3D mesh generation. It combines the flexibility of point clouds with the processing efficiency of structured data and directly parameterizes a differentiable mesh representation (FlexiCubes), thereby achieving efficient end-to-end training and high-quality mesh generation.
[0083] For ease of understanding, such as Figure 1 As shown, this embodiment discloses a three-dimensional mesh generation method, including:
[0084] Step S101: Obtain the unordered point cloud;
[0085] Step S102: Map the disordered point cloud onto a regular grid to obtain a structured point cloud;
[0086] This embodiment is a structured point cloud built based on Optimal Transport, combining the flexibility of point clouds with the processing efficiency of structured data.
[0087] Given an input of an unordered point cloud, the goal of this embodiment is to map this unordered point cloud to a regular 3D mesh. The optimal transport problem aims to find a mapping that transforms one probability distribution (point cloud in this embodiment) into another probability distribution (regular mesh in this embodiment) while minimizing the total transport cost. In this embodiment, squared Euclidean distance is used as the transport cost function.
[0088] The structured point cloud disclosed in this embodiment contains two pieces of information: the assigned grid position and the offset. This representation retains the flexibility of point clouds (through offset vectors) while also possessing the regularity of structured data (through grid positions), enabling it to be efficiently processed by 3D convolution.
[0089] Step S103: Encode the structured point cloud into the latent space to obtain the first latent space encoding;
[0090] Optionally, encoding the structured point cloud into a latent space to obtain a first latent space encoding includes:
[0091] The structured point cloud is encoded into the latent space using a VAE encoder;
[0092] Step S104: Process the first latent space code using the trained diffusion model to obtain the second latent space code;
[0093] The process of using a trained diffusion model to process the first latent space code to obtain the second latent space code includes:
[0094] The second latent space encoding is obtained by performing T iterations using the trained diffusion model.
[0095] Step S105: Obtain structured latent space points based on the second latent space encoding;
[0096] The process of obtaining structured latent space points based on the second latent space encoding includes:
[0097] The second latent space is encoded and decoded into structured latent space points using a VAE decoder.
[0098] Step S106: Obtain FlexiCubes mesh based on structured latent space points;
[0099] Step S107: Extract the 3D mesh from the FlexiCubes mesh.
[0100] The input unordered point cloud is transformed into a structured representation, and structured latent space points are defined to directly control the differentiable FlexiCubes mesh representation.
[0101] Each structured latent space point is associated with a grid location (from the optimal transport step) and comprises the following two components:
[0102] Position Offset: Represents the offset of a hidden space point relative to its grid position.
[0103] Feature vector: Represents the local geometric features of a point in the latent space.
[0104] Parameterizing the FlexiCubes Mesh: Given structured latent space points, we use them to parameterize the FlexiCubes mesh. For each voxel in the FlexiCubes mesh, its parameters are computed by aggregating the eigenvectors of nearby structured latent space points. For a voxel location, a weighted radial basis function (RBF) kernel is used to aggregate the eigenvectors of nearby structured latent space points.
[0105] To improve computational efficiency, only the k structured latent space points closest to each FlexiCubes grid are considered.
[0106] Optionally, FlexiCubes meshes are obtained based on structured latent space points, including:
[0107] Multiple structured latent space points are obtained by aggregating the initial structured latent space points;
[0108] Voxelize multiple structured latent space points onto a set mesh, the resolution of which is lower than that of the target FlexiCube mesh;
[0109] For each occupying voxel in the set grid, sample its corresponding 8 voxels in the target FlexiCube grid to obtain a voxel set;
[0110] Take the 26 neighborhoods from the voxel set to obtain the sampling set;
[0111] FlexiCubes meshes are calculated based on the sample set.
[0112] The target FlexiCube mesh is the calculated FlexiCubes mesh, whose resolution and voxels are set according to the target FlexiCube mesh.
[0113] Specifically, to further improve computational efficiency, a conservative sampling strategy is adopted, calculating only the parameters of voxels close to the reconstructed surface. The specific steps are as follows:
[0114] 1. Initial point: Start from the position of the structured latent space points, which are located within the FlexiCubes mesh.
[0115] 2. Coarse Voxelization: Voxelize these points onto a coarser mesh with a resolution half that of the target FlexiCube mesh.
[0116] 3. Dilation: For each occupying voxel in the coarse mesh, sample its corresponding 8 voxels in the target FlexiCube mesh.
[0117] 4. Neighborhood expansion: For the expanded voxel set, take its 26-neighborhood to obtain the final sampling set.
[0118] 5. Voxel parameters are calculated only for the final sample set.
[0119] This conservative sampling strategy significantly reduces computation while ensuring the quality of the reconstructed surface.
[0120] FlexiCubes is a differentiable mesh representation method that improves upon existing Marching Cubes algorithms by introducing additional parameters to control the mesh geometry and topology. Unlike Marching Cubes, which directly determines isosurfaces based on voxel mesh values, FlexiCubes stores the following parameters in each mesh cell:
[0121] Splitting Weights: Used to control the position of dual vertices.
[0122] Interpolation weights: Used to control the segmentation of triangles.
[0123] Deformation Vectors: Used to control the spatial position of mesh vertices.
[0124] These parameters allow FlexiCubes to represent complex geometries and topologies more flexibly while maintaining differentiability, enabling seamless integration with deep learning frameworks.
[0125] This embodiment constructs a variational autoencoder (VAE) to encode the structured point cloud into a compact latent space and decodes the parameters of the structured latent space points from the latent space.
[0126] VAE consists of an encoder and a decoder.
[0127] Encoder:
[0128] The encoder receives a structured point cloud as input, where each point is stored in a 32-bit grid. Because the points are organized on a regular grid, efficient processing can be achieved using 3D convolutional layers. The encoder contains the following components:
[0129] 1. A series of 3D convolutional layers: Downsampling is performed using convolutions with strides to progressively reduce the spatial resolution of the feature maps.
[0130] 2. Transformer layer: Eight Transformer layers are applied to the downsampled feature map to capture global dependencies.
[0131] 3. Fully connected layer: Two fully connected layers are used to output the mean and log-variance of the latent space distribution, respectively.
[0132] 4. Latent space coding: The latent space code z is obtained by sampling from the latent space distribution. The resolution of the latent space z is 16.
[0133] Decoder:
[0134] The decoder receives the latent space code z as input and reconstructs the structured latent space points. The decoder's structure is mirror-symmetric to the encoder and includes the following components:
[0135] 1. Fully connected layer: Maps the latent space encoding z to the initial feature map.
[0136] 2. A series of 3D deconvolution layers: Upsampling is performed using deconvolution with stride to gradually increase the spatial resolution of the feature map.
[0137] 3. Output layer: The output layer predicts the position offset and feature vector of each structured latent space point.
[0138] 4. FlexiCubes parameterization: The FlexiCubes mesh is parameterized using structured latent space points according to the method described in this embodiment.
[0139] 5. Mesh Extraction: The final 3D mesh is extracted from the parameterized FlexiCubes mesh using the FlexiCubes isosurface extraction process. To obtain a smoother, finer surface, two sparse convolutional layers are applied to upsample the FlexiCubes mesh to a resolution of 256 for the final mesh extraction.
[0140] The training of VAE is end-to-end, using a combination of the following loss functions:
[0141] Reconstruction Loss: Using multi-view rendering as a supervisory signal, this loss compares the differences between the rendered image (depth, mask, and normals) and the ground truth image. L1 loss is used to calculate the differences between the normal map and the mask map, and mean squared error (MSE) is used to calculate the differences in the depth map.
[0142] KL Divergence Loss: Used to regularize the latent space, making it approximate the standard normal distribution.
[0143] Regularization Loss: Uses a loss derived from standard FlexiCubes optimizations to normalize the internal representation.
[0144] To accelerate the training process, a warm-up phase is used, where the VAE is initially trained using only the Chamfer distance to encourage the locations of the decoded structured latent space points to be roughly aligned with the input point cloud.
[0145] This embodiment trains a diffusion model on the latent space of a VAE to generate new latent space encodings, and then uses the VAE decoder to generate new 3D meshes. The diffusion model uses a denoising diffusion probability model (DDPM) based on a U-Net structure. The input and output of the diffusion model have the same dimensions as the latent space of the VAE.
[0146] The specific architecture of the diffusion model is as follows:
[0147] 1. Input layer: Use a convolutional layer with stride to downsample the latent space encoding of the input by a factor of 2.
[0148] 2. Intermediate layer: Contains 16 densely stacked attention blocks (Transformer blocks are only applied to downsampled low-resolution features). Each attention block contains 2 AdaIN layers for incorporating temporal and class-conditional embeddings.
[0149] 3. Upsampling layer: The feature map is upsampled by a factor of 2 using a transposed convolutional layer with stride, restoring it to its original size, mirroring the downsampling structure.
[0150] 4. Initial network resolution: Contains only residual convolutional layers.
[0151] The training of the diffusion model follows the standard DDPM process. The forward process adds Gaussian noise to the latent space code z over T time steps according to variance scheduling. Cosine scheduling is used, with T=1000. The training loss is the mean square error between the predicted noise and the actual added noise.
[0152] To generate new 3D shapes, a latent space code z is first sampled from a standard normal distribution. Then, a trained diffusion model is used for T-step iterative denoising to obtain the denoised latent space code z. Finally, z is input into a VAE decoder to generate structured latent space points, which are then further used to generate FlexiCubes meshes.
[0153] The "structured latent space point" representation method proposed in this embodiment cleverly combines the flexibility of point clouds with the processing efficiency of structured data, and directly parameterizes a differentiable mesh representation, achieving efficient end-to-end training and high-fidelity mesh generation. This method significantly reduces the training budget and improves generation efficiency while ensuring mesh quality, and provides convenience for subsequent shape editing and manipulation.
[0154] Structured Latent Space Point Representation: This embodiment uses structured latent space point representation, which combines the flexibility of point clouds (reflected by the offset of each point relative to its grid cell) with the efficient processing capabilities of structured data (by organizing points onto a regular grid). This representation method is key to achieving high-fidelity, high-efficiency 3D mesh generation.
[0155] This embodiment uses the Optimal Transport algorithm to map the input unordered point cloud onto a regular 3D mesh, assigning each point a mesh position and an offset vector. This method ensures that the point cloud is structured while retaining its ability to represent details.
[0156] This implementation uses a structured latent space point to directly parameterize a differentiable FlexiCubes mesh representation. Each latent space point contributes to the parameters of nearby FlexiCubes mesh cells using a weighted radial basis function (RBF) kernel function through its position offset and eigenvector. This direct parameterization method avoids the conversion process from implicit representation (such as SDF) to mesh in traditional methods, improving generation efficiency and quality.
[0157] Compared with existing technologies (such as 3DShape2VecSet, LaGeM, 3DILG, XCube, TRELLIS, OctFusion, Mosaic-SDF, 3DTopia, etc.), the advantages of this embodiment are as follows:
[0158] 1. Higher Generation Quality and Detail Preservation: Existing techniques often struggle to generate high-quality meshes while retaining fine geometric details. The structured latent space point representation method proposed in this embodiment combines the flexibility of point clouds with the processing efficiency of structured data and directly parameterizes differentiable FlexiCubes meshes, enabling the generation of high-fidelity meshes with richer details. This is because structured latent space points allow for finer control over the local geometry of the mesh, and FlexiCubes representation itself possesses greater expressive power than traditional Marching Cubes.
[0159] 2. Higher computational efficiency and lower training budget: Existing technologies, especially voxel-based methods and some implicit representation-based methods, typically have high computational and memory overhead, resulting in high training costs. This embodiment improves computational efficiency in the following ways:
[0160] Structured point cloud representation: allows for the use of efficient 3D convolution operations.
[0161] Optimal transport: Efficiently organize point clouds onto a regular grid.
[0162] Conservative sampling strategy: Only calculate FlexiCubes parameters for voxels close to the prediction surface.
[0163] End-to-end training: avoids multi-stage training and complex pre-processing / post-processing steps.
[0164] These improvements enable this embodiment to significantly reduce the computational resource requirements and training time while maintaining the quality of the generated data.
[0165] 3. End-to-End Training Process: Many existing techniques struggle to achieve end-to-end training from latent space encoding to final mesh generation, typically requiring multiple independent training stages or complex preprocessing / postprocessing steps. The method proposed in this embodiment is fully differentiable; the entire process, from the input point cloud to the final generated mesh, can be optimized end-to-end through backpropagation. This simplifies the training process, improves generation efficiency, and helps the model learn better representations.
[0166] 4. Direct Mesh Output: Many existing technologies generate implicit representations (such as SDF) or point clouds, requiring additional post-processing steps to obtain the final mesh. This embodiment directly generates FlexiCubes mesh representations without post-processing, simplifying the generation process and avoiding errors and artifacts that may be introduced by post-processing.
[0167] 5. Improved controllability: The structured latent space point representation in this embodiment facilitates subsequent shape editing and manipulation. Because each latent space point corresponds to a local region of the mesh, modifying the latent space points allows for fine-grained control over the generated shape.
[0168] In summary, this embodiment achieves high-fidelity, high-efficiency, end-to-end 3D mesh generation with direct output mesh by using a structured latent space point representation method combined with VAE and diffusion model. It outperforms existing technologies in terms of generation quality, computational efficiency, and training methods.
[0169] This embodiment maps an unordered point cloud onto a regular grid. Alternatively, a neural network can be used to learn how to map an unordered point cloud onto a regular grid. For example, an encoder-decoder structure can be used, where the encoder encodes the point cloud into a latent vector, and the decoder predicts the position and offset of each point in the regular grid based on the latent vector.
[0170] FlexiCubes parameterization can also use the following methods:
[0171] Use different kernel functions. Instead of using the RBF kernel function, use other kernel functions to aggregate the feature vectors of structured latent space points. For example, you can use the polynomial kernel function, the Gaussian kernel function, or the thin-plate spline kernel function.
[0172] Different kernel functions can affect the smoothness and locality of FlexiCubes parameters, which in turn affects the quality of the generated mesh.
[0173] Parameterization can also be performed using neural networks. Instead of using kernel-based aggregation methods, a neural network can be used to directly predict the FlexiCubes parameters for each voxel. For example, the locations and feature vectors of structured latent space points can be used as input, and a multilayer perceptron (MLP) can be used to predict the FlexiCubes parameters.
[0174] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.
[0175] The processor may be a central processing unit (CPU) or other processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the three-dimensional mesh generation methods described in the foregoing embodiments of this disclosure.
[0176] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0177] like Figure 2This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0178] like Figure 2 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0179] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 2 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0180] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the three-dimensional mesh generation method of embodiments of this disclosure are performed.
[0181] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0182] The computer-readable storage medium disclosed in this embodiment stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the three-dimensional mesh generation methods described in the foregoing embodiments of this disclosure are performed.
[0183] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0184] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0185] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0186] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0187] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0188] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0189] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0190] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0191] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for generating a three-dimensional mesh, characterized in that, include: Obtain an unordered point cloud; Unordered point clouds are mapped to regular grids to obtain structured point clouds. The transfer cost function for mapping unordered point clouds to regular grids is the squared Euclidean distance. The step of mapping an unordered point cloud to a regular grid to obtain a structured point cloud includes: mapping an unordered point cloud to a regular grid using optimal transmission. The structured point cloud is encoded into the latent space to obtain the first latent space encoding; The second latent space code is obtained by processing the first latent space code using the trained diffusion model. Structured latent space points are obtained based on the second latent space encoding; FlexiCubes mesh is obtained based on structured latent space points; Extracting a 3D mesh from a FlexiCubes mesh; The points in the structured point cloud include assigned grid positions and offsets; The structured point cloud includes position offset and feature vector. The position offset is used to represent the offset of a latent point relative to its grid position, and the feature vector is used to represent the local geometric features of the latent point. FlexiCubes meshes are obtained based on structured latent space points, including: The eigenvectors of latent space points are aggregated using a weighted radial basis function kernel; Calculate the parameters of each voxel in the FlexiCubes mesh based on the feature vectors; The method of aggregating structured latent space point feature vectors using weighted radial basis function kernels includes: Aggregate the k nearest structured latent space points to each FlexiCubes grid; Multiple structured latent space points are voxelized onto a set mesh with a resolution lower than that of the target FlexiCube mesh, and each structured latent space point is associated with a mesh location from the optimal transfer. For each occupying voxel in the set grid, sample its corresponding 8 voxels in the target FlexiCube grid to obtain a voxel set; Take the 26 neighborhoods from the voxel set to obtain the sampling set; FlexiCubes meshes are calculated based on the sample set.
2. The three-dimensional mesh generation method according to claim 1, characterized in that, The parameters stored in each grid cell of the FlexiCubes grid include: Split weights are used to control the division of triangles; Interpolation weights are used to control the position of the dual vertex; And deformation vectors, used to control the spatial position of mesh vertices.
3. The three-dimensional mesh generation method according to claim 1, characterized in that, The step of encoding the structured point cloud into a latent space to obtain the first latent space encoding includes: The structured point cloud is encoded into the latent space using a VAE encoder; The process of using a trained diffusion model to process the first latent space code to obtain the second latent space code includes: The second latent space encoding is obtained by performing T iterations using the trained diffusion model. The process of obtaining structured latent space points based on the second latent space encoding includes: The second latent space is encoded and decoded into structured latent space points using a VAE decoder.
4. The three-dimensional mesh generation method according to claim 3, characterized in that, The VAE encoder includes: 3D convolutional layers use convolutions with strides to downsample, gradually reducing the spatial resolution of the feature maps; Transformer layers are applied to the downsampled feature maps to capture global dependencies; The first fully connected layer is used to output the mean and log-variance of the latent space distribution; Latent space coding is used to sample latent space distributions to obtain latent space codes.
5. The three-dimensional mesh generation method according to claim 3, characterized in that, The VAE decoder includes: The second fully connected layer maps the latent space encoding to the initial feature map; The 3D deconvolution layer uses deconvolution with stride for upsampling, gradually increasing the spatial resolution of the feature map. The output layer is used to predict the position offset and feature vector of each structured latent space point; FlexiCubes parameterization uses structured latent space points to parameterize FlexiCubes meshes; Mesh extraction: Extract the 3D mesh from the parameterized FlexiCubes mesh using isosurfaces. In mesh extraction, sparse convolutional layers are applied to upsample the FlexiCubes mesh.
6. The three-dimensional mesh generation method according to claim 3, characterized in that, The trained diffusion model includes: The input layer uses convolutional layers with strides to downsample the latent space encoding of the input; The intermediate layer contains multiple stacked attention blocks, which are used to incorporate temporal and category-conditional embeddings. The upsampling layer uses a transposed convolutional layer with stride to upsample the feature map.
7. The three-dimensional mesh generation method according to claim 6, characterized in that, The forward pass of training the diffusion model adds Gaussian noise to the latent space encoding within T time steps according to variance scheduling.