Generative sparse latent color field method for three-dimensional raw texture generation

Abstract

The application provides a generative sparse latent color field method for three-dimensional original texture generation, comprising: acquiring geometric information of a target three-dimensional grid to form point cloud data; inputting the point cloud data into a pre-trained texture variational autoencoder to generate geometric conditioned latent features aligned with a texture latent space; based on a geometric conditioning generation model, sampling and generating a latent feature representation of a target texture in a sparse latent space with the geometric conditioned latent features as conditions; inputting the latent feature representation of the target texture into a pre-trained continuous color field decoder to query and decode surface points of the target three-dimensional grid to obtain seamless texture applied to the target three-dimensional grid. The problem of inconsistent geometry / texture coding in the existing projection method is solved.

Description

[0001] Cross-references to related applications This invention patent application claims priority to Chinese Patent Application No. 202511805029.1, filed on December 3, 2025, entitled “Generative Sparse Latent Color Field Method for Generating 3D Native Textures”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention belongs to the field of computer graphics, and in particular relates to a generative sparse latent color field method for generating three-dimensional native textures. Background Technology

[0003] Current texture generation methods mainly inherit from a two-dimensional-centric paradigm and can be divided into two main categories: Multi-view Texture Synthesis: This method leverages advancements in powerful 2D diffusion models by synthesizing multiple 2D views of an object and then projecting them onto a 3D mesh. Early work traded off between slow per-object optimization and progressive generation with the risk of view misalignment. Subsequent research has focused on improving multi-view consistency, for example, by synchronizing denoising processes across views or jointly generating all views. Some work has also focused on generating physically based rendering (PBR) materials to support realistic relighting. However, no matter how sophisticated these methods are, they must all face the inherent pathological problem of harmonizing multiple 2D images into a single coherent 3D surface.

[0004] 2D UV-unwrapped Space Generation: This type of method attempts to generate textures directly in the 2D UV-unwrapped space of an object. While this method avoids occlusion and view inconsistency issues in multi-view projection, its survivability is limited by the quality of UV parameterization. Because the UV unwrapping process often leads to geometric distortion and topological dependencies, the generated textures are prone to distortion and seams when pasted back onto the 3D model.

[0005] To overcome the inherent problems of 2D projection, the 3D Native method has emerged, which is dedicated to directly representing and generating textures within the space of an object itself.

[0006] Early 3D native work included learning on mesh surfaces, using 3D texture fields, or generating spatial textures with point clouds / 3D Gaussian sputtering (3DGS).

[0007] In summary, the inventors found the following problems with the prior art during the implementation of this embodiment: Inherent problems of two-dimensional methods: Projection method: It is difficult to reconcile texture information from different perspectives, especially when there is occlusion and lighting changes, resulting in obvious seams, discontinuities and artifacts on the texture on the mesh.

[0008] UV method: Texture quality and coherence heavily depend on non-unique UV parameterization, and UV unwrapping often introduces severe geometric distortion, causing textures to be distorted on three-dimensional surfaces or to produce seams at UV boundaries.

[0009] Problems with native 3D methods: Existing native methods that utilize sparse voxel structures to encode textures by projecting pre-trained 2D features (such as DINO) suffer from the following three key drawbacks: Viewpoint dependence and inconsistency: The projection process is viewpoint dependent, which can cause inconsistency or ambiguity when there are occlusions or inconsistent view predictions.

[0010] Resolution limitation: Pre-trained 2D features (such as DINO) have limited resolution, which greatly limits the ability to capture high-frequency details of 3D surfaces.

[0011] Feature coupling limitation: Features coupled with two-dimensional fundamental knowledge may not optimally represent the fine properties of three-dimensional surfaces, such as the complex properties required by physically based rendering (PBR) materials.

[0012] Insufficient representational capacity: Overall, the pioneering native work of the prior art, due to its chosen representational methods, struggles to capture high-frequency details within a compact latent space, making it unsuitable for high-fidelity generative modeling. Summary of the Invention

[0013] To address the problems existing in the prior art, this invention provides a generative sparse latent color field method for generating 3D native textures, which at least partially solves the problem of inconsistent geometry / texture encoding in the projection methods of the prior art.

[0014] This disclosure provides a generative sparse latent color field method for generating 3D native textures, including: Obtain the geometric information of the target 3D mesh to form point cloud data; The point cloud data is input into a pre-trained texture variational autoencoder to generate geometric conditional latent features aligned with the texture latent space. Based on the geometrically conditional generation model, the latent feature representation of the target texture is sampled and generated in the sparse latent space, using the latent features of the geometric conditions as conditions. The latent feature representation of the target texture is input into a pre-trained continuous color field decoder to query and decode the surface points of the target 3D mesh, thereby obtaining a seamless texture applied to the target 3D mesh. The continuous color field decoder is a function that fuses sparse latent features with three-dimensional spatial coordinates and maps them to continuous color values ​​through a multilayer perceptron.

[0015] The generative sparse latent color field method for generating 3D native textures provided by this invention extracts geometric latent features based on a texture variational autoencoder, samples and generates latent feature representations of the target texture based on a geometrically conditional generative model, and then decodes them using a continuous color field decoder to obtain texture information. This ensures perfect alignment between geometric signals and texture representations, eliminates the need for auxiliary networks, and thus solves the problem of inconsistent geometry / texture encoding in existing projection methods. Attached Figure Description

[0016] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0017] Figure 1 A flowchart of a generative sparse latent color field method for generating three-dimensional native textures provided in an embodiment of this disclosure. Detailed Implementation

[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0019] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0023] like Figure 1 As shown, this embodiment discloses a generative sparse latent color field method for generating 3D native textures, including: Obtain the geometric information of the target 3D mesh to form point cloud data; The point cloud data is input into a pre-trained texture variational autoencoder to generate geometric conditional latent features aligned with the texture latent space. Based on the geometrically conditional generation model, the latent feature representation of the target texture is sampled and generated in the sparse latent space, using the latent features of the geometric conditions as conditions. The latent feature representation of the target texture is input into a pre-trained continuous color field decoder to query and decode the surface points of the target 3D mesh, thereby obtaining a seamless texture applied to the target 3D mesh. The continuous color field decoder is a function that fuses sparse latent features with three-dimensional spatial coordinates and maps them to continuous color values ​​through a multilayer perceptron.

[0024] In a specific implementation, the geometric conditional generation model uses the texture variational autoencoder to extract geometric conditions; The texture variational autoencoder is trained to encode colored point clouds containing color information into a sparse latent feature grid that follows a Gaussian prior distribution, and / or to encode uncolored point clouds containing only geometric information into geometrically conditional latent features aligned with the encoding of the uncolored point clouds.

[0025] In a specific implementation, the texture variational autoencoder is based on a sparse Transformer architecture to process voxelized 3D point cloud data and captures local details and global coherence of the texture through window self-attention and shift window mechanisms.

[0026] In a specific implementation, the training of the texture variational autoencoder includes: A colored point cloud is used as training data, which includes three-dimensional coordinates, surface normals and color information. The texture variational autoencoder is trained by minimizing the color reconstruction loss and the KL divergence loss of the latent distribution.

[0027] In a specific implementation, the geometric conditional generation model is a conditional modified flow model; The conditional modified flow model learns the velocity field from the prior distribution to the latent feature distribution of the target texture through a generative network. The generative network is based on the diffusion Transformer architecture, processes sparse latent features in a serialized manner, and incorporates geometric conditions and time step information.

[0028] In a specific implementation, the conditional correction flow model includes: The target distribution in the data space is connected to the prior distribution by a linear ordinary differential equation, which includes the velocity field. Based on generative networks Learning speed field; By training the objective function Minimize the generator network With ideal velocity field The differences between the training conditions are used to correct the flow model.

[0029] In a specific implementation, the linear ordinary differential equation is: , , in, It is a velocity field. For a geometric latent feature mesh aligned with the texture latent space, For dynamic texture latent features, It is a time variable; For the target distribution in the data space, For the prior distribution, It is a dataset. It is a uniform distribution.

[0030] In a specific implementation, a generative network is generated. This includes DiT, which consists of multiple layers of multi-head self-attention blocks, with each self-attention block followed by a feedforward network for nonlinear transformation; DiT output sequence Dimensions mapped back to the original latent features Output sequence That is, the network Predicted velocity field This is used to correct the update direction of the stream; Serialization of DiT input data includes sparse latent feature grids. All activated latent feature vectors and their corresponding three-dimensional spatial coordinates Extracted, forming a serialized latent feature sequence. ; DiT encoding and embedding, including feature embedding, positional encoding and conditional embedding; The feature embedding is each latent feature vector. Feature dimensions mapped from linear layers to Transformers ; Position encoding is the corresponding three-dimensional coordinates Encoded as absolute position embedding And add it to the feature embedding; The conditional embedding includes geometric conditions. and time conditions, Geometric conditions Serialized into a geometric latent feature mesh And serve as key and value inputs across attention, or with a sparse latent feature grid. Feature sequence concatenation; The time condition is that the time step is encoded as a high-dimensional vector. High-dimensional vectors It is injected as a global condition into the normalization layer of each Transformer block.

[0031] In a specific implementation, the decoding of surface points by the continuous color field decoder includes: Local latent features are extracted from the sparse latent feature grid surrounding the query point through interpolation. The local latent features are concatenated with the spatial coordinates of the query point to form a composite input vector; The composite input vector is input into a multilayer perceptron, which outputs the color value of the query point.

[0032] In a specific implementation, the geometric conditional generation model also receives latent text features as additional semantic conditions to achieve 3D texture generation based on text description.

[0033] This embodiment discloses a generative sparse latent color field method and system for 3D native texturing, abbreviated as Lafite. This method introduces a novel, highly expressive sparse latent color field (SLCF) representation, combines it with a variational autoencoder (VAE) structure for encoding and decoding, and finally utilizes a conditional rectified flow generative model to generate high-fidelity, seamless, and topology-independent textures on arbitrary 3D geometries.

[0034] This invention comprises three main modules: Sparse Latent Color Field VAE Encoder (SLCF-VAE Encoder): Responsible for encoding high-resolution color 3D point cloud data into a compact and structured sparse latent feature grid z.

[0035] Continuous Color Field VAE Decoder (SLCF-VAE Decoder): Responsible for resolving any query point in three-dimensional space. Latent feature extraction and interpolation are performed, and then the result is decoded into a continuous surface color c.

[0036] The Geometric Conditioned Rectified FlowGenerative Model is responsible for synthesizing high-fidelity textures in the latent space z, using a sparse latent representation of the geometry. As a conditional input.

[0037] This embodiment first defines a novel texture representation: a three-dimensional generative sparse latent color field. The core of this representation is: , in, It is a three-dimensional texture field that can display any point in three-dimensional space. Mapped to its corresponding surface color (generally (RGB color).

[0038] However, modeling directly in high-resolution three-dimensional space Inefficient. This embodiment uses the concept of a latent field and decomposes it into two core steps: Sparse latent feature grid z: A sparse voxel grid is used to store the latent feature vectors z, where and It is the dimension of the potential features. The mesh is activated only near the surface of the 3D object, thus achieving sparsity and high efficiency.

[0039] Continuous decoder A small multilayer perceptron (MLP) is used as the decoder. The local latent features obtained by interpolation from z Mapped to the final color c.

[0040] The final color decoding process can be represented as: , in It is at point The latent features extracted from the sparse grid z These are the coordinates of a point. It is the surface normal (optional, used to enhance details). This decomposition achieves a perfect combination of compact latent representation, topological decoupling, and high-frequency detail capture.

[0041] Geometric Input and Data Representation: To achieve 3D native and high-fidelity encoding, this embodiment abandons traditional 2D images or low-resolution geometry as input, and instead uses a densely sampled color point cloud. ).

[0042] Point cloud data structure: ,in These are point coordinates. It is a high-fidelity surface color (such as from scanning or high-quality rendering). It is the surface normal.

[0043] This representation provides unobstructed, view-independent raw surface properties, which is crucial for capturing the highest frequency texture details and is the core reason why the reconstruction fidelity (>10 dB PSNR) of this embodiment is far superior to that of the prior art.

[0044] Implementation of the Sparse Latent Color Field VAE Encoder (SLCF-VAE Encoder): The function of an encoder is to learn a mapping. The input point cloud data is converted into a sparse latent mesh z and its variational parameters. and .

[0045] VAE encoder network structure: Sparse convolutional network: To efficiently process unstructured high-density colored point clouds Furthermore, this embodiment captures global and local dependencies in high-frequency texture details, employing an advanced Sparse Transformer structure as the encoder. The core computing unit.

[0046] Sparse Transformer Structure Selection: Encoder It adopts a hierarchical design similar to the Swing Transformer, but is customized and optimized for 3D sparse data to achieve efficient feature extraction and multi-scale representation learning.

[0047] Input feature initialization: Input point cloud First, it is voxelized onto a mesh of a predefined resolution (e.g., ).

[0048] Each activated voxel v stores the points it contains. Aggregated input features Input features Including color normal line Information such as relative coordinates.

[0049] Sparse adaptation of Swin's attention mechanism: Window partitioning: Traditional Transformer self-attention computation is computationally complex. This embodiment borrows the Swing attention mechanism and performs self-attention computation within a window. This mechanism reduces the computational complexity from global quadratic to local linear, while enabling deep interaction of local texture details.

[0050] Cross-window connectivity: To capture dependencies over longer distances (global coherence), the encoder alternates between regular window partitioning and shifted window partitioning. This alternation allows information to flow between adjacent windows, ensuring global feature coherence. All attention computations are performed strictly on sparse activation voxels, ensuring high computational efficiency.

[0051] Latent feature generation: After processing multiple non-hierarchical sparse Transformer blocks, the output features are... Maintain the same spatial resolution and sparse structure as the input: , The sparse Transformer structure, especially when combined with Swin-like attention, gives the encoder powerful nonlocal modeling capabilities, enabling it to better understand and compress complex texture patterns, surpassing the limitations of traditional sparse convolution in long-distance dependency modeling.

[0052] Reparameterization Trick and Sparse Latent Mesh z: To enable the variational autoencoder (VAE) to be trained via backpropagation, this embodiment employs a reparameterization technique: 1. From the standard Gaussian distribution Medium sampling noise .

[0053] 2. Through Generate the final sparse latent feature grid z.

[0054] The mesh z is stored only in sparse voxels near the object's surface, and its dimensions are... It is much smaller than the original color channel C, which enables high compression and latent representation of complex textures.

[0055] Implementation of the Continuous Color Field VAE Decoder (SLCF-VAE Decoder): The function of the decoder is to map the potential mesh z back to the continuous surface color c.

[0056] Local extraction and interpolation of latent features: The continuous color field decoder used in this embodiment Achieved from sparse latent mesh To high-fidelity continuous surface color The key to the transformation lies in the extraction of local latent features and the fusion of features with coordinates.

[0057] Local extraction of latent features: For any query point on the grid surface (its coordinates are) The decoder first needs to obtain its corresponding local latent features. .

[0058] Feature and coordinate fusion decoding: Unlike traditional trilinear interpolation, this embodiment uses local latent features with the three-dimensional coordinates of the point Concatenate the vectors to form a composite input vector. Then it is directly input into the continuous color field MLP decoder. middle.

[0059] , , MLP architecture: MLP decoder Include The layers employ ReLU or SiLU activation functions and are designed to learn the complex nonlinear mapping between latent features and spatial locations.

[0060] Importance of coordinates: The design of this embodiment enables the decoder to... It simultaneously relies on the semantic information of latent features and absolute spatial coordinates. This greatly enhances the decoder's ability to capture high-frequency spatial details, namely: local features. It provides macroscopic texture style and color information, while coordinates This ensures that texture details can be accurately located and resolved, thereby enabling surface color queries with infinite resolution (Implicit / Continuous Representation).

[0061] The "sparse feature extraction + coordinate fusion MLP decoding" mechanism is the key to achieving high-fidelity seamless textures in this embodiment, because it utilizes the high efficiency of sparse structures and avoids the smoothing loss of high-frequency information by traditional interpolation methods through the powerful nonlinear capabilities of MLP.

[0062] Continuous Field Decoding MLP: Local latent features obtained by interpolation Input into a small multilayer perceptron (MLP) network In order to predict the final color .

[0063] Input layer: The input vector includes interpolated latent features. .point Positional encoding Position encoding uses high-frequency sine / cosine functions to... Mapping to a higher-dimensional space helps the MLP capture higher-frequency spatial variations, enhancing texture detail. Surface normals (Optional, used to predict normal maps in PBR texture generation).

[0064] Hidden layer: MLP contains The layer uses ReLU or SiLU activation functions.

[0065] Output layer: The final output is a C-dimensional color vector. .

[0066] The continuous field decoder design allows the final resolution of the texture to be no longer limited by the resolution of the voxel mesh, but to achieve surface color queries with infinite resolution (Implicit / Continuous Representation).

[0067] Construction and training of geometrically conditional generative models: Efficient geometrically conditional generative models for synthesizing new textures in the learned sparse latent space z.

[0068] Geometric condition encoder: Encoder multiplexing mechanism: To ensure perfect alignment between geometric information and the texture latent space z, this embodiment employs a multiplexed VAE encoder. The mechanism for extracting geometric conditions: Geometry Input The input is the uncolored point cloud of the target mesh. This means... Location only and normal Information, and color It is set to a neutral color (e.g., white or gray).

[0069] Geometric latent features :Will The input is fed into the same parameter-shared encoder as during VAE training. In the middle. Due to the encoder Its design enables it to extract features from spatial attributes (position, normals), and it can generate a compact, geometrically latent feature mesh aligned with the texture latent space. .

[0070] , The encoder multiplexing method has the following characteristics: Alignment: It guarantees the latent texture feature z and geometric conditions. It is naturally aligned in terms of spatial structure, voxel resolution, and potential dimensions.

[0071] Efficiency: It avoids designing and training a separate, complex geometric coding network.

[0072] The principle of the Conditional Rectified Flow model: The generative model employs Conditional Rectified Flow (C-RF) as its generative paradigm. Compared to traditional diffusion models, the rectified flow has a simpler and more linear latent spatial trajectory.

[0073] Corrected flow definition: A corrected flow distributes the target data in the data space. With a simple prior distribution (e.g., standard Gaussian distribution) Connected together, they are represented by a simple linear ordinary differential equation (ODE): , in It is the velocity field, which represents the velocity field from... arrive The shortest path.

[0074] Network objective: Generate a network The task is to learn the velocity field An ideal corrected flow seeks a straight path, i.e. .

[0075] Training objective function ( The model minimizes the prediction generator network. With ideal velocity field Training based on the differences between them: , in These are linear interpolation points along the training trajectory. It is a dataset. It is a uniform distribution.

[0076] Generative Networks Architecture implementation: Generative Networks The task is to learn the velocity field of the conditionally corrected flow. To efficiently process sparse latent feature meshes. Complex condition generation is performed on it, and this embodiment uses an advanced Diffusion Transformer (DiT) structure as its core architecture.

[0077] DiT is a network architecture that applies the Transformer architecture to noise prediction tasks in diffusion models (or modified flow models). Its advantages lie in its global receptive field and adaptability to variable-length inputs (sparse voxel sequences).

[0078] Input data serialization: Sparse latent feature grid (and geometric conditions) ) is a sparse three-dimensional tensor. To accommodate the structure of the Transformer, this embodiment will... All activated (non-zero) latent feature vectors and their corresponding three-dimensional spatial coordinates Extracted to form a serialized latent feature sequence .

[0079] ,in It refers to the number of activated voxels.

[0080] Transformer encoding and embedding: Feature embedding: Each latent feature vector Feature dimensions mapped from linear layers to Transformers .

[0081] Position encoding: corresponding three-dimensional coordinates (or its voxel index) is encoded as an absolute position embedding. (For example, through Fourier feature maps or learnable embeddings), and added to the feature embedding.

[0082] Conditional embedding: Geometric conditions : Also serialized as And serve as the key and value input for cross-attention, or with The feature sequences are concatenated. This invention preferably employs an adaptive normalization (AdaIN / AdaLN) mechanism to... Global statistics are injected into each layer of the Transformer to ensure that geometric conditions have a deep impact on the generation process.

[0083] Time conditions Time step Encoded as a high-dimensional vector (via sinusoidal positional encoding or MLP), and injected as a global condition into the normalization layer of each Transformer block via AdaLN to guide the network to make predictions at the correct noise level.

[0084] DiT core structure: Self-attention mechanism: DiT consists of multiple layers of multi-head self-attention blocks. This enables the network to compute sequences. The global dependencies between any two potential voxels are crucial for generating globally coherent textures across large geometric features.

[0085] Feedforward Network: Each self-attention block is followed by a feedforward network (FFN) for nonlinear transformation.

[0086] Sparsity handling: DiT only operates on active voxels, thus naturally inheriting the input. Its sparsity avoids the huge computational overhead of three-dimensional dense meshes.

[0087] Output layer: After processing through multiple DiT blocks, the output sequence is... Dimensions mapped back to the original latent features .

[0088] The output sequence That is, the network Predicted velocity field Used to correct the update direction of the stream: .

[0089] This DiT-based generator structure fully leverages the advantages of Transformer in long-distance dependency modeling, enabling this embodiment to generate seamless 3D native textures that are not only rich in local detail but also highly consistent in global style.

[0090] Training details and loss function: The phased training in this embodiment is key: first, the VAE is trained to build high-quality latent representations, and then the generative model is trained to learn the distribution of the latent space.

[0091] The training objective of VAE is to minimize the original input. With reconstruction output The differences between them, and the constraints on the potential distribution.

[0092] (1) Color reconstruction loss : Ensure color reconstruction It approximates the true color c at the pixel level. L1 loss is typically used (more robust to color).

[0093] , (2) KL divergence loss The is the core of VAE, used to force sparse latent features z to approximate a prior distribution (usually a standard Gaussian distribution). This helps smooth the latent space, making generative models easier to learn.

[0094] , Total VAE Loss: The final VAE training loss is the weighted sum of the individual losses. , in It is a weighting parameter that balances the various losses.

[0095] Generative model training loss function: The generative model is trained using only the modified flow loss described above. During training, the input network... of From VAE encoder The encoded texture latent features extracted from it.

[0096] Texture generation process (Inference): The advantage of this embodiment is that after the VAE and C-RF models are trained, the process of generating new textures (inference) is efficient and direct. The specific process includes: Step 1: Geometry Input and Conditional Encoding: Input: The target 3D mesh from which the texture is to be generated ; Processing: From Extracting geometric information (location) and normal ), forming uncolored point clouds ; Conditional extraction: Input to the trained VAE encoder In the process of obtaining geometrically conditional latent features .

[0097] Step 2: Latent Feature Sampling and Generation Prior sampling: from a predefined simple prior distribution An initial sparse latent feature grid is sampled from the sample; Texture generation: Geometric conditions and time Input into the trained conditional modified stream generator network middle; Efficient Sampling: The advantage of the modified flow model lies in the linearity of its paths. During the inference phase, fewer sampling steps can be used, and this fewer-step generation process significantly improves the generation speed compared to traditional diffusion models.

[0098] Step 3: Continuous Color Decoding and Application: Decoding: Generating latent texture features Input to VAE continuous color field decoder ; Surface query: in All surface points At this point, trilinear interpolation is used to extract... Then by Real-time decoding of the final color ; Output: The final result is directly applied to the target mesh. High-fidelity, seamless 3D native textures.

[0099] In this embodiment, the sparse voxel structure in the latent feature storage based on the sparse voxel grid z can be replaced with other efficient 3D sparse data structures, such as octrees or hash-mapped grids (as used in Instant-NGP). These structures can also achieve the goal of sparsity by storing features only near the surface. In this case, trilinear interpolation will be replaced by corresponding octree interpolation or hash lookup interpolation.

[0100] In the combination of sparse latent fields and explicit / implicit fields, the z-latent features can be replaced with latent parameters of neural radiation fields (NeRF) or three-dimensional Gaussian sputtering (3DGS) stored on sparse meshes. For example, sparse voxels can store latent codes for parameters such as color, covariance, and opacity of 3DGS, which are then converted into the final 3DGS parameters by the decoder D before rendering.

[0101] In the VAE architecture, which employs a sparse convolutional network E and an MLP decoder D, the encoder sparse convolutional network E can be replaced by other deep learning architectures for processing 3D sparse data, such as the sparse Transformer or point cloud-based point networks (PointNet / PointCNN). These networks can also achieve the mapping from high-density point clouds to compact latent features z.

[0102] The continuous-field MLP decoder D can be replaced by implicit functions with different architectures, such as Siren networks (using sinusoidal activation functions to enhance high-frequency detail capture) or large MLPs. Alternatively, smaller-scale convolutional networks can be used to replace MLPs for local... Perform more complex feature extraction, and then map it to... .

[0103] The Conditional Rectified Flow (C-RF) model can be replaced by the Conditional Denoising Diffusion Probabilistic Models (cDDPM). This alternative can also generate latent features, but at the cost of a significant reduction in inference speed (requiring hundreds of sampling steps).

[0104] A Conditional Generative Adversarial Network (cGAN) is trained in the latent space z to utilize geometric latent features. As a condition, GAN models can typically achieve faster single-step generation, but they are difficult to train and may be subject to the risk of mode collapse.

[0105] Shared VAE encoder Perform geometric condition coding It is possible to design and train a standalone sparse convolutional network. To encode geometric information While this may lose the natural alignment advantage with the texture latent space z, it can be forced by an additional alignment loss. Output and The potential space is close.

[0106] It is possible to avoid using sparse meshes Instead of using it as a condition, the surface attributes of the target geometry, such as curvature and local shape indices, are directly embedded into the input of the generative network and added as feature channels to the z-axis to be generated.

[0107] This embodiment has the following objectives: 1. We propose and learn a generative sparse latent color field as a latent representation of 3D textures. This representation is completely decoupled from mesh topology and UV coordinates, thereby fundamentally avoiding the seams, distortions and topological constraints that occur in projection and UV-dependent methods.

[0108] 2. Design a variational autoencoder (VAE) to directly and unbiasedly extract data from densely sampled color point clouds (containing high-fidelity original colors). ,point and normal The texture is encoded in the input, which provides native, unobstructed and viewpoint-independent surface information, thereby enabling effective capture of high-frequency details and achieving unprecedented reconstruction fidelity (e.g., a >10 dB PSNR improvement over existing technologies).

[0109] 3. A unified and efficient geometry conditioning method is introduced, which extracts a pure geometric latent feature by reusing the texture encoder to encode the uncolored mesh (i.e., the "white" mesh). This design provides a complete, unoccluded 3D shape prior, ensuring perfect alignment between the geometric signal and the texture representation, eliminating the need for auxiliary networks, and thus solving the problem of inconsistency between geometry / texture encoding in existing projection methods.

[0110] 4. Based on this superior representation and conditionalization mechanism, novel textures are synthesized in the sparse latent space using the Conditional RectifiedFlow model, achieving the highest level of quality and coherence to date, thus laying the foundation for the next generation of 3D content creation workflows.

[0111] This embodiment combines high-density color point cloud input, sparse convolution, and continuous color field decoding. This invention effectively encodes high-frequency texture details into a compact latent space and reconstructs them losslessly using a continuous field mechanism. The sparse structure is activated only near the surface, ensuring computational and memory efficiency, enabling it to handle large-scale 3D models.

[0112] The conditional modified flow model endows the system with powerful generative capabilities. By simply modifying the input text description or geometric conditions, texture style transfer or texture editing can be achieved while preserving geometry. Meanwhile, From The exported geometry and texture generation process are perfectly aligned, ensuring the stability and controllability of the generated results.

[0113] In summary, the technical solution of this embodiment is an end-to-end 3D native texture generation system with high fidelity, high efficiency and high controllability, which fundamentally solves the bottleneck problem of existing technologies in the field of 3D texture generation.

[0114] In this embodiment, the high-fidelity texture of the three-dimensional mesh surface is encoded as a sparse voxel mesh z, where z stores the latent feature vector only in the active voxels near the object surface. This representation is completely decoupled from the topology and two-dimensional UV parameters of the underlying three-dimensional mesh, thereby ensuring the seamlessness and topology independence of the generated texture.

[0115] By combining a sparse latent grid z with a small multilayer perceptron (MLP) continuous field decoder By combining these methods and utilizing trilinear interpolation techniques, we can achieve query points on any three-dimensional surface. Potential features Continuous, differentiable extraction and decoding into color. This allows the final texture resolution to no longer be limited by the voxel mesh, enabling texture queries at infinite resolutions.

[0116] High-density color point cloud (Including location) ,color normal line The input to the variational autoencoder (VAE) is passed through a sparse convolutional network. Encoding unobstructed, view-independent raw surface properties directly from 3D space effectively captures high-frequency texture details, resulting in a significant improvement in reconstruction fidelity (PSNR>10 dB).

[0117] For the first time, a VAE architecture based on sparse convolution is proposed and trained, which enables it to efficiently process and compress high-dimensional texture data in 3D space and enforce Gaussian prior constraints (KL loss) in the latent space to ensure the generativeness and smoothness of the latent space.

[0118] Multiplexed VAE encoders are used The method is to color the uncolored point cloud of the target geometry. Input to In this process, geometrically conditional latent features that are perfectly aligned with the texture latent space are generated. This mechanism ensures geometric consistency and alignment during the texture generation process.

[0119] Trajectories are learned and generated in the z-latent space using a conditionally modified flow model, and... As a conditional input, the modified flow model enables sampling in one or very few steps, significantly improving the speed and efficiency of new texture generation.

[0120] Generative models support conditions other than geometry. In addition, latent text features can be extracted through text encoders (such as CLIP encoders). As an additional semantic condition, the generation control from text to 3D texture is realized, which greatly enhances the flexibility and practicality of the system.

[0121] A complete system and methodology comprising three core modules—sparse VAE encoding, continuous field decoding, and conditional modified stream generation—is used to achieve end-to-end generation from target geometry to high-fidelity, seamless 3D native textures.

[0122] Based on the structured properties of sparse latent features z, texture editing and interpolation are performed in the latent space, and then the edited 3D native texture is rendered by the decoder.

[0123] The solution in this embodiment has the following advantages: 1. By using high-density color point clouds as input and combining them with a continuous field decoder, this method successfully addresses the problem of insufficient high-frequency detail capture in existing native 3D methods. Experiments demonstrate that in reconstruction tasks, this method achieves a PSNR more than 10 dB higher than state-of-the-art sparse representation methods, marking a significant leap in reconstruction fidelity. This improved fidelity directly translates into visual realism and detail in the generated textures.

[0124] 2. It completely eliminates the dependence on UV coordinates or multi-view projection, directly encoding texture information in a 3D sparse latent space and decoding it through continuous fields. This fundamentally eliminates artifacts such as texture seams, geometric distortions, and view inconsistencies, ensuring that textures generated on any complex geometry are spatially coherent and seamlessly connected.

[0125] 3. A conditional modified flow model is used as the generator, and its inherent near-linear trajectory characteristics are utilized to achieve one-step (or very few-step) sampling generation of new textures. This makes the inference speed of this embodiment hundreds of times faster than the traditional multi-step diffusion model, greatly improving the efficiency of 3D content generation.

[0126] 4. The innovative shared encoder geometry conditioning mechanism ensures that the generated texture latent features z are consistent with the target geometry. Perfect spatial alignment. This strong alignment makes the generation process more stable and controllable, and can accurately adapt textures to geometric details, avoiding the problem of textures "floating" or misaligning on the geometry.

[0127] 5. The sparse latent color field z is a compact and structured representation that is not only applicable to RGB colors, but also easily extended to the representation and generation of physically based rendering (PBR) material properties (such as albedo, roughness, normal maps, etc.), greatly expanding the application potential of this embodiment in the field of professional graphics rendering.

[0128] 6. The sparse voxel mesh structure design ensures that information is stored only in voxels near the object's surface. Compared to fully dense voxel or fully implicit field representations, this embodiment significantly reduces memory usage and computational resources when processing high-resolution 3D models.

[0129] In summary, this embodiment provides a state-of-the-art solution for generating native 3D textures. Its positive effects include significantly lowering the barrier to entry for creating high-fidelity 3D assets and accelerating the AI-driven 3D content generation process. Users can quickly and seamlessly generate realistic textures for complex geometries (such as scanned models and hand-drawn models), thereby generating significant economic value and technological advancements in fields such as game development, VR / AR production, and e-commerce model rendering.

[0130] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0131] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A generative sparse latent color field method for generating 3D native textures, characterized in that, include: Obtain the geometric information of the target 3D mesh to form point cloud data; The point cloud data is input into a pre-trained texture variational autoencoder to generate geometric conditional latent features aligned with the texture latent space. Based on the geometrically conditional generation model, the latent feature representation of the target texture is sampled and generated in the sparse latent space, using the latent features of the geometric conditions as conditions. The latent feature representation of the target texture is input into a pre-trained continuous color field decoder to query and decode the surface points of the target 3D mesh, thereby obtaining a seamless texture applied to the target 3D mesh. The continuous color field decoder is a function that fuses sparse latent features with three-dimensional spatial coordinates and maps them to continuous color values ​​through a multilayer perceptron. The geometrically conditional generation model is a conditionally modified flow model; The conditional modified flow model includes: The target distribution in the data space is connected to the prior distribution through a linear ordinary differential equation; The linear ordinary differential equation is: ， ， in, It is a velocity field. For a geometric latent feature mesh aligned with the texture latent space, For dynamic texture latent features, It is a time variable; For the target distribution in the data space, As a prior distribution, It is a dataset. It is a uniform distribution. It is the objective function. It is a generative network; Point cloud data structure is ,in These are point coordinates. It is a high-fidelity surface color. It is the surface normal.

2. The generative sparse latent color field method for generating 3D native textures according to claim 1, characterized in that, The geometric conditional generation model uses the texture variational autoencoder to extract geometric conditions; The texture variational autoencoder is trained to encode colored point clouds containing color information into a sparse latent feature grid that follows a Gaussian prior distribution, and / or to encode uncolored point clouds containing only geometric information into geometrically conditional latent features aligned with the encoding of the uncolored point clouds.

3. The generative sparse latent color field method for generating 3D native textures according to claim 2, characterized in that, The texture variational autoencoder is based on a sparse Transformer architecture and is used to process voxelized 3D point cloud data. It captures local details and global coherence of textures through window self-attention and shift window mechanisms.

4. The generative sparse latent color field method for generating 3D native textures according to claim 2, characterized in that, The training of the texture variational autoencoder includes: A colored point cloud is used as training data, which includes three-dimensional coordinates, surface normals and color information. The texture variational autoencoder is trained by minimizing the color reconstruction loss and the KL divergence loss of the latent distribution.

5. The generative sparse latent color field method for generating 3D native textures according to claim 1, characterized in that, The conditional modified flow model learns the velocity field from the prior distribution to the latent feature distribution of the target texture through a generative network. The generative network is based on the diffusion Transformer architecture, processes sparse latent features in a serialized manner, and incorporates geometric conditions and time step information.

6. The generative sparse latent color field method for generating 3D native textures according to claim 5, characterized in that, The linear ordinary differential equation includes the velocity field; Based on generative networks Learning speed field; By training the objective function Minimize the generator network With ideal velocity field The differences between the training conditions are used to correct the flow model.

7. The generative sparse latent color field method for generating 3D native textures according to claim 1, characterized in that, Generative Networks This includes DiT, which consists of multiple layers of multi-head self-attention blocks, with each self-attention block followed by a feedforward network for nonlinear transformation; DiT output sequence Dimensions mapped back to the original latent features Output sequence That is, a generative network Predicted velocity field This is used to correct the update direction of the stream; Serialization of DiT input data includes sparse latent feature grids. All activated latent feature vectors and their corresponding three-dimensional spatial coordinates Extracted, forming a serialized latent feature sequence. ; DiT encoding and embedding This includes feature embedding, positional encoding, and conditional embedding; The feature embedding is each latent feature vector. Feature dimensions mapped from linear layers to Transformers ; Position encoding is the corresponding three-dimensional coordinates Encoded as absolute position embedding And add it to the feature embedding; The conditional embedding includes geometric conditions and temporal conditions. Geometric conditions are serialized into a geometric latent feature mesh. And serve as key and value inputs across attention, or with a sparse latent feature grid. Feature sequence concatenation; The time condition is that the time step is encoded as a high-dimensional vector. High-dimensional vectors It is injected as a global condition into the normalization layer of each Transformer block.

8. The generative sparse latent color field method for generating 3D native textures according to claim 1, characterized in that, The continuous color field decoder decodes surface points, including: Local latent features are extracted from the sparse latent feature grid surrounding the query point through interpolation. The local latent features are concatenated with the spatial coordinates of the query point to form a composite input vector; The composite input vector is input into a multilayer perceptron, which outputs the color value of the query point.

9. The generative sparse latent color field method for generating 3D native textures according to claim 1, characterized in that, The geometrically conditional generation model also receives latent text features as additional semantic conditions to achieve 3D texture generation based on text description.

Images