Foreground-aware multi-view diffusion method for three-dimensional texture generation

Abstract

The application provides a foreground-aware multi-view diffusion method for three-dimensional texture generation, comprising: acquiring a reference image and a three-dimensional mesh to be textured; rendering a geometric condition map of multiple views from the three-dimensional mesh; encoding into latent variables; obtaining target noise latent variables based on the latent variables; obtaining reference image latent variable tokens, geometric condition latent variable tokens and target noise latent variable tokens; splicing into a unified sequence and inputting into a backbone network of a multi-view diffusion model for joint modeling; according to a foreground mask, deleting background tokens and retaining only foreground tokens before the unified sequence enters the backbone network of the multi-view diffusion model; performing multi-view diffusion generation on the unified sequence with retained foreground tokens to obtain denoised foreground latent variables; decoding the denoised foreground latent variables into multi-view texture images; and back-projecting the multi-view texture images to the surface of the three-dimensional mesh to obtain a textured three-dimensional asset. The purpose of reducing redundant calculation is achieved.

Description

Technical Field

[0001] This invention belongs to the field of 3D modeling technology, and in particular relates to a foreground-aware multi-view diffusion method for 3D texture generation. Background Technology

[0002] Currently, texture generation methods based on multi-view diffusion models have become an important technical approach for 3D texture generation. Examples include multi-view diffusion 3D texture generation methods based on DiT or Multi-Modal Diffusion Transformer (MM-DiT).

[0003] Existing multi-view diffusion methods have achieved certain results at low resolutions, but they typically operate at 512 or 768 resolutions, making it difficult to preserve fine-grained texture information in high-resolution reference images. If the resolution is directly increased to 2048×2048, the corresponding token sequence grows dramatically. Taking six-view generation as an example, the uniform sequence length can exceed 200,000 tokens, leading to excessively high memory and computational overhead for the attention layer, feedforward layer, and other per-token operations in the DiT backbone.

[0004] Furthermore, a large number of background regions exist in multi-view rendering of the object's center. These regions do not participate in the final 3D texture generation, but existing complete sequence modeling methods usually still encode them as tokens and input them into the network backbone for processing, resulting in significant redundant computation. Although existing methods have attempted to reduce the computational load in the attention layer, the complete token sequence still runs through the DiT backbone, making it difficult to fundamentally solve the problem of high-resolution long sequence modeling.

[0005] Multi-view diffusion 3D texture generation methods based on DiT or Multi-Modal Diffusion Transformer (MM-DiT) encode the entire multi-view canvas into tokens and input all tokens into the DiT backbone for processing, without distinguishing between the foreground regions that actually participate in texture generation and the background regions that do not. Therefore, at a resolution of 2048×2048, the background tokens and the excessively long unified sequence lead to significant redundant computation, making them unsuitable for applications such as dynamic modeling.

[0006] In the process of implementing this invention, the inventors discovered the following problems with the prior art: (1) It is difficult to balance high-resolution detail fidelity with computational feasibility: Existing multi-view diffusion texture generation methods typically operate at lower resolutions such as 512 or 768, making it difficult to fully preserve the fine-grained texture information in high-resolution reference images. Directly scaling these methods to 2048×2048 resolution would drastically increase the length of the corresponding token sequence. For example, in a six-view generation task, the uniform input sequence length could exceed 200,000 tokens, leading to excessive memory usage and computational latency in the attention layer, feedforward layer, and other per-token computation modules of the DiT backbone, making training and inference practically impossible.

[0007] (2) Complete multi-view canvas modeling has obvious redundant calculations: In object-centric multi-view rendering layouts, the foreground regions of the object are the ones that truly participate in 3D texture generation. A large number of background regions do not correspond to the 3D object surface and do not participate in the final texture formation. However, existing complete sequence modeling methods typically still encode background regions as tokens and input them into the DiT backbone along with foreground tokens, resulting in significant sequence-level redundancy. Furthermore, the effective interactions between foreground tokens are not entirely dense; continuing to perform fully connected attention computations on all foreground token pairs will further introduce unnecessary attention computation overhead.

[0008] (3) The existing decoding process is difficult to adapt to the requirements of high-resolution foreground sparse inference: Existing multi-view diffusion texture generation methods typically rely on standard VAE decoders to reconstruct multi-view images from the generated latent. This type of decoding process assumes the input latent is distributed normally after denoising. However, when reducing high-resolution inference costs by updating only the foreground latent corresponding to the object surface and minimizing background region computation, the background latent may still be noisy or have a non-standard distribution. Existing standard VAE decoders lack the ability to adapt to this mixed latent input of "valid foreground and anomalous background," easily causing foreground artifacts, color shifts, or detail loss, thus affecting the quality of the final multi-view texture image. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a foreground-aware multi-view diffusion method for 3D texture generation.

[0010] This disclosure provides a foreground-aware multi-view diffusion method for 3D texture generation, including: Obtain the reference image and the 3D mesh to be textured; Render geometric condition maps from multiple perspectives from the 3D mesh; The reference image is encoded as a reference image latent variable, and the geometric condition map is encoded as a geometric condition latent variable; The target noise latent variable is obtained based on the latent variables of the image and the latent variables of the geometric condition. Reference image latent variable tokens, geometric condition latent variables, and target noise latent variables are obtained based on reference image latent variables, geometric condition latent variables, and target noise latent variables, respectively. The reference image latent variable token, the geometric condition latent variable token, and the target noise latent variable token are concatenated into a unified sequence and then input into the backbone network of the multi-view diffusion model for joint modeling. Based on the foreground mask, background tokens are removed before the unified sequence enters the backbone network of the multi-view diffusion model, and only foreground tokens are retained. Multi-view diffusion generation is performed on a unified sequence that preserves foreground tokens to obtain denoised foreground latent variables; The denoised foreground latent variables are decoded into a multi-view texture image; The multi-view texture image is back-projected onto the surface of the three-dimensional mesh to obtain a textured three-dimensional asset.

[0011] Optionally, the multi-view diffusion generation uses Flow Matching as the training objective, constructing noisy target latent variables along a linear path. , And the model predicts the velocity field to approximate the target velocity. For training objectives; in, For the target multi-view texture image latent variables, For time steps, It is Gaussian noise. The target speed.

[0012] Optionally, the geometric condition map includes a normal map, a depth map, a position map, and / or a contour map; The foreground mask is constructed based on the alpha channel, depth map, or semantic segmentation results of multi-view rendering.

[0013] Optionally, after deleting the background token, the retained foreground token maintains the spatial position index of the foreground token in the original multi-view canvas.

[0014] Optionally, the multi-view diffusion generation is trained using a foreground region-constrained Flow Matching loss, including: During the training phase, target noise latent variables are first constructed on the complete latent variable grid, and then the foreground token sequence is extracted based on the foreground token position set. The model predicts the speed only for the target tokens corresponding to the foreground region, and the loss function is calculated only on the foreground tokens, as follows: , in, The flow matching loss represents the foreground region limitation. This represents the velocity field predicted by the model for the foreground token. Indicates the location set based on the foreground tokens The operation to retrieve the corresponding token. Indicates Gaussian noise. Represents the latent variables of the target multi-view texture image.

[0015] Optionally, the step of performing multi-view diffusion generation on a unified sequence that preserves foreground tokens yields denoised foreground latent variables. This includes employing a block sparse attention mechanism, which includes: Divide the query Q, key K, and value V into multiple blocks; Calculate the block-level correlation between the query block and the key block; Select the query block and key block based on the block-level relevance; Token-level attention computation is performed only between the selected query block and key block pair.

[0016] Optionally, the multi-view diffusion generation includes selective denoising based on the foreground region, that is, performing denoising updates only on the latent variables corresponding to the foreground token positions; The update process is represented as follows: , in, This represents the potential latent variables at the current time step. This represents the updated prospective latent variable. This indicates the speed at which the model predicts the foreground token. Represents the set of foreground token positions. Indicates the updated time step. Indicates the current time step.

[0017] Optionally, decoding the denoised foreground latent variables into a multi-view texture image includes: using foreground-aware VAE decoding. The foreground-aware VAE decoding includes normalized background latent variable replacement, which includes: After obtaining the denoised foreground latent variable, the noise latent variable of the background region is replaced with the normalized background latent variable, and combined with the denoised foreground latent variable to obtain the mixed latent variable; The canonical background latent variable is obtained by encoding the canonical background image using a frozen foreground-aware VAE encoder. The frozen foreground-aware VAE encoder keeps the encoder frozen and only updates the VAE decoder.

[0018] Optionally, the foreground-aware VAE decoding includes foreground region-constrained VAE decoder adaptation, which includes: The foreground-aware VAE decoder is adapted and trained. During the training process, the foreground-aware VAE encoder is kept frozen and only the foreground-aware VAE decoder is updated. The training loss is calculated only in the pixel-level foreground region, and is expressed as: , in, This represents the decoding loss of a foreground-aware VAE. Represents the pixel-level foreground area. This represents a foreground-aware VAE decoder. Represents a real target image. Indicates the position index.

[0019] Optionally, back-projecting the multi-view texture image onto the three-dimensional mesh surface includes: Based on the camera parameters used during rendering, the visibility of the 3D mesh surface, and the view fusion rules, the generated multi-view texture image is back-projected onto the 3D mesh surface; For the same surface region observed simultaneously from multiple viewpoints, texture information from different viewpoints is fused based on one or more of the following: viewpoint visibility, normal angle, viewpoint distance, and confidence level, to obtain a texture result aligned with the three-dimensional geometry.

[0020] The foreground-aware multi-view diffusion method for 3D texture generation provided by this invention removes background tokens and uses a foreground mask to remove background tokens that do not participate in texture generation before the tokens enter the diffusion backbone. This shortens the model input length at the sequence level, allowing computational resources to be concentrated on the foreground region that truly corresponds to the 3D surface texture, thereby reducing redundant computation. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0022] Figure 1 A flowchart of a foreground-aware multi-view diffusion method for 3D texture generation provided in this disclosure embodiment; Figure 2 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0024] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] It should be noted that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice method. Furthermore, this device and / or practice method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0026] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0027] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0028] Flow matching refers to the matching of streams. VAE stands for Variational Autoencoder; 3D assets refer to digital objects that include 3D mesh geometry and can be texture-mapped within the graphics pipeline. Examples include game character models, product models used for e-commerce displays, virtual items, and virtual humans.

[0029] This embodiment relates to automatic texture generation technology in 3D content generation. Texture is a key factor determining the visual realism and detail of 3D assets, and high-quality and efficient automatic texture generation is an important task in 3D content generation.

[0030] This embodiment aims to address the core technical bottleneck in 3D texture generation based on the multi-view diffusion model, which struggles to balance high-fidelity detail generation with efficient training and inference at a high resolution of 2048×2048. Existing multi-view texture generation methods face severe computational redundancy, memory consumption, and inference latency issues when scaled up to 2K resolution, making them unsuitable for direct application to high-resolution 3D texture generation tasks.

[0031] Specifically, the technical problem solved by this embodiment is as follows: (1) Solve the problem that existing multi-view diffusion texture generation methods have difficulty preserving fine-grained texture details at high resolution: Existing multi-view diffusion texture generation methods typically operate at lower resolutions such as 512 or 768, making it difficult to fully preserve high-frequency details in high-resolution reference images. Directly increasing the resolution to 2048×2048 results in a dramatic increase in the corresponding token sequence; for example, in six-view generation, the uniform sequence length can exceed 200,000 tokens, leading to excessive memory usage and computational latency for the Diffusion Transformer (DiT) backbone. This embodiment aims to address the challenge of balancing texture detail fidelity with computational feasibility at 2K resolution.

[0032] (2) Solving the problem of redundant calculations in multi-view rendering of object center: In object-centric multi-view rendering layouts, the foreground region of the object is the only part that truly participates in 3D texture generation, while a large number of background regions do not participate in the final texture formation. Existing methods typically still encode background regions as tokens and input them into the network backbone, resulting in significant sequence-level redundancy. Furthermore, the effective interactions between foreground tokens are not entirely dense, and continuing to use fully connected attention computation incurs additional overhead. This embodiment aims to address how to remove background tokens and further reduce redundant attention computation on the foreground token sequence.

[0033] (3) Solve the problem of decoding quality degradation caused by selective denoising inference based on foreground regions: To improve inference efficiency, the model can denoise and update only the latent tokens corresponding to the foreground region, while leaving the latent variables in the background region in their initial Gaussian noise state. However, directly inputting this mixed latent variable into the Variational Autoencoder (VAE) decoder can easily lead to artifacts, color shifts, or loss of detail in the foreground region. This embodiment needs to address the issue of high-quality decoding after selective denoising inference.

[0034] This embodiment aims to fundamentally reduce redundant computation in 2048×2048 multi-view diffusion texture generation by introducing a foreground-aware high-resolution multi-view diffusion texture generation framework and combining it with mechanisms such as background token removal, block-sparse attention, and foreground-aware variational autoencoder decoding. This achieves efficient and geometrically aligned 3D texture generation while maintaining the ability to generate high-resolution details.

[0035] The foreground-aware multi-view diffusion method for high-resolution 3D texture generation disclosed in this embodiment achieves the following specific objectives: (1) Construct a multi-view diffusion texture generation framework that supports 2K resolution, so that the model can retain the high-frequency details and fine appearance of the reference image while avoiding the infeasibility of training and inference due to the complete long sequence modeling, thereby providing high-resolution, multi-view texture observation for high-quality 3D texture generation.

[0036] (2) Constructing an efficient computing mechanism for foreground perception: A background token deletion mechanism is introduced. Based on the foreground mask in multi-view rendering at the object center, background tokens that do not participate in texture generation are deleted before the tokens enter the DiT backbone. This shortens the model input length at the sequence level, allowing the computation to focus on the foreground region that truly corresponds to the 3D surface texture.

[0037] Block-Sparse Attention is employed on the preserved foreground token sequence. By sparsening the attention computation between foreground tokens, unnecessary fully connected attention interactions are reduced, thereby further reducing the memory usage and computation latency of high-resolution multi-view diffusion generation.

[0038] (3) Constructing a foreground-aware VAE decoding mechanism: To address the problem of mixed latent inputs caused by selective denoising inference based on foreground regions, Foreground-Aware VAE Decoding is adopted. This replaces noisy background latents with normalized background latents and adapts the VAE decoder through foreground region constraints, thereby reducing decoding artifacts and color shifts and ensuring the reconstruction quality of the final high-resolution multi-view texture image.

[0039] This embodiment aims to achieve multi-view 3D texture generation with a high resolution of 2048×2048, which combines detail fidelity, computational efficiency, and decoding quality.

[0040] This embodiment proposes a foreground-aware multi-view diffusion method for high-resolution 3D texture generation. The method takes a reference image and a 3D mesh to be textured as input. First, it renders a multi-view geometric condition map from the 3D mesh. Then, it uses a multi-view diffusion model to generate a 2048×2048 high-resolution multi-view texture image aligned with the geometric conditions. Finally, it back-projects the generated multi-view texture image onto the 3D mesh surface to obtain the final textured 3D asset.

[0041] Unlike approaches that directly input the complete multi-view canvas into the diffusion Transformer backbone, this embodiment leverages the difference between the foreground and background regions in multi-view rendering at the object center. Before the tokens enter the diffusion backbone, background tokens are removed based on the foreground mask, and block sparse attention is further applied to the retained foreground token sequence to reduce redundant computation in high-resolution multi-view diffusion generation. Simultaneously, to address the issue that selective denoising inference based on the foreground region may cause background latent variables to remain in a noisy state, this embodiment further employs a foreground-aware VAE decoding mechanism to ensure the decoding quality of the final multi-view texture image.

[0042] The overall technical process of this embodiment includes: multi-view geometric conditional rendering, latent variable token encoding, background token deletion based on foreground mask, block sparse attention computation on foreground token sequence, selective denoising inference based on foreground region and foreground-aware VAE decoding, and multi-view texture backprojection.

[0043] like Figure 1 As shown, this embodiment discloses a foreground-aware multi-view diffusion method for 3D texture generation, including: Step S101: Obtain the reference image and the 3D mesh to be textured; Input includes reference image and the 3D mesh to be textured Reference images are used to provide information on the appearance, color, and texture style of the target 3D asset; 3D meshes are used to provide the geometry of the texture to be generated.

[0044] Step S102: Render a geometric condition map from multiple perspectives from the three-dimensional mesh; First, multiple viewpoint geometry maps are rendered from the 3D mesh M. In a specific application scenario, six viewpoint normal maps are rendered as geometry maps to represent the surface orientation and local geometry of the 3D object from different viewpoints. This geometry map is denoted as... The target multi-view texture image is denoted as... .

[0045] Step S103: Encode the reference image as a reference image latent variable, and encode the geometric condition map as a geometric condition latent variable; The reference image, geometry condition map, and target texture image are encoded as latent variable tokens; This embodiment uses a variational autoencoder (VAE) to encode the input image into a latent variable space. (Reference image) Geometric condition diagram and target multi-view texture image They are respectively encoded into corresponding latent variable representations. , and .

[0046] , in, Indicates VAE encoder, Represents the latent variables of the reference image. Represents geometrically conditional latent variables. This represents the latent variables of the target multi-view texture image. During the training phase, The model is a real multi-view texture image. During the inference phase, the multi-view diffusion model generates the latent variables corresponding to the target multi-view texture image based on the reference image and geometric conditions, starting from the noise latent variables.

[0047] Step S104: Obtain the target noise latent variable based on the image latent variable and the geometric condition latent variable; This embodiment can use Flow Matching as the diffusion generation training objective. For the latent variables of the target multi-view texture image... Given a time step t and Gaussian noise ε, construct the target latent variable of the noise at time step t. .

[0048] , Under this linear path, the target velocity can be expressed as the direction from the target latent variable to the noise latent variable. , The multi-view diffusion model uses latent variables of the noise target, reference image, geometric condition, and time step. As input, predict the corresponding velocity field. The training objective is to make the velocity field predicted by the multi-view diffusion model approximate the target velocity, thereby learning to generate target multi-view texture latent variables under the constraints of the reference image and geometric conditions.

[0049] Step S105: Obtain reference image latent variable tokens, geometric condition latent variable tokens, and target noise latent variable tokens based on reference image latent variables, geometric condition latent variables, and target noise latent variables, respectively; In this embodiment, the latent variable refers to the continuous feature tensor output by the VAE encoder; the latent variable token refers to the feature vector at each position obtained by flattening the tensor along the spatial dimension. Therefore, by splitting the reference image latent variable, geometric condition latent variable, and target noise latent variable into vectors according to their spatial positions, the corresponding reference image latent variable token, geometric condition latent variable token, and target noise latent variable token are obtained.

[0050] Step S106: The reference image latent variable token, geometric condition latent variable token, and target noise latent variable token are concatenated into a unified sequence and then input into the backbone network of the multi-view diffusion model for joint modeling; This embodiment uses the Diffusion Transformer as an example to illustrate the multi-view diffusion model. In the basic multi-view diffusion framework, the reference image latent variable tokens, geometric condition latent variable tokens, and target noise latent variable tokens are concatenated into a unified sequence and input into the Diffusion Transformer backbone for joint modeling. This method can simultaneously utilize the appearance information of the reference image and the multi-view geometric condition information to generate a multi-view texture image aligned with the input 3D geometry.

[0051] However, at a resolution of 2048×2048, the number of tokens corresponding to the reference image, geometric condition map, and six-view target texture image is extremely large. If the entire multi-view canvas is still encoded as tokens and input into the backbone network, it will result in excessive memory usage and computational latency. Therefore, this embodiment further introduces a foreground-aware token reduction mechanism on top of the basic multi-view diffusion framework.

[0052] Step S107: Based on the foreground mask, remove the background tokens and retain only the foreground tokens before the unified sequence enters the backbone network of the multi-view diffusion model; Remove background tokens based on the foreground mask, retaining only foreground tokens that participated in texture generation; In object-centric multi-view rendering, 3D objects typically occupy only a portion of the canvas in each viewpoint, with a large area belonging to the background. The background area does not correspond to the surface of the 3D object and does not participate in the final texture backprojection. Therefore, encoding the background and foreground areas together as tokens and inputting them into the diffusion backbone would result in significant computational inefficiency.

[0053] This embodiment utilizes this feature to remove background tokens based on the foreground mask before the tokens enter the diffusion Transformer backbone, so that subsequent network computations are focused on the foreground region that actually participates in the generation of 3D textures.

[0054] A binary foreground mask is constructed based on the multi-view rendering results. In a specific application scenario, the foreground region of an object can be determined using the alpha channel of the rendered image. Since the image will form a latent variable token grid after VAE downsampling and tokenization of the diffusion backbone, the pixel-level foreground mask needs to be downsampled to the token grid resolution.

[0055] During downsampling, the foreground mask can be moderately dilated to preserve effective texture information near object boundaries and avoid edge texture loss due to excessively narrow mask boundaries. The set of foreground token positions after downsampling and dilation is denoted as... .

[0056] Based on the set of foreground token positions In the target noise latent variable tokens, geometric condition latent variable tokens, and corresponding spatial location condition tokens, only the positions belonging to the foreground region are retained, and the tokens corresponding to the background region are deleted. After deleting the background tokens, the input to the diffusion backbone is no longer the dense token sequence corresponding to the complete 2048×2048 multi-view canvas, but a compressed foreground token sequence.

[0057] This operation occurs before the diffusion of the Transformer backbone, thus reducing not only the computation of the attention layer, but also the computation of the feedforward network, normalization layer, and other token-by-token modules.

[0058] After deleting the background tokens, the remaining foreground tokens are no longer continuous in the one-dimensional sequence. To avoid losing the spatial position of the foreground tokens in the original multi-view canvas, this embodiment retains the original position index of each foreground token. That is, although the foreground tokens form a compressed sequence, their position codes still correspond to their positions in the original multi-view image.

[0059] This method allows for the reduction of sequence length while maintaining an understanding of the original spatial layout, multi-view positional relationships, and three-dimensional geometric correspondences.

[0060] Step S108: Perform multi-view diffusion generation on the unified sequence that preserves the foreground tokens to obtain the denoised foreground latent variables; Multi-view diffusion generation is performed on the foreground token sequence, and block sparse attention is used to reduce the computational cost of attention. Step S109: Decode the denoised foreground latent variables into a multi-view texture image; High-resolution multi-view texture images are obtained through foreground-aware VAE decoding. Step S110: Back-project the multi-view texture image onto the three-dimensional mesh surface to obtain a textured three-dimensional asset.

[0061] This embodiment outputs 2048×2048 high-resolution texture images from multiple viewpoints. Subsequently, based on the camera parameters during rendering, the visibility of the 3D mesh surface, and the viewpoint fusion rules, the generated multi-view texture images are back-projected onto the 3D mesh surface to obtain the final textured 3D asset.

[0062] In a specific application scenario, the training loss is limited by the foreground region: During the training phase, target noise latent variables are first constructed on the complete latent variable grid, and then based on the foreground token position set... Extract the foreground token sequence. Predict the velocity only for the target tokens corresponding to the foreground region, and calculate the loss function only on the foreground tokens, as shown in the following formula: , in, The flow matching loss represents the foreground region limitation. This represents the velocity field predicted by the model for the foreground token. Indicates the location set based on the foreground tokens The operation of extracting the corresponding token. This loss causes the model to focus on learning texture generation in the foreground region, rather than consuming training computation on invalid background regions.

[0063] Block sparse attention on the foreground token sequence: After removing background tokens, the length of the input sequence is significantly reduced due to attention redundancy within the foreground tokens. However, at a resolution of 2048×2048, the number of retained foreground tokens is still relatively large. The effective interactions between foreground tokens are not entirely dense, and continuing to perform full connected attention computation on all foreground token pairs will still result in redundant computation.

[0064] To further reduce attention costs, this embodiment employs block sparse attention on the compressed foreground token sequence.

[0065] Token segmentation and block-level correlation calculation are performed, assuming the length of the foreground token sequence after background token removal is N, and the attention head dimension is d. The corresponding Query, Key, and Value matrices are denoted as Q, K, and V, respectively.

[0066] First, Q is divided into multiple query blocks, and K and V are divided into multiple key-value blocks. Then, the token features within each block are pooled to obtain a block-level representation, and the block-level correlation matrix between the query block and the key block is calculated. : , Here, `pool` represents the pooling operation on the token features within a block. This block-level correlation matrix is ​​only used to determine the sparse attention layout and does not replace the subsequent actual token-level attention calculation.

[0067] Top-K block selection and token-level attention computation: For each query block, select several key blocks with high relevance based on block-level relevance scores. Let the set of selected key blocks be... Then the query block is only related to Token-level attention computation is performed on the key blocks, while unselected block pairs are skipped. , in, This represents the attention output of the i-th query block. This represents the query features in the i-th query block. and These represent the key and value characteristics corresponding to the selected key-value blocks, respectively.

[0068] The key feature of this approach is that block-level relevance is used only to select the block pairs that need to be computed, while the actual attention output is still calculated from the original, unpooled token features. Therefore, this embodiment can reduce redundant attention computations between irrelevant foreground regions while preserving the fine-grained token interaction capabilities within relevant regions.

[0069] Based on selective denoising inference of the foreground region, this embodiment performs denoising updates only on the latent variable tokens corresponding to the foreground region during the inference stage to further reduce the computational cost of high-resolution generation. Specifically, the target latent variable is initialized from Gaussian noise, and in each sampling time step, only the latent variables belonging to the foreground token position set are denoised. The latent variable tokens are predicted and updated; the latent variables corresponding to the background area are not included in the noise reduction update.

[0070] For the foreground token position Its update process can be represented as: , in, This represents the latent potential variables at the current time step. This represents the updated prospective latent variable. This indicates the speed at which the model predicts the foreground token. This strategy focuses inference computation on the foreground region that will ultimately participate in texture generation, avoiding unnecessary denoising computation on the background region that does not participate in 3D texture formation.

[0071] Foreground Aware VAE Decoding: Hybrid latent variable decoding, while improving efficiency through selective foreground denoising inference, introduces new decoding challenges. Because background latent variables are not properly denoised, the latent variables input to the VAE decoder may consist of both denoised foreground latent variables and noisy background latent variables. Standard VAE decoders are typically unsuitable for this mixed input distribution, and direct decoding can easily lead to artifacts, color shifts, or loss of detail in the foreground region.

[0072] To address this issue, this embodiment proposes a foreground-aware VAE decoding mechanism, as detailed below: This embodiment introduces a standardized background latent variable replacement to replace the undisturbed noisy background latent variable. Specifically, a frozen VAE encoder is used to encode the standardized background image to obtain background latent variables within the normal latent variable distribution. In a specific application scenario, the standardized background image can be a pure black RGB image.

[0073] After obtaining the foreground denoising result through inference, the noise latent variables in the background region are replaced with normalized background latent variables, and combined with the denoised foreground latent variables to obtain mixed latent variables. , , in, This represents the denoised foreground latent variable at position i. Let represent the normalized background latent variable at position i. Through this replacement operation, the background region input to the VAE decoder is no longer Gaussian noise, but a background latent variable within a normal distribution, thereby reducing the interference of background anomalies on the foreground decoding quality.

[0074] Foreground region-constrained VAE decoder adaptation: To further improve the decoding quality of mixed latent variables, this embodiment performs adaptation training on the VAE decoder. During the adaptation process, the VAE encoder remains frozen, and only the VAE decoder is updated. The training loss is calculated only at the pixel level foreground region, allowing the decoder to focus on learning how to recover a high-quality foreground texture image from inputs of denoised foreground latent variables and normalized background latent variables.

[0075] The decoder reconstruction loss with foreground region constraints can be expressed as: , in, This represents the decoding loss of a foreground-aware VAE. This represents the pixel-level foreground region, and D represents the VAE decoder. This represents the true target image. This loss causes the VAE decoder to focus on the reconstruction quality of the foreground region, thereby reducing decoding artifacts and color shifts after selective denoising inference.

[0076] Multi-view texture backprojection involves decoding with a foreground-aware VAE to obtain high-resolution texture images from multiple viewpoints. Subsequently, based on the camera parameters used during rendering, the visibility of the 3D mesh surface, and viewpoint fusion rules, the generated multi-view texture images are backprojected onto the 3D mesh surface. During the backprojection process, for situations where the same surface area may be observed simultaneously from multiple viewpoints, texture information from different viewpoints can be fused based on viewpoint visibility, normal angle, viewpoint distance, or confidence level, resulting in a continuous, clear, and geometrically aligned final texture result.

[0077] Training data and methods: To train a 2048×2048 resolution multi-view diffusion texture generation model, this embodiment can use large-scale high-resolution multi-view rendering data. This data includes a large number of 3D assets, as well as multi-view geometry buffer maps, reference images, normal condition maps, and high-resolution rendering observations obtained from the rendering of 3D assets.

[0078] In a specific application scenario, a six-view normal map is used as the geometric condition, and the training view is set to a resolution of 2048×2048. During training, the system learns to generate corresponding high-resolution multi-view texture images under the constraints of reference image appearance information and multi-view geometric conditions. Background token removal, foreground region loss calculation, and block sparse attention are used together to reduce the training cost of high resolution; foreground-aware VAE decoder adaptation is used to ensure the image reconstruction quality after selective denoising inference.

[0079] This embodiment achieves multi-view 3D texture generation with high detail fidelity, computational efficiency, and decoding quality at a high resolution of 2048×2048 by using foreground mask-guided background token deletion, block sparse attention on the foreground token sequence, selective denoising inference based on the foreground region, and foreground-aware VAE decoding.

[0080] Token-level attention computation refers to a method in block sparse attention where, for query-block and key-value block pairs selected based on block-level relevance, scaled dot-product attention is performed using the unpooled raw latent variable token features within that block to generate attention output at a token-by-token granularity. Unselected block pairs do not participate in this computation. This computation is equivalent to standard Transformer attention in granularity, but by limiting the set of tokens participating in the interaction through block-level routing, redundant computation on the foreground token sequence is reduced.

[0081] This embodiment does not simply adopt the complete multi-view canvas modeling approach. Instead, it addresses the main computational redundancy in 2048×2048 high-resolution 3D texture generation by proposing a foreground-aware sequence compression, block sparse attention, and foreground-aware VAE decoding mechanism. This significantly reduces training and inference costs while preserving high-resolution texture details. Through the foreground-aware multi-view diffusion texture generation framework, 3D texture generation at 2048×2048 high resolution is achieved, combining detail fidelity, computational efficiency, and decoding quality. Compared to existing multi-view diffusion texture generation schemes, it offers the following advantages: 1. Reduce sequence-level redundant calculations through a background token removal mechanism: This embodiment utilizes a foreground mask in multi-view rendering to remove background tokens that do not participate in texture generation before the tokens enter the diffusion backbone, retaining only the foreground tokens corresponding to the 3D object surface, thus shortening the model input length at the sequence level. This mechanism not only reduces the computational load of the attention layer but also simultaneously reduces the computational overhead of the feedforward network, normalization layer, and other token-by-token modules, allowing computational resources to be concentrated on the foreground region that truly corresponds to the 3D surface texture. This significantly reduces redundant computations in high-resolution multi-view diffusion generation, lowering memory usage and inference latency.

[0082] 2. Further reduce the attention overhead within the foreground token by using block sparse attention: After removing background tokens, this embodiment further employs a block-sparse attention mechanism on the retained foreground token sequence. By dividing the foreground tokens into query blocks and key-value blocks, block-level relevance estimation is used to select the block pairs that need interaction. Token-level attention computation is performed only between the selected block pairs, skipping block pairs with low relevance. This mechanism further reduces unnecessary fully connected attention computation between foreground tokens while preserving effective texture information interaction, thus improving the computational efficiency of high-resolution multi-view diffusion generation.

[0083] 3. Ensure decoding quality after selective denoising inference through foreground-aware VAE decoding: This embodiment addresses the mixed latent variable decoding problem caused by selective denoising inference based on foreground regions by proposing a foreground-aware VAE decoding mechanism. By replacing the undennoised background region latent variables with normalized background latent variables obtained by encoding the normalized background image using a frozen VAE encoder, and by performing foreground region constraint-based adaptation training on the VAE decoder, the decoder can recover high-quality foreground texture images from a mixed input of denoised foreground latent variables and normalized background latent variables. This mechanism effectively reduces foreground region artifacts, color shifts, and detail loss caused by abnormal background latent variables, ensuring the final decoding quality of high-resolution multi-view texture images.

[0084] 4. Supports high-resolution texture generation and improves training and inference efficiency: This embodiment supports multi-view texture generation at a resolution of 2048×2048. While preserving fine-grained texture information and high-frequency appearance details in the reference image, it significantly reduces memory usage and computational latency during training and inference. Compared to existing texture generation methods operating at 512 or 768 resolution, this embodiment provides richer and clearer texture details. Compared to solutions that directly extend the complete sequence modeling method to high resolution, this invention has a significant advantage in computational efficiency. The final generated multi-view texture image is strictly aligned with the input 3D geometry, and after backprojection, a high-quality 3D texture asset is obtained, suitable for applications with high texture quality requirements such as games, film and television, e-commerce, and digital twins.

[0085] 5. Higher training and reasoning efficiency: Because this embodiment removes background tokens before the DiT backbone and further employs block sparse attention on the foreground token sequence, it can significantly reduce the number of tokens and the computational cost of attention in high-resolution multi-view diffusion generation. Compared to the complete sequence modeling scheme, this invention can reduce memory usage, shorten training and inference time, and make multi-view texture generation at 2048×2048 resolution more practically feasible.

[0086] This implementation uses a six-view normal map obtained from 3D mesh rendering as the geometric condition to guide the multi-view diffusion model in generating a texture image aligned with the 3D geometry. The normal map can be replaced by a depth map, position map, contour map, visibility map, or mask map, or a multi-channel geometric condition map formed by combining various geometric attributes such as normal maps, depth maps, and position maps. As long as the geometric condition map can represent the geometric structure of the 3D object from various viewpoints and can constrain the generated texture image to align with the 3D geometry, it can be used in this embodiment.

[0087] The six-view rendering layout can also employ four-view, eight-view, twelve-view, or other multi-view layouts; each viewpoint can also be a surround view, orthogonal view, perspective view, or a viewpoint adaptively selected based on the object's shape. As long as the generated multi-view image can cover the main visible surfaces of the 3D object to be textured and can be used for subsequent texture back projection, it can be used in this embodiment.

[0088] The geometric conditions can be obtained by rendering the input 3D mesh directly, or by rendering a point cloud, voxel, implicit surface, or other 3D geometric representation after conversion. As long as a geometric condition map for multi-view texture generation can be obtained, it can be used in this embodiment.

[0089] This implementation extracts a binary foreground mask from the alpha channel of the multi-view rendered image and downsamples the foreground mask to the latent variable token grid resolution to determine the foreground token positions that need to be retained.

[0090] Foreground masks can be generated independently of the alpha channel, based on depth maps, object contours, render visibility, semantic segmentation results, background color differences, or visible face information obtained through rasterization. As long as the mask can distinguish between foreground regions involved in 3D texture generation and background regions not involved in texture generation, it can be used for background token removal or background token reduction.

[0091] When downsampling to token resolution, the foreground mask can employ nearest neighbor downsampling, average pooling, max pooling, thresholding, or morphological dilation. For object boundary regions, a certain width of boundary extension region can also be preserved to avoid deleting tokens that may contain valid texture information at the edges.

[0092] The foreground mask can be a binary hard mask or a soft mask with continuous weights. For a binary hard mask, foreground tokens are directly retained and background tokens are deleted; for a soft mask, tokens can be selected to be retained based on their weights, or the weights can be used to control the degree to which different tokens participate in the backbone computation. Both methods can achieve the goal of reducing redundant background computation.

[0093] Regarding the background token removal and sequence reduction method, this embodiment removes background tokens based on the foreground mask before the tokens enter the DiT or MM-DiT backbone, retaining only the foreground tokens that participate in texture generation, and preserving the position index of these foreground tokens in the original multi-view canvas.

[0094] Background tokens can be inadvertently removed. Instead, they are aggregated or downsampled based on tokens identified as background regions by the foreground mask, forming a small number of aggregated background tokens. These aggregated tokens are only used to provide necessary background context or canvas boundary information and do not participate in the full high-resolution token computation as the main texture generation region. This approach shares similarities with directly removing background tokens in that both reduce the participation of tokens corresponding to background regions in the backbone network computation before the tokens enter the DiT or MM-DiT backbone, thereby reducing the sequence length and computational cost in high-resolution multi-view diffusion generation.

[0095] Different token preservation strategies can be employed for foreground regions, object boundary regions, and pure background regions far from objects. For example, foreground tokens can be fully preserved, tokens near the boundary can be partially preserved, and background tokens far from the object can be deleted or aggregated. This approach also relies on foreground masks to reduce invalid background computations while maintaining texture generation quality near object boundaries.

[0096] Token filtering can be based on an initial foreground mask and dynamically adjust the foreground expansion area and background reduction range by combining object boundary width, geometric visibility, foreground proportion, or generation quality requirements. This approach does not deviate from the foreground mask's distinction between foreground and background regions, but rather adjusts the range of tokens retained within this premise, thereby balancing computational efficiency and boundary texture quality.

[0097] After removing or aggregating background tokens, the spatial location of foreground tokens can be represented by the original 2D position index, camera view index, relative position encoding, rotational position encoding, or learnable position encoding. This embodiment can be used as long as the model can recover or utilize the spatial relationships of the foreground tokens in the original multi-view canvas after sequence reduction.

[0098] For the block sparse attention mechanism, this embodiment adopts the block sparse attention mechanism on the preserved foreground token sequence. That is, the Query, Key and Value are first divided into multiple blocks, the key blocks with high relevance are selected by block-level relevance estimation, and then token-level attention calculation is performed only between the selected block pairs.

[0099] The token blocks can be divided according to a one-dimensional sequence, or according to the original two-dimensional image position, viewpoint number, spatial neighborhood, or geometrically visible area. As long as a query block and key-value block can be formed for sparse attention selection, it can be used in this embodiment.

[0100] Block-level correlations can be calculated using average pooling features, or they can be calculated using max pooling features, attention pooling features, learnable aggregate features, or low-rank projection features. As long as the correlation is used to determine whether subsequent token-level attention calculations are needed between different blocks, it can be used in this embodiment.

[0101] Top-K block selection can be replaced by fixed-ratio selection, threshold selection, adaptive-ratio selection, local window selection, global key block selection, or geometric adjacency selection. As long as the selection strategy can skip low-relevance block pairs and reduce redundant attention computations between foreground tokens, it can be used in this embodiment.

[0102] Block sparse attention can also be replaced by local window attention, sliding window attention, clustering sparse attention, low-rank attention, hash attention, or geometry-aware sparse attention. As long as its target is the token sequence after foreground filtering and it is used to reduce the amount of attention computation between foreground tokens, it can be used in this embodiment.

[0103] For selective denoising based on the foreground region, this embodiment only performs denoising updates on the latent variable tokens corresponding to the foreground region during the inference stage, while the latent variables in the background region do not participate in the denoising update, so as to reduce the cost of high-resolution inference.

[0104] During the inference phase, full denoising can be performed on the foreground region, a smaller number of denoising steps on the boundary region, and no denoising is performed on the background region far from the object. This hierarchical denoising method also allows the main computation to be focused on the foreground region involved in texture generation.

[0105] The background area can also be filled in using a low-resolution background, a uniform background, a normalized background, or a simple interpolated background, instead of maintaining the initial noise state. As long as the background area no longer occupies full high-resolution diffusion computing resources and does not affect the quality of foreground texture generation, it can be used in this embodiment.

[0106] The foreground region retained during the inference phase can be dynamically adjusted based on object size, foreground proportion, generation quality requirements, or computational resources. For objects with high detail requirements, the foreground region can be expanded; for scenarios with limited computational resources, the region can be reduced. This approach still belongs to the category of efficient inference schemes based on foreground regions.

[0107] For the foreground-aware VAE decoding method, this embodiment replaces the noisy background latent variable with the normalized background latent variable obtained by freezing the VAE encoder from the normalized background image, and performs foreground region constraint adaptation on the VAE decoder.

[0108] The standard background image can be a pure black image, a pure white image, a pure gray image, a transparent background image, an average background image, or a background image obtained statistically from training data. As long as the background latent variable obtained after VAE encoding is within a normal distribution and can replace the noisy background latent variable, it can be used in this embodiment.

[0109] The normalized background latent variables can be obtained by encoding with a frozen VAE encoder, or they can be used as learnable parameters, average latent variables of the dataset, average latent variables of the categories, or predicted by a lightweight network. As long as they can provide stable and normally distributed latent variable inputs for the background region, they can be used in this embodiment.

[0110] When adapting the VAE decoder, in addition to foreground region reconstruction errors, perceptual loss, color consistency loss, boundary smoothing loss, or foreground mask weighted loss can also be used. As long as the training objective can enable the decoder to recover a high-quality foreground texture image under mixed latent variable input, it can be used in this embodiment.

[0111] The standard VAE decoder can be replaced with an adapted image decoder, lightweight convolutional decoder, Transformer decoder, or other latent variable-to-image decoding network. As long as the decoder can receive a mixture of latent variables consisting of denoised foreground and normalized background latent variables and output a high-quality multi-view texture image, it can be used in this embodiment.

[0112] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0113] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the foreground-aware multi-view diffusion method for three-dimensional texture generation described in the foregoing embodiments of this disclosure.

[0114] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0115] like Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0116] like Figure 2 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0117] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 2 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0118] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the foreground-aware multi-view diffusion method for three-dimensional texture generation according to embodiments of this disclosure are performed.

[0119] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0120] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the foreground-aware multi-view diffusion method for three-dimensional texture generation according to the foregoing embodiments of the present disclosure are performed.

[0121] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0122] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0123] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0124] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0125] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0126] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0127] Various changes, substitutions, and modifications can be made to the techniques described herein without departing from the teachings defined in this embodiment. Furthermore, the scope of this embodiment is not limited to the specific aspects of the processes, machines, manufacturing processes, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufacturing processes, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, this embodiment includes such processes, machines, manufacturing processes, events, means, methods, or actions within its scope.

[0128] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0129] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A foreground-aware multi-view diffusion method for 3D texture generation, characterized in that, include: Obtain the reference image and the 3D mesh to be textured; Render geometric condition maps from multiple perspectives from the 3D mesh; The reference image is encoded as a reference image latent variable, and the geometric condition map is encoded as a geometric condition latent variable; The target noise latent variable is obtained based on the image latent variable and the geometric condition latent variable; Reference image latent variable tokens, geometric condition latent variables, and target noise latent variables are obtained based on reference image latent variables, geometric condition latent variables, and target noise latent variables, respectively. The reference image latent variable token, the geometric condition latent variable token, and the target noise latent variable token are concatenated into a unified sequence and then input into the backbone network of the multi-view diffusion model for joint modeling. Based on the foreground mask, background tokens are removed before the unified sequence enters the backbone network of the multi-view diffusion model, and only foreground tokens are retained. Multi-view diffusion generation is performed on a unified sequence that preserves foreground tokens to obtain denoised foreground latent variables; The denoised foreground latent variables are decoded into a multi-view texture image; The multi-view texture image is back-projected onto the surface of the three-dimensional mesh to obtain a textured three-dimensional asset.

2. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, The multi-view diffusion generation uses Flow Matching as the training objective, constructing noise target latent variables along a linear path. , And the model predicts the velocity field to approximate the target velocity. For training objectives; in, For the target multi-view texture image latent variables, For time steps, It is Gaussian noise. The target speed.

3. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, The geometric condition map includes a normal map, a depth map, a position map, and / or a contour map; The foreground mask is constructed based on the alpha channel, depth map, or semantic segmentation results of multi-view rendering.

4. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, After removing the background token, the retained foreground token maintains the spatial position index of the foreground token in the original multi-view canvas.

5. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, The multi-view diffusion generation is trained using a foreground region-constrained Flow Matching loss, including: During the training phase, target noise latent variables are first constructed on the complete latent variable grid, and then the foreground token sequence is extracted based on the foreground token position set. The model predicts the speed only for the target tokens corresponding to the foreground region, and the loss function is calculated only on the foreground tokens, as follows: ， in, The flow matching loss represents the foreground region limitation. This represents the velocity field predicted by the model for the foreground token. Indicates the location set based on the foreground tokens The operation to retrieve the corresponding token. Indicates Gaussian noise. Represents the latent variables of the target multi-view texture image.

6. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, The process involves performing multi-view diffusion generation on a unified sequence that preserves foreground tokens to obtain denoised foreground latent variables. This includes employing a block sparse attention mechanism, which includes: Divide the query Q, key K, and value V into multiple blocks; Calculate the block-level correlation between the query block and the key block; Select the query block and key block based on the block-level relevance; Token-level attention computation is performed only between the selected query block and key block pair.

7. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, The multi-view diffusion generation includes selective denoising based on the foreground region, that is, denoising updates are performed only on the latent variables corresponding to the foreground token positions; The update process is represented as follows: ， in, This represents the latent potential variables at the current time step. This represents the updated prospective latent variable. This indicates the speed at which the model predicts the foreground token. Represents the set of foreground token positions. Indicates the updated time step. Indicates the current time step.

8. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, Decoding the denoised foreground latent variables into a multi-view texture image includes: using foreground-aware VAE decoding. The foreground-aware VAE decoding includes standardized background latent variable replacement, which includes: After obtaining the denoised foreground latent variable, the noise latent variable of the background region is replaced with the normalized background latent variable, and combined with the denoised foreground latent variable to obtain the mixed latent variable; The canonical background latent variable is obtained by encoding the canonical background image using a frozen foreground-aware VAE encoder. The frozen foreground-aware VAE encoder keeps the encoder frozen and only updates the VAE decoder.

9. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 8, characterized in that, The foreground-aware VAE decoding includes foreground region-constrained VAE decoder adaptation, which includes: The foreground-aware VAE decoder is adapted and trained. During the training process, the foreground-aware VAE encoder is kept frozen and only the foreground-aware VAE decoder is updated. The training loss is calculated only in the pixel-level foreground region, and is expressed as: ， in, This represents the decoding loss of a foreground-aware VAE. Represents the pixel-level foreground area. This represents a foreground-aware VAE decoder. Represents a real target image. Indicates the position index.

10. The foreground-aware multi-view diffusion method for 3D texture generation according to claim 1, characterized in that, Back-projecting the multi-view texture image onto the three-dimensional mesh surface includes: Based on the camera parameters used during rendering, the visibility of the 3D mesh surface, and the view fusion rules, the generated multi-view texture image is back-projected onto the 3D mesh surface; For the same surface region observed simultaneously from multiple viewpoints, texture information from different viewpoints is fused based on one or more of the following: viewpoint visibility, normal angle, viewpoint distance, and confidence level, to obtain a texture result aligned with the three-dimensional geometry.

Images