Pixel-level accurate texture generation method and device based on multi-view artistic works

Abstract

The invention discloses a pixel-level accurate texture generation method based on a multi-view artistic work. The method comprises the following steps: inputting an image file; performing multi-view projection on the 3d model for a plurality of views corresponding to the plane image to construct a texture space; aiming at coexisting points on the surface of the 3d model in the multiple visual angles, calculating a dot product of each visual angle direction and a normal of the surface of the 3d model so as to construct an initial texture map corresponding to the 3d model; projecting all contour points of the 3d model into a texture space to obtain corresponding texture information, and comparing the texture information with contour points contained in the initial texture map to construct a mask area; generating a shielding texture map corresponding to the mask area through a diffusion model; and carrying out rendering processing on the 3d model based on the initial texture map and the sheltered texture map so as to output a high-quality model with texture information. The invention further provides a pixel-level accurate texture generation device. According to the method provided by the invention, the high-quality texture can be generated for the grid, and meanwhile, the pixel-level consistency with the reference art is ensured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a pixel-level accurate texture generation method and device based on multi-view artistic works. BACKGROUND

[0002] Nowadays, the progress of two-dimensional diffusion models enables the automatic generation of textures for 3D meshes, which mainly falls into two categories: text-guided and image-guided methods. Text-guided texture generation methods initially utilize iterative inpainting or fractional distillation sampling to optimize color fields. These iterative techniques handle each view independently, resulting in artifacts and excessive time consumption. To address the multi-view inconsistency issue, some methods combine multi-view diffusion with geometric constraints and blend latent features from different viewpoints in UV space. However, these methods still fail to ensure complete consistency, often producing blurry artifacts. While text prompts reduce the difficulty of texture generation and provide concise semantic guidance, they lack precision in managing styles and patterns. To address this issue, image-guided texture generation methods provide richer visual cues. Single-image methods provide guidance but lack back and side information, and even frontal details may not fully match the input. Multi-view image-based methods provide additional context but often rely on Stable Diffusion fine-tuning based on semantic content without ensuring pixel-level detail consistency with input images.

[0003] In recent years, some research has integrated image prompts into text-to-image diffusion models that were initially controlled only by text conditions. DreamBooth is a pioneering work that demonstrates the ability to update the entire diffusion model using only a small number of images of specific themes or styles. Other fine-tuning methods, such as text inversion and LoRA, further minimize the parameter requirements for fine-tuning. Subsequent methods introduce theme-driven image generation without the need for fine-tuning, significantly reducing computational costs. However, these techniques often fail to ensure pixel-level detail consistency.

[0004] Patent document CN119809920A discloses a virtual fitting system and method based on a diffusion model for preserving local details, which includes a module for acquiring images of clothes and users to be fitted, a feature processing module for semantic segmentation and posture morphology of model images, a diffusion model generation module for generating a virtual fitting effect diagram, and an output module for feeding back the generated virtual fitting effect diagram to the user; the diffusion model generation module generates a high-fidelity fitting effect diagram from a noise image through a pre-trained diffusion model; the local detail enhancement module uses a double-flow structure and different weight loss functions to improve the detail fidelity of the clothing area; the feature processing module can obtain the features required by the diffusion model based on the user-inputted model image to be fitted and the clothing white background image; and the output module can feed back the generated result to the user terminal in real time.

[0005] Patent document CN119963708A discloses a three-dimensional digital human generation method and system, which includes: acquiring a front image data set and preprocessing to obtain a target data set; constructing a back surface generation normal fusion network model, and training and testing the back surface generation normal fusion network model according to the target data set; inputting the front image of the three-dimensional digital human to be generated into the back surface generation normal fusion network model after testing, and the back surface generation normal fusion network model outputs the three-dimensional digital human corresponding to the front image of the three-dimensional digital human to be generated. SUMMARY

[0006] The purpose of the present application is to provide a pixel-level accurate texture generation method and device based on multi-view artistic works, which can generate high-quality textures for grids while ensuring pixel-level consistency with reference art.

[0007] To achieve the first purpose of the present application, the following technical solution is provided: a pixel-level accurate texture generation method based on multi-view artistic works, comprising the following steps: Inputting an image file including multiple planar images with texture information under multiple views and a corresponding 3D model; Performing multi-view projection on the 3D model for multiple views corresponding to the planar images to construct a texture space containing texture information in each view; For coexisting points on the surface of the 3D model in multiple views, calculating the dot product of each view direction and the normal of the 3D model surface, and taking the texture information under the maximum view corresponding to the dot product result as the final texture information of the coexisting points, and synchronizing the texture space using the final texture information to construct an initial texture map corresponding to the 3D model; Projecting all contour points of the 3D model into the texture space to obtain texture information corresponding to all contour points, and comparing the contour points contained in the initial texture map to construct a mask area; input the initial texture map and the mask region into a pre-built diffusion model to generate an occlusion texture map corresponding to the mask region; perform rendering processing on the 3D model based on the initial texture map and the occlusion texture map to output a high-quality model with texture information.

[0008] Specifically, the image file needs to be preprocessed before input, and the preprocessing includes view alignment of planar images with texture information under multiple viewing angles based on the 3D grid and orthogonal camera of the 3D model.

[0009] Specifically, the view alignment is implemented by using a double U-Net framework, which includes an Align U-Net for extracting pixel-level details in the input image and a Denoise U-Net for performing a denoising process.

[0010] Specifically, the expression of the view alignment is as follows: ; wherein, represents the Align U-Net, represents the number of output views, represents a randomly initialized noise latent variable, represents a geometry condition map rendered from the input grid, and represents the trainable weights of the Align U-Net, the Denoise U-Net, and the geometry encoder, and represents the pre-training parameters of the VAE and the CLIP encoder.

[0011] Specifically, the expression of the mask region is as follows: ; wherein, T represents a threshold, and superscripts i and j represent the indices of the contour points, represents the dot product of the corresponding contour points, represents the unoccluded region, and .

[0012] Specifically, the rendering processing on the 3D model also includes rendering work on the connection area between the initial texture map and the occlusion texture map, which is completed by a geometry-aware color extension method.

[0013] Specifically, the process of the color extension method is as follows: The contour point coordinates based on the 3D model are used to construct a KD tree, and the contour points contained in the connection area between the initial texture map and the occlusion texture map are searched through the KD tree to obtain K nearest neighbors of each contour point, and the RGB values of the K nearest neighbors are used to calculate the texture information of the corresponding contour point based on the distance of the K nearest neighbors in the model space through reverse weighting calculation. The 3D model is rendered based on the initial texture map, the texture information of all contour points in the connection area, and the occlusion texture map.

[0014] To achieve the second object of the present application, the following technical solution is provided: a pixel-level accurate texture generation device for executing the steps of the pixel-level accurate texture generation method based on multi-view artwork described above.

[0015] Compared with the prior art, the present application has the following advantages: Without fine-tuning, the reference image is directly used to ensure that the generated results remain consistent under different viewing angles and achieve high-precision pixel-level detail reproduction, thereby improving the quality and efficiency of 3D texture generation. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the pixel-level accurate texture generation method based on multi-view artwork provided by the present embodiment is shown in the figure. Figure 2 The result schematic diagram of the different rendering methods of the connection area provided by the present embodiment is shown in the figure. DETAILED DESCRIPTION

[0017] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0018] To better illustrate the content provided by the present example, non-generic abbreviations or self-defined terms are described: AlignTex: the method proposed in the present study for generating high-quality textures from multi-view artwork, ensuring pixel-level alignment and optimizing the 3D asset creation process.

[0019] AlignNet: The core component of the AlignTex method, a diffusion model-based alignment network that extracts pixel-level details of artistic works while ensuring geometric alignment with 3D meshes.

[0020] UV Texture Space: A two-dimensional texture coordinate space that the 3D model is unfolded into, used for mapping and storing texture information.

[0021] Texture Extension: A method used to handle UV seam artifacts by filling the UV boundaries with color extension based on 3D geometry, reducing visible cracks during rendering.

[0022] KDTree: A data structure used for efficient nearest neighbor search, used in the AlignTex method for texture extension based on 3D geometric positions.

[0023] ControlNet: An additional component of the diffusion model that allows the input of additional geometric or depth information, used by AlignTex for comparative experiments on geometric alignment.

[0024] U-Net: A deep learning network architecture with an encoder-decoder structure that combines high-level features with low-level features through skip connections, effectively capturing multi-scale features.

[0025] CLIP Encoder: Used to extract high-level semantic features of the input artistic work and guide AlignNet to generate aligned textures.

[0026] As Figure 1 shown, the pixel-level accurate texture generation method based on multi-view artistic works provided in this embodiment includes the following steps: To generate images that can be projected into the texture space, standard orthogonal projection images derived from multi-view artistic works are used to ensure consistency with 3D meshes and reference artistic works.

[0027] In the denoising process, a dual U-Net framework is used to generate high-quality and aligned images. This framework utilizes geometric information from the input mesh and captures fine-grained details from the reference artwork. The dual U-Net framework consists of two components: an Align U-Net that extracts pixel-level details from the reference artwork, and a Denoise U-Net that performs the denoising process.

[0028] To enhance AlignNet's perception of orthogonal views and 3D meshes, we render depth and normal maps of the input mesh from orthogonal perspectives as geometric cues. These maps contain the contours and fine geometric details of the model, and provide important camera pose information.

[0029] A geometry encoder processes these depth and normal maps, extracting geometric condition features consistent with the latent space dimensionality.

[0030] To preserve the integrity of these features and prevent interference from harmful noise, a zero convolution layer is employed. This ensures that the extracted features remain accurate and unaffected during the fine-tuning process.

[0031] Finally, the processed geometric condition features are integrated into the denoising pipeline by being added to the noise latent variable as input to the Denoise U-Net.

[0032] The main function of Align U-Net is to extract pixel-level details from the reference artwork, just like Denoise U-Net, Align U-Net leverages the weights inherited from the original Stable Diffusion model. The artwork is first encoded using a VAE encoder to generate a latent representation, which serves as input to Align U-Net. Then, Align U-Net generates intermediate feature maps, which are subsequently concatenated with the feature maps from Denoise U-Net as input to self-attention layers, providing precise pixel-level guidance throughout the denoising process.

[0033] To incorporate general semantic information, a CLIP encoder is used to extract high-level semantic features from the input artwork, which are then injected into the cross-attention mechanisms of Align U-Net and Denoise U-Net.

[0034] Formally, the high-quality multi-view image alignment process can be represented as: ; where, represents Align U-Net, represents the number of output views, represents a randomly initialized noise latent variable, represents the geometry condition map rendered from the input mesh, and represents the trainable weights of Align U-Net, Denoise U-Net, and the geometry encoder, and represents the pre-trained parameters of the VAE and CLIP encoder.

[0035] After obtaining the camera-aligned images from AlignNet, we project them back to texture space using the known orthographic camera matrices and treat the view-space images as textures, and “render” each of the four views into texture space, which effectively improves efficiency and effectively eliminates hole artifacts and distortions. For each texel, we choose the color from the view with the highest weight, resulting in a clearer, more coherent initial texture map, denoted as . In addition, we generate a weight texture map that records the maximum weight across views, where the weight texture map serves as a key indicator for identifying peripheral regions in subsequent steps.

[0036] After the aligned views are projected into texture space, certain regions require further refinement, and these regions include the peripheral regions at the edges of the field of view, as well as regions that are completely occluded or invisible from all orthographic views. We will outline the process of identifying these regions, which are represented using a repair mask.

[0037] where the peripheral regions often exhibit seam artifacts due to slight color differences between views. These regions can be identified by the dot product between the surface normal and the view direction, denoted as . Larger dot product values correspond to higher confidence for a given viewpoint. Regions where the confidence values across all viewpoints are below a certain threshold require additional processing to ensure consistency and accuracy.

[0038] For completely invisible regions, including regions inaccessible from the provided views (e.g., the top of the head, the bottom of the feet) or self-occluded surfaces in complex models, in this embodiment, we calculate the positional discrepancy at the pixel level in model space, i.e., render the 3D positions to view space and project them back to texture space, and then compare the results with the ground-truth position map. Any discrepancies between these measurements highlight occluded regions. Then, the mask in UV space is used to effectively represent all unoccluded regions.

[0039] where the final repair mask is calculated as:

[0040] where denotes a predefined threshold, and the superscripts and represent the indices of the masks.

[0041] After fine-tuning a repair diffusion model designed specifically for UV space, we integrate the necessary conditional information through ControlNet, and utilize the repair mask to seamlessly blend known and unknown areas.

[0042] The denoising process is described by the following equation:

[0043] in, Based on a given texture ( latent features) sampling, while Based on the previous iteration These components are then sampled from the model using the inpainting mask Combined to form an updated sample .

[0044] Conditional Mapping for ControlNet C including Projected Textures , confidence map , normal maps that provide geometric details, and position maps to ensure spatial continuity within the 3D space.

[0045] The process of rendering the 3D model also includes rendering the connection area between the initial texture map and the occlusion texture map, which is completed by a geometry-aware color expansion method.

[0046] like Figure 2 , which is a schematic diagram of the results of different rendering methods of the connection area provided by this embodiment.

[0047] Due to the limited resolution of texture maps, directly using UV coordinates to sample and render texture maps often results in discontinuous colors in the connection areas of UV patches.

[0048] like Figure 2 As shown in Figure 2 (a), sampling texels (highlighted in green) results in black seams in the rendered output. A simple approach to address the junctions between UV patches is to dilate the original color. However, simply randomly selecting the nearest known pixel fails to account for the spatial connectivity between distant UV islands, leading to discontinuities in pattern alignment. This problem is particularly pronounced with highly detailed patterns or overly fragmented UV layouts, as shown in Figure 2 (b).

[0049] The method provided in this embodiment uses a geometry-aware color expansion method, as shown in Figure 2(c). A KD tree is constructed using the 3D position value P of the texel indexed by (i, j) in the UV position map. The KD tree facilitates efficient querying of spatial relationships between texels in model space, which is crucial for handling fragmented UV islands.

[0050] For texture information (m, n) that needs color filling, the KD tree is queried using its 3D position value P to identify the K nearest neighbors in the model space and obtain their corresponding texel indices. Then the position and color information of these neighbors are retrieved. The final color of each texel is calculated by inversely weighting the RGB values of these neighbors according to their distances in the model space, to ensure smooth transitions across UV boundaries and maintain the continuity of the pattern.

[0051] The embodiment also provides a pixel-level accurate texture generation device for executing the steps of the pixel-level accurate texture generation method based on multi-view concept art provided by the above-mentioned embodiment.

[0052] In order to better illustrate the technical effects of the method provided by the embodiment, the method is compared with the current popular (SOTA) image-guided texture generation baseline. For methods that can process multi-view input, such as TEXTure and TextureDreamer, we provide four-view concept art similar to our method. In terms of single-view methods, including PGC3D, Paint3D and EASI-Tex, they are evaluated using only the front view concept art as input. The evaluation dataset of our method contains 50 character models and their corresponding concept art.

[0053] The 3D models are assets obtained from published online games, and the accompanying concept art is recreated by artists we hired according to confidentiality requirements. In addition, we also include 50 models randomly selected from Objaverse. Since the Objaverse models do not have original concept art, we generate multi-view concept art by rendering images with perturbed camera poses and models to For evaluation purposes, the final quantitative results are shown in Table 1.

[0054]

[0055] In summary, the beneficial results brought by the method can be summarized as follows: A high-precision texture generation method based on multi-view concept art and geometric alignment, which is basically superior to other texture generation methods based on text or images in various quantitative indicators.

[0056] The method can realize pixel-level detail alignment without manual adjustment, significantly improving the applicability and generalization ability of automatic texture generation, making it adaptable to diverse 3D asset production needs.

[0057] The method strictly maintains the visual details of the input artwork while ensuring geometric consistency, thereby achieving both high-quality texture generation and artistic style preservation.

[0058] In addition, the terms "upper", "lower", "inner", "outer", "front", "back" are only used for description purposes and cannot be understood as indicating or implying relative importance. Unless otherwise specified, the relative steps, numerical expressions and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0059] Of course, the above only describes specific embodiments of the present application and is not intended to limit the scope of the present application. Equivalent changes or modifications made in accordance with the structure, features and principles described in the patent application scope of the present application shall be included in the patent application scope of the present application.

[0060] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, and are not intended to limit the scope of the present application. The protection scope of the present application does not limit to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make equivalent changes or modifications to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or easily think of changes or modifications to the technical solutions described in the foregoing embodiments, or make equivalent substitutions to some technical features. The essence of the corresponding technical solution does not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be limited to the protection scope of the claims.

Claims

1. A method for generating pixel-level accurate texture based on multi-view artwork, characterized in that, The method comprises the following steps: inputting an image file comprising planar images with texture information under multiple perspectives and a corresponding 3D model; performing multi-perspective projection on the 3D model for multiple perspectives corresponding to the planar images to construct a texture space containing texture information in each perspective; calculating the dot product of each perspective direction and the normal of the surface of the 3D model for the coexisting points on the surface of the 3D model, and taking the texture information corresponding to the perspective with the maximum dot product as the final texture information of the coexisting points, and synchronizing the texture space by using the final texture information to construct an initial texture map corresponding to the 3D model; projecting all contour points of the 3D model into the texture space to obtain the texture information corresponding to all contour points, and comparing the contour points contained in the initial texture map to construct a mask region; inputting the initial texture map and the mask region into a pre-constructed diffusion model to generate an occlusion texture map corresponding to the mask region; performing rendering processing on the 3D model based on the initial texture map and the occlusion texture map to output a high-quality model with texture information.

2. The method of claim 1, wherein, The image file needs to be preprocessed before input, and the preprocessing comprises view alignment of the planar images with texture information under multiple perspectives based on the 3D grid of the 3D model and orthogonal cameras.

3. The method of claim 2, wherein, The view alignment is implemented by using a double U-Net framework, which comprises an Align U-Net for extracting pixel-level details in the input image and a Deniose U-Net for performing a denoising process.

4. The method of claim 3, wherein, The expression of the view alignment is as follows: ； wherein, denotes an Align U-Net, denotes the number of output views, denotes a randomly initialized noise latent variable, denotes a geometry condition map rendered from the input mesh, with denotes the trainable weights of the Align U-Net, DenoiseU-Net, and the geometry encoder, and denotes the pre-trained parameters of the VAE and CLIP encoders.

5. The method of claim 1, wherein, The expression of the mask region is as follows: ； In the formula, T represents a threshold, superscripts i and j represent indices of contour points, representing ( ) a dot product of the corresponding contour points, representing an unoccluded region in the .

6. The method of claim 1, wherein, In the process of rendering processing on the 3D model, the rendering work of the transition area between the initial texture map and the occlusion texture map is also included, which is completed by a geometrically aware color expansion method.

7. The method of claim 7, wherein, The process of the color expansion method is as follows: KD trees are constructed based on the contour point coordinates of the 3D model as p values, and the contour points contained in the transition area between the initial texture map and the occlusion texture map are searched by the KD trees to obtain K nearest neighbors of each contour point, and the RGB values of the K nearest neighbors are used to perform reverse weighted calculation based on the distances of the K nearest neighbors in the model space to determine the texture information of the corresponding contour point; performing rendering on the 3D model based on the initial texture map, the texture information of all contour points in the transition area, and the occlusion texture map.

8. A pixel-level accurate texture generation device, characterized in that: A step for performing the pixel-level accurate texture generation method based on multi-perspective artwork according to any one of claims 1-8.

Images