Rendering texture map generation method and device based on ultrahigh-resolution portrait
Through the strategies of global texture extraction and local texture extraction, high-quality rendered texture maps are directly generated from ultra-high resolution portraits, solving the problems of inaccurate texture details and high computing resources in the existing technology, and achieving efficient detail capture and resource conservation.
Patent Information
- Application Number
- CN202411561729.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-11-05
AI Technical Summary
The prior art is difficult to generate high-quality facial material maps directly from ultra-high resolution inputs, resulting in inaccurate or loss of texture details, and the huge computing resources for processing ultra-high resolution images, limiting the widespread application of the method.
Adopt the divide-and-conquer strategy of global texture extraction and local texture extraction, extract global textures through downsampling and hash encoding, and combine with neural networks to optimize high-frequency details, directly generate end-to-end rendered texture maps from ultra-high-resolution portraits to avoid middle super-super Texture distortion caused by resolution technology.
It realizes accurate capture of details from ultra-high resolution inputs, and the generated texture map is consistent with the input, which is robust and high-detail expressive, reducing the computing resource requirements and improving the scalability of the method.
Smart Images

Figure CN119068098A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer graphics and image technology, and specifically relates to a method and apparatus for generating rendering texture maps based on ultra-high resolution human portraits. Background Technology
[0002] With the widespread application of virtual reality, gaming, and multimedia platforms, realistic and personalized virtual avatars play a crucial role in enhancing the user's immersive experience. To achieve highly realistic virtual avatar rendering, especially in facial rendering, it is necessary to accurately capture subtle facial features such as pores, blush, wrinkles, and freckles. These details are essential for enhancing the realism of facial appearance and helping to narrow the gap between digital virtual avatars and physical reality. To obtain textures with rich facial details, including albedo maps, micronormal maps, and specular maps, it is usually necessary for the subject to personally visit the collection location and use complex and professional dome light field (Light Stage) capture equipment and accurate calibration procedures to capture a large amount of data. This results in expensive manpower and equipment costs for obtaining these ultra-high-definition textures, making it difficult to use widely.
[0003] In recent years, advancements in mobile device camera technology have made it incredibly convenient to capture single 4K (4096×4096) resolution ultra-high-definition portraits using handheld devices. This has enabled the generation of equally high-resolution (4K) facial texture maps from single ultra-high-definition portrait photos. However, existing monocular face texture reconstruction methods are typically based on input sizes of 256x256 or 512x512, making it impossible to fully utilize the extremely rich information in 4K input to generate detailed textures. Consequently, a significant amount of information from the 4K input is wasted during the downsampling process to the input size.
[0004] While some existing methods attempt to generate ultra-high resolution (2K or 4K) physically based rendering material maps, they typically only handle low-resolution inputs and rely on super-resolution techniques to upscale the generated image to a higher resolution. One existing method reconstructs relit facial geometry and textures using high-quality dome light field data and supports the generation of 4K resolution material maps. However, its original input and output material resolutions are limited to 512x512, which fails to fully utilize the rich information in the 4K input image. To achieve higher resolution outputs, this method upscales the results to 4K via a super-resolution network, but this process may introduce distortion of texture patterns. Furthermore, this method relies on expensive, high-quality dome light field data, further limiting its practicality and scalability for widespread applications.
[0005] In recent years, generative methods for obtaining ultra-high-resolution textures have emerged. For example, DreamFace (Progressive Generation of Animatable 3D Faces under Text Guidance) supports generating geometric and material information from text and images as input. DreamFace supports generating high-fidelity albedo, micronormal, and specular maps at up to 4K resolution using 4K input images. DreamFace employs a two-stage framework and utilizes a diffusion model to learn prior information from large-scale, high-quality datasets. This approach generates material maps that are consistent with the overall input and effectively preserve details. However, DreamFace's workflow does not generate 4K material maps end-to-end. Instead, it achieves the final 4K resolution by applying super-resolution techniques to an initial 512×512 output. This somewhat limits the accurate capture of details and the optimal expression of the generated effect. Although these methods introduce prior knowledge of ultra-high-resolution datasets into the super-resolution module, the detail representation of the output image remains relatively limited due to the limitations of input and output resolution. Therefore, these methods struggle to capture sufficiently fine texture details directly from ultra-high resolution input, resulting in inaccuracies or loss of detail in the generated material maps compared to the original input.
[0006] One reason existing methods don't directly use ultra-high resolution input is the enormous memory consumption of GPUs. Even processing 1K resolution images requires extremely high computational resources. If the resolution is scaled up to 4K, the required memory increases 16 times, making training costs extremely high and beyond the reach of many research teams. Therefore, due to computational resource constraints, many methods struggle to achieve direct processing from ultra-high resolution input to output. Summary of the Invention
[0007] In view of the above, the purpose of this invention is to provide a method and apparatus for generating rendering texture maps based on ultra-high resolution portraits, which effectively overcomes the bottleneck of existing technologies in processing ultra-high resolution facial material generation, and performs particularly well in capturing subtle facial features.
[0008] To achieve the above-mentioned objectives, an embodiment provides a method for generating rendering texture maps based on ultra-high resolution portraits, comprising the following steps: Global texture extraction: After downsampling the ultra-high resolution portrait, predict the global geometric feature map and global UV map. Hash the global UV map to obtain the features of geometric space and texture mapping. Combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract global texture. Local texture extraction: After dividing the ultra-high resolution portrait into small portrait blocks, the local texture of each small portrait block is extracted by combining the corresponding position in the global geometric feature map and the global UV map. The global texture is introduced as a guide in the local texture extraction. After post-processing, the local textures of all small portrait blocks are mapped onto the geometry to obtain the final rendered texture map.
[0009] Preferably, a multi-resolution hash table with multiple trainable textures is used to hash-encode the global UV map to obtain features of the geometric space and texture mapping. The loss function gradient from the global distribution extractor that extracts the global texture is used to optimize the features of the geometric space and texture mapping, as follows: ; in, and Let represent the features of the geometric space and texture mapping in round k and round (k+1), respectively. Indicates spread to The gradient of the loss function of the global distribution extractor.
[0010] Preferably, a global distribution extractor is used to extract global texture based on features of geometric space and texture mapping, downsampled ultra-high resolution portrait, global geometric feature map and global UV map. The global distribution extractor is optimized before being applied. The loss function used for parameter optimization includes a loss constructed based on the distribution difference between the extracted global texture and the real global texture.
[0011] Preferably, combining each portrait patch with its corresponding location in the global geometric feature map and the global UV map includes: After upsampling the global geometric feature map and global UV map to the ultra-high resolution human portrait size, they are divided into small geometric feature blocks and UV blocks. Human portrait blocks, geometric feature blocks and UV blocks in the same position are then merged and combined.
[0012] Preferably, a high-frequency detail optimizer based on a neural network is used to extract the local texture of each portrait patch, and global texture is introduced as a guide in the local texture extraction, including: Features are extracted through multiple neural network layers. The global texture is upsampled to the size of an ultra-high resolution human portrait and then divided into blocks to obtain small global texture patches. These small global texture patches are then stitched onto a later feature layer to guide texture extraction. The stitched features are then used to extract local textures through subsequent neural network layers. The high-frequency detail optimizer is optimized before being applied. The loss function used for parameter optimization includes a loss constructed based on the distribution difference between the extracted local texture and the real local texture.
[0013] Preferably, the loss function includes L1 loss and Laplace loss.
[0014] Preferably, the local textures of all portrait patches are post-processed, including: The local textures of all portrait patches are assembled, and anisotropic Gaussian filtering is applied to the gap areas of the assembly to achieve gap smoothing.
[0015] To achieve the above-mentioned objectives, embodiments of the present invention also provide a rendering texture mapping generation device based on ultra-high resolution portraits, comprising: The global texture extraction module is used to predict the global geometric feature map and global UV map after downsampling the ultra-high resolution portrait. The global UV map is hashed to obtain the features of geometric space and texture mapping. The features of geometric space and texture mapping are combined with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract the global texture. The local texture extraction module is used to divide the ultra-high resolution portrait into small portrait blocks, combine each small portrait block with the corresponding position in the global geometric feature map and global UV map to extract the local texture of each small portrait block, introduce the global texture as a guide in the local texture extraction, and map the local texture of all small portrait blocks onto the geometry after post-processing to obtain the final rendered texture map.
[0016] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned method for generating rendering texture maps based on ultra-high resolution portraits.
[0017] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-mentioned method for generating rendering texture maps based on ultra-high resolution portraits.
[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention employs an end-to-end framework for directly generating ultra-high-definition textures from ultra-high-resolution portraits, avoiding texture distortion issues that may arise from intermediate super-resolution techniques. By dividing the ultra-high-resolution portrait into blocks using a divide-and-conquer strategy, it efficiently processes the ultra-high-resolution portrait input. A global texture extraction method ensures consistency when extracting local fine details from individual batches. Furthermore, a geometric normal prior is applied to handle non-planar inputs, maximizing the preservation of high-frequency details in the portrait. This ensures that the generated texture not only matches the input portrait but also exhibits superior detail preservation, guaranteeing the robustness of the texture generation process. This invention also introduces a facial hashing encoding method to enhance the ability to adaptively learn global and local texture distributions, thereby improving the detail extraction effect. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart of the rendering texture map generation method based on ultra-high resolution portrait provided in the embodiment; Figure 2 This is a flowchart of the rendering texture map generation method based on ultra-high resolution portrait provided in the embodiment; Figure 3 This is a comparison diagram of micronormals in global and local textures provided in the embodiment; Figure 4 This is a schematic diagram of the structure of the rendering texture mapping generation device based on ultra-high resolution human portrait provided in the embodiment. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0022] The inventive concept of this invention is as follows: Existing monocular high-resolution face texture reconstruction methods cannot directly and fully utilize 4K ultra-high resolution input images. Due to limitations in input and output resolution, the detail representation of the output image remains relatively limited. Therefore, these methods struggle to capture sufficiently fine texture details directly from ultra-high resolution input, potentially leading to inaccuracies or loss of detail in the generated texture maps compared to the original input. Furthermore, directly processing 4K and other ultra-high resolution input images results in significant GPU memory consumption, limiting the computational resources available for these methods. To fully utilize the rich information of ultra-high resolution input images while avoiding the GPU challenges posed by directly processing them, this invention provides a rendering texture map generation method and apparatus based on ultra-high resolution portraits. This method employs a divide-and-conquer strategy, dividing the ultra-high resolution input image into several small blocks for separate processing. However, this divide-and-conquer strategy faces three main challenges: 1) Different facial regions have different texture features, and segmented images cannot guarantee that corresponding patches from different faces will align to the same facial region; 2) Generating each patch individually cannot ensure the continuity and seamless stitching of textures at the boundaries; 3) Similar high-resolution texture reconstruction methods usually assume that the input geometry is planar, but the complex facial geometry cannot satisfy this assumption.
[0023] To further address these issues, this invention introduces a global texture extraction stage (Stage 1), using downsampled images to determine the global distribution of textures and providing guidance for the local texture extraction stage (Stage 2). Furthermore, a novel facial UV hashing encoding method is employed in Stage 1 to better capture regional information in the generated global texture. This not only ensures global consistency for Stage 2 in Stage 1 but also outputs rich, continuous details, further facilitating the finer processing in Stage 2. Finally, prior knowledge of geometric normals is incorporated to ensure that local texture generation is independent of global geometry, thus resolving the dependency on approximate plane assumptions found in similar methods.
[0024] like Figure 1 and Figure 2 As shown in the embodiment, a method for generating rendering texture maps based on ultra-high resolution portraits includes the following steps: S1, Global Texture Extraction: After downsampling the ultra-high resolution portrait, predict the global geometric feature map and global UV map. Hash the global UV map to obtain the features of geometric space and texture mapping. Combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract global texture.
[0025] In the global texture extraction stage of this embodiment, a guiding global texture is extracted from the ultra-high-resolution portrait. Specifically, the ultra-high-resolution portrait (e.g., 4K (4096×4096 pixels)) is downsampled (e.g., downsampled to 1024×1024 pixels) and input into the predictor to generate a global geometric feature map and a global UV map. Notably, the generated global geometric feature map includes a global normal map. The generated global UV map is used for hash encoding of the facial region. The encoded channel features are combined with the downsampled ultra-high-resolution portrait, the global geometric feature map, and the global UV map, and then processed by the Global Distribution Extractor (GDE) to generate a 1024×1024 global texture.
[0026] In this embodiment, the goal of texture asset reconstruction is to reconstruct the diffuse, micronormal, and specular maps wrapped around the geometric surface, collectively referred to as texture maps. Direct reconstruction from image space input to texture space requires the assumption that the object itself is planar or approximately planar. However, the lack of prior knowledge about global geometric normals can negatively impact the extraction of local micronormals, especially when extracting micronormals on small blocks due to the limitation of the receptive field. Without guidance from global normals in the geometry, the network may incorrectly merge global geometric normals with local micronormals.
[0027] Therefore, this invention pre-trains a predictor during the global texture extraction stage. The predictor can employ a neural network, such as the U-Net network. The predictor uses the global geometric feature map and global UV map based on the downsampled ultra-high resolution portrait, and uses the obtained global UV map and global geometric feature map as auxiliary inputs to the global distribution extractor (GDE) and high-fidelity detail extractor (HFDR). With the help of the global geometric feature map, the focus can be effectively concentrated on learning the micronormal distribution in the texture space, which can be expressed as: ; in, It is a global geometric feature map obtained through a predictor. Represents the global UV map. This represents the original ultra-high resolution portrait. The model represents the square of the distance between the two. This indicates that the predicted micronormal is based on the input within the brackets. It is a constant, indicating that it depends on the original ultra-high resolution portrait. At that time, the model converged to The learning difficulty is reduced significantly. This indicates that the predicted global geometric feature map facilitates the model's learning of subtle and high-frequency features in the micronormal distribution. Simultaneously, the additional geometric normal prior alleviates the relighting process stress on the model when predicting diffuse and specular maps. The method of this invention is less affected by geometry self-shadowing, resulting in cleaner and more refined diffuse maps.
[0028] In facial texture reconstruction tasks, employing an effective spatial encoding method is crucial because different facial regions exhibit a strong correlation with their corresponding texture distributions. Appropriate encoding can enhance the network's ability to learn facial detail distributions, especially in areas such as the lips and corners of the eyes, where the texture and color often differ significantly from other parts of the face. This encoding method effectively improves the model's accuracy in capturing local details, thereby ensuring the realism and accuracy of facial reconstruction.
[0029] In existing technologies, facial segmentation networks are typically used to segment faces, and the masks predicted from the decomposition results are used as prior information to assist input data in texture generation, or facial feature points are used as prior information to achieve the same purpose. Although these methods are simple to implement and can utilize prior information learned from large datasets of facial images, their segmentation masks or facial feature points only provide very coarse spatial encoding. For example, a portion of the mask encodes the entire region using a single ID, and the feature points only provide the relative positions of parts such as the eyes and nose, lacking a more fine-grained description. This may reduce the network's ability to adaptively capture subtle details.
[0030] In order to further enhance the ability of the Global Distribution Extractor (GDE) to adaptively learn subtle differences in texture distribution in different regions of the face, this embodiment of the invention applies hash encoding to the global UV map predicted by the predictor. The predictor generates a global UV map composed of two-dimensional coordinates of the face in a unified UV space. These two-dimensional coordinates are used as input to the hash encoding, which enables the network to better capture the fine texture details of each region of the face.
[0031] The hash encoding consists of a multi-layered, trainable textured, multi-resolution hash table. The encoded geometric space and texture mapping features are optimized using stochastic gradient descent. This compact representation helps the network capture complex spatial patterns and fine-grained details in high-resolution data. Specifically, the number of learnable hash texture layers can be defined as follows: Each layer contains There are 10 features, therefore after hash encoding, the final output channel number is 10. Channels. During training, it was observed that 8-channel hash encoding was sufficient, as increasing the number of channels did not bring significant performance improvements but instead slowed down the training speed.
[0032] In this embodiment, the hash encoding and the Global Distributed Extractor (GDE) network are jointly trained. That is, when the global UV map is hash-encoded using a multi-resolution hash table with multiple layers of trainable textures to obtain the features of the geometric space and texture mapping, the gradients from the GDE can optimize the features of the geometric space and texture mapping. This can be described as follows: ; in, and Let represent the features of the geometric space and texture mapping in round k and round (k+1), respectively. Indicates spread to The loss function gradient of the global distribution extractor enables the learning of prior information closely related to the facial geometric UV distribution.
[0033] For ultra-high resolution portraits (e.g., 4096x4096 pixels), direct input into the network is impractical due to current GPU memory limitations, making training impossible. Therefore, dividing the portrait into small patches for local prediction has become a common practice. However, restricting the network's input to a single patch leads to a significant loss of global contextual information, thus limiting the receptive field and making it difficult to capture a broader texture distribution. The lack of constraints on the texture value distribution between neighboring patches causes the average distribution of each patch to fluctuate independently. This inconsistency often produces noticeable blocky artifacts and mosaic effects between adjacent patches. These problems typically arise when the network relies solely on local information and fails to incorporate global context.
[0034] To enhance the High Frequency Detail Optimizer (HFDR)'s ability to capture long-range structural dependencies, this invention introduces an additional module, the Global Distribution Extractor (GDE), designed to capture global long-range contextual features. However, directly using RGB images as input to GDE or HFDR conflicts with near-plane prior assumptions. To address this issue, we use the global geometric feature map and global UV map output from our predictor as auxiliary inputs, providing additional global geometric distribution information to ensure that both GDE and HFDR can better capture subtle high-frequency detail context of the face.
[0035] In this embodiment, GDE adopts a 12-layer UNet-like backbone architecture. The specific process of the global texture extraction stage is as follows: The ultra-high resolution portrait (4096×4096 pixels) was downsampled to a resolution of 1024×1024 pixels to obtain the image. Then, use the predictor To predict the corresponding global geometric feature map and global UV map .
[0036] ; Using a multi-resolution hash table based on a global UV graph Generate 8-channel geometry-texture mapping features. These geometry-texture mapping features are compared with the downsampled image. Predicted global UV map and global geometric feature map The concatenated data is then used as input data for GDE. .
[0037] ; Input data The data is fed into the GDE network to generate a 7-channel global texture map. : ; in , Represents diffuse reflection diagram, Represents the micronormal plot. This represents a highlight image.
[0038] The Global Distribution Extractor (GDE) underwent parameter optimization before being used. The loss function used for parameter optimization included a loss constructed based on the distribution difference between the extracted global texture and the real global texture, specifically the L1 loss between the extracted global texture and the real global texture. and Laplace's loss The weighted combination of L1 and L2 loss enhances the network's ability to perceive and learn high-frequency textures, which is especially crucial for capturing skin pores, wrinkles, and other subtle features that are often missed by standard L1 loss alone. Its loss function... The formula is as follows: ; Among them, weight Laplace loss Computation is performed by convolving with a Laplacian kernel to perceive sharp discontinuities in the space, thereby encouraging the network to learn in high-frequency distributions. Laplacian kernel The definition can be: ; Given the Laplace kernel The Laplace loss is calculated as follows:
[0039] All losses are calculated across all channels.
[0040] It is worth noting that the GDE was trained independently, rather than jointly with the HFDR in a cascaded manner. Experiments showed that joint training in a cascaded setting caused the GDE to be continuously affected by local receptive field gradients from the HFDR. This interference hindered the GDE's ability to focus on the global context, ultimately reducing its ability to effectively guide the HFDR.
[0041] S2, Local Texture Extraction: After dividing the ultra-high resolution portrait into small portrait blocks, the local texture of each small portrait block is extracted by combining the corresponding position in the global geometric feature map and the global UV map. The global texture is introduced as a guide in the local texture extraction. After post-processing, the local textures of all small portrait blocks are mapped onto the geometry to obtain the final rendered texture map.
[0042] In this embodiment, to obtain ultra-high resolution (UHR) textures with rich local details, it is necessary to predict highly detailed textures at the original resolution level, thereby maximizing the utilization of detail information in ultra-high resolution portraits. During the local texture extraction stage, each batch is processed separately, while the global texture from the global texture extraction stage (stage one) is used as a guide to ensure overall consistency.
[0043] In this embodiment, the ultra-high resolution portrait (e.g., 4K (4096×4096 pixels)) is divided into multiple (e.g., 8×8) portrait blocks, each with a size of 512x512 pixels. A High Frequency Details Optimizer (HFDR) is used to extract the local texture of each portrait block. This includes combining each portrait block with its corresponding position in the global geometric feature map and global UV map, and then inputting the result into the HFDR. Specifically, the global geometric feature map (e.g., 1024×1024 pixels) and global UV map (e.g., 1024×1024 pixels) are upsampled to the ultra-high resolution portrait size (e.g., 4096×4096 pixels) and then divided into multiple (e.g., 8×8) geometric feature blocks and UV blocks according to the same block division method. Each geometric feature block and UV block is also 512x512 pixels. Then, the portrait blocks, geometric feature blocks, and UV blocks at the same position are merged and combined, and then input into the HFDR for calculation to extract the local texture of each portrait block.
[0044] When extracting local textures for each portrait patch, HFDR also introduces global textures as a guide. This global texture allows HFDR to more efficiently focus on capturing high-granularity, high-frequency local features within the portrait patch. Similar to GDE, HFDR adopts a UNet-like backbone structure, extracting features through multiple neural network layers. The global texture is upsampled to an ultra-high resolution portrait size (e.g., 4096×4096 pixels) and then divided into multiple (e.g., 8×8) global texture patches according to the portrait patch segmentation method. Each global texture patch is 512x512 pixels in size. These global texture patches are then concatenated to a later feature layer (e.g., a 17-layer neural network feature layer) as texture extraction guidance. The concatenated features continue to be used by subsequent neural network layers to extract local textures, which also include high-quality albedo maps, micronormal maps, and specular maps. This significantly reduces the burden on HFDR in predicting the overall texture distribution. This also introduces long-distance contextual information into HFDR, effectively alleviating the problem of discontinuity between adjacent patch distributions.
[0045] In this embodiment, the HFDR is optimized before application to enable it to learn high-frequency detail textures more effectively. The loss function used for parameter optimization includes a loss constructed based on the distribution difference between the extracted local texture and the real local texture, specifically the L1 loss between the extracted local texture and the real local texture. and Laplace's loss The weighted combination formula is: ; Among them, weight and The ratio is 1:2.
[0046] In this embodiment, after obtaining the local texture, post-processing is also performed, specifically including assembling the local textures of all portrait patches, for example, assembling 64 local textures with a size of 512x512 into a whole texture of 4096x4096 pixels. The specific assembly process is as follows: ; in, Indicates coordinate position Texture value, represents the modulo operation, and P represents the texture calculation operation.
[0047] With the help of global texture guidance from GDE, the discontinuity between local textures corresponding to different portrait patches in the output is significantly reduced. However, due to resolution limitations, GDE cannot capture high-frequency fine-grained details, so there may still be obvious gaps at the boundaries of patches, resulting in patterns that cannot be perfectly matched.
[0048] To address this issue, this invention applies anisotropic Gaussian filtering to both horizontal and vertical slit regions, achieving smooth slit processing while preserving some local high-frequency details. Specifically, adjacent slit regions are processed with a fill size of 8 pixels. For horizontal slits, a convolution kernel is used, defined as x=2 and y=16; for vertical slits, another convolution kernel is used, defined as x=16 and y=2. The standard deviation of both convolution kernels is σ=5. Gaussian filtering convolution is performed using this kernel, and results show that this method effectively reduces the impact of slit artifacts.
[0049] After obtaining the overall texture, it is transformed from image space to the corresponding UV space of the specific geometry. The overall texture is then unwrapped in the UV space of the geometry using NvDiffrast to optimize the texture, and blank areas are repaired accordingly. This process is achieved by minimizing the loss between the predicted image space texture and the texture projected (or rasterized) into the UV space; essentially, the predicted overall image space texture is unfolded onto the corresponding UV coordinates. To capture the highest possible fidelity prediction output, a cascaded texture optimization strategy is employed. First, a maximum texture resolution is set, for example, 8192×8192. Then, the maximum texture resolution is downsampled, for example, to 4096×4096. Finally, a predefined texture is used to interpolate the generated UV space texture. The interpolation weights are determined based on the average color of the facial region.
[0050] The example also compares the global texture output by GDE and the local texture output by HFDR. The comparison results of the micronormals in the textures are as follows: Figure 3 As shown, experimental results demonstrate that the method of this invention significantly improves texture reconstruction quality. This method is independent of geometry reconstruction and easily integrated into other physically based rendering (PBR) texture reconstruction workflows. It leverages advancements in mobile device camera technology.
[0051] like Figure 4As shown, the embodiment also provides a rendering texture map generation device 40 based on ultra-high resolution portraits, including a global texture extraction module 41 and a local texture extraction module 42. The global texture extraction module 41 is used to predict the global geometric feature map and global UV map after downsampling the ultra-high resolution portrait, hash-encode the global UV map to obtain the features of geometric space and texture mapping, and combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract the global texture. The local texture extraction module 42 is used to divide the ultra-high resolution portrait into portrait blocks, combine each portrait block with the corresponding position in the global geometric feature map and global UV map to extract the local texture of each portrait block, introduce the global texture as guidance in the local texture extraction, and map the local texture of all portrait blocks onto the geometry after post-processing to obtain the final rendering texture map.
[0052] It should be noted that the rendering texture mapping method and apparatus based on ultra-high resolution portraits provided in the above embodiments should be illustrated by the division of the above functional modules when generating rendering texture maps. The functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the rendering texture mapping apparatus based on ultra-high resolution portraits provided in the above embodiments and the rendering texture mapping method embodiments based on ultra-high resolution portraits belong to the same concept. For details of its implementation process, please refer to the rendering texture mapping method embodiments based on ultra-high resolution portraits, which will not be repeated here.
[0053] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described method for generating rendering texture maps based on ultra-high resolution portraits, specifically including the following steps: S1, Global Texture Extraction: After downsampling the ultra-high resolution portrait, predict the global geometric feature map and global UV map. Hash the global UV map to obtain the features of geometric space and texture mapping. Combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract global texture. S2, Local Texture Extraction: After dividing the ultra-high resolution portrait into small portrait blocks, the local texture of each small portrait block is extracted by combining the corresponding position in the global geometric feature map and the global UV map. The global texture is introduced as a guide in the local texture extraction. After post-processing, the local textures of all small portrait blocks are mapped onto the geometry to obtain the final rendered texture map.
[0054] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the ultra-high resolution portrait rendering texture mapping generation method described in S1-S2 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0055] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described method for generating rendering texture maps based on ultra-high resolution portraits, specifically including the following steps: S1, Global Texture Extraction: After downsampling the ultra-high resolution portrait, predict the global geometric feature map and global UV map. Hash the global UV map to obtain the features of geometric space and texture mapping. Combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, global geometric feature map and global UV map to extract global texture. S2, Local Texture Extraction: After dividing the ultra-high resolution portrait into small portrait blocks, the local texture of each small portrait block is extracted by combining the corresponding position in the global geometric feature map and the global UV map. The global texture is introduced as a guide in the local texture extraction. After post-processing, the local textures of all small portrait blocks are mapped onto the geometry to obtain the final rendered texture map.
[0056] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.
[0057] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for generating a rendering texture map based on an ultra-high resolution portrait, characterized in that: The following steps are involved: Global texture extraction: After downsampling the ultra-high-resolution portrait, the global geometric feature map and global UV map are predicted, and the global UV map is hash-encoded to obtain the features of geometric space and texture mapping. The features of geometric space and texture mapping are combined with the downsampled ultra-high-resolution portrait, global geometric feature map and global UV map to extract the global texture. Local texture extraction: After dividing the ultra-high-resolution portrait into small portrait blocks, each portrait block is combined with the corresponding position in the global geometric feature map and the global UV map to extract the local texture of each portrait block. The global texture is introduced as a guide in the local texture extraction, and the local textures of all portrait blocks are post-processed and mapped to the geometric body to obtain the final rendered texture map.
2. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 1, characterized in that: The global UV map is hash-encoded using a multi-resolution hash table of a multi-layer trainable texture to obtain the features of the geometric space and texture mapping. The features of the geometric space and texture mapping are optimized using the loss function gradient of the global distribution extractor that extracts the global texture, which is expressed as: ; in, and Represent the features of geometric space and texture mapping of the kth round and the k+1th round respectively, Indicates that it spreads to The gradient of the loss function of the global distribution extractor.
3. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 1, characterized in that: A global distribution extractor is used to extract global texture based on the features of geometric space and texture mapping, the downsampled ultra-high resolution portrait, the global geometric feature map and the global UV map. The global distribution extractor is parameter optimized before application. The loss function used in the parameter optimization includes the loss constructed based on the distribution difference between the extracted global texture and the true global texture.
4. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 1, characterized in that: Combine each portrait patch with the corresponding position in the global geometry map and the global UV map, including: The global geometric feature map and the global UV map are upsampled to the ultra-high-resolution portrait size and then divided into small geometric feature blocks and small UV blocks. The portrait blocks, geometric feature blocks and UV blocks in the same position are merged.
5. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 1, characterized in that: A high-frequency detail optimizer based on a neural network is used to extract the local texture of each small block of the portrait, and the global texture is introduced as a guide in the local texture extraction, including: Through multiple neural network layers to extract features, the global texture is upsampled to the ultra-high resolution portrait size and then divided into small global texture blocks. The small global texture blocks are then spliced to a later layer of features for texture extraction guidance. The spliced features are then passed through the following neural network layers to extract local textures. The high-frequency detail optimizer is parameter optimized before being applied, and the loss function used in the parameter optimization includes a loss constructed based on the distribution difference between the extracted local texture and the true local texture.
6. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 3 or 5, characterized in that: The loss functions include LI loss and Laplace loss.
7. The method for generating a rendering texture map based on an ultra-high resolution portrait according to claim 1, characterized in that: All local textures of small portrait blocks are post-processed, including: The local textures of all small portrait blocks are assembled, and anisotropic Gaussian filtering is applied to the assembled gap areas to achieve gap smoothing.
8. A rendering texture map generation device based on ultra-high resolution portrait, characterized in that: include: A global texture extraction module is used to predict a global geometric feature map and a global UV map after downsampling the ultra-high resolution portrait, perform hash coding on the global UV map to obtain features of geometric space and texture mapping, and combine the features of geometric space and texture mapping with the downsampled ultra-high resolution portrait, the global geometric feature map and the global UV map to extract global texture; The local texture extraction module is used to divide the ultra-high-resolution portrait into small portrait blocks, combine each small portrait block with the corresponding position in the global geometric feature map and the global UV map, extract the local texture of each small portrait block, introduce the global texture as a guide in the local texture extraction, and map the local textures of all small portrait blocks to the geometric body after post-processing to obtain the final rendered texture map.
9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the method for generating a rendering texture map based on an ultra-high-resolution portrait according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the rendering texture map generation method based on an ultra-high resolution portrait described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Face super-resolution reconstruction device and method based on face 3D prior information
CN116703719A
Ultrahigh-definition image enhancement method and system based on non-uniform sampling
CN117196988A
Single-image detail three-dimensional face reconstruction method, system, equipment and medium
CN117557721A
High-precision dynamic face reconstruction and high-resolution texture generation method and system
CN118736087A
Neural rendering method based on multi-resolution network structure
WO2023225891A1
