A method for large-scale terrain landscape generation using differentiable rendering and texture layering

Abstract

The application discloses a large-scale terrain landscape generation method using differentiable rendering and texture layering, and steps specifically include: 1) using a differentiable rendering technique to derive a reverse analysis algorithm to decouple the shadow in the texture, thereby inversely solving the layered terrain texture weight map under the condition of a given terrain map and a source texture; 2) using the large-scale terrain weight map in a modern engine to generate a terrain landscape with high definition and rich details. The application constructs a large-scale, visually realistic virtual terrain for a wide range of multimedia applications, and the method is efficient and effective in constructing a large-scale and richly detailed virtual terrain, and is suitable for a wide range of multimedia applications.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, specifically to a method for generating large-scale terrain landscapes using differentiable rendering and texture layering to obtain layered terrain texture weight maps and then generate large-scale terrain landscapes. Background Technology

[0002] Generated terrain landscapes, as a major component of virtual environments, are widely used in various application fields such as film, video games, and geological analysis. In recent years, terrain modeling techniques and manual tools have made significant progress. However, generating large-scale terrain that conforms to the aesthetics of real-world terrain remains a huge challenge because the visual and ecological styles (such as grasslands and rocks) formed over millions of years of geological evolution are difficult to automatically synthesize or manually render, even in modern engines. To enable free roaming adventures in virtual environments, the need for high-resolution textures is urgent, providing more surface detail when viewed up close. If the terrain texture is the same size as the model, the terrain within the field of view will appear monochrome, severely impacting immersion, because in modern engines (such as Unreal and Unity), each rendering unit corresponds to only one pixel. However, even with textures of significantly higher resolution, the number of pixels per rendering unit is still limited, failing to provide sufficient visual detail.

[0003] Several pioneering studies have proposed methods for decomposing and storing textures, storing high-resolution textures for common terrain elements (such as rocks, grass, soil, and gravel) and recording their locations for real-time loading and rendering. Weight maps are a widely used technique in this field. In this technique, the high-resolution textures used to record elements such as rocks and grass are called source textures, while the weight map represents the proportion of the source texture on each vertex of the model. A weight map of a specific source texture can then be used to record the weights of all vertices in the model. However, currently, there is no technique that can reasonably and efficiently inversely calculate weight maps for terrain elements, and there is also a lack of methods for applying weight maps representing different terrain elements to terrain landscape generation. Invention content

[0004] The purpose of this invention is to provide a method for generating large-scale terrain landscapes using differentiable rendering and texture layering, and a multi-layer texture mapping algorithm, which can obtain a layered terrain texture weight map, thereby realizing the generation of large-scale terrain landscapes, and providing users with a virtual geographic environment with rich details and less storage burden.

[0005] This invention introduces a texture layering technique. The purpose of this invention is to generate terrain weight maps to guide texture blending, rather than directly generating extremely large-scale textures. The main problems addressed by this invention include: 1) the lack of standard templates for terrain texture layering, and the absence of universal data for synthesizing real-world terrain weight maps. 2) compared to texture synthesis, multi-layer weight maps with different texture templates need to conform to inherent correlations and constraints related to terrain geometry and appearance. 3) how to simultaneously stitch adjacent weight maps under limited resources and apply weight map and texture artifacts to support terrain of arbitrary size.

[0006] The objective of this invention is achieved as follows:

[0007] A method for generating large-scale terrain landscapes utilizes differentiable rendering and texture layering. It re-illuminates satellite images using differentiable rendering to decouple the influence of shadows on terrain textures and optimizes the weight map corresponding to a specific source texture. Through a process of inverse weight map calculation, multi-layer texture mapping is achieved to obtain large-scale terrain textures with rich detail and strong scalability for generating large-scale terrain landscapes. Specifically, the method includes:

[0008] Step 1: Weight graph optimization, specifically including:

[0009] 1.1: Obtain topographic maps and satellite images with the same center coordinate matching and scaling level as the targets for weighted map optimization;

[0010] 1.2: Select sample textures representing different land surface distributions from the satellite imagery;

[0011] 1.3: The average color value of the sampled texture is used as the source texture to provide color components for weight map optimization;

[0012] 1.4: Construct a differentiable renderer to perform weight graph inversion;

[0013] 1.5: Collect several sets of weighted maps corresponding to different landforms;

[0014] Step 2: Large-scale terrain and landscape generation, specifically including:

[0015] 2.1: Obtain a set of weighted maps corresponding to large-scale terrain through texture stitching;

[0016] 2.2: Select a multi-layer high-resolution texture map corresponding to the source texture;

[0017] 2.3: Import the terrain map into a modern engine such as Unreal or Unity to generate the corresponding large-scale terrain;

[0018] 2.4: Import the weight map as the basis for blending multi-layer high-definition texture maps into the modern engine, generate high-definition blended textures in the modern engine, and obtain a 3D textured terrain landscape with rich details through the blended textures.

[0019] Specifically, step 1.4 involves constructing a differentiable renderer and performing weight graph inversion. The detailed process is as follows:

[0020] 1.41: Settings for Differentiable Renderers

[0021] i) Model the terrain, setting corresponding vertices for each pixel in the terrain map based on the renderer;

[0022] ii) By weighting and blending the source texture and combining it with lighting conditions, a texture map of the corresponding terrain is obtained;

[0023] 1.42: Weighted Graph Inverse Calculation

[0024] i) Using a terrain differentiable renderer, lighting parameters are identified by optimizing the shadow distribution within satellite images to obtain the restored shadowless terrain texture;

[0025] ii) Optimize the weight map from satellite imagery in an end-to-end manner using the following equation (I):

[0026]

[0027] Where M(M1,M2,M3,...,M) n W(W1,W2,W3,...,W) represents the provided source texture. n () represents the weight map to be optimized, T represents the terrain, L represents the lighting conditions, and S represents the user-provided satellite image. Let represent the synthesized texture, where i represents any one of the n source textures and weights in the weight map, R() represents the differentiable rendering process, and ||......||1 represents the calculation process of the mean absolute error. This represents the process of iteratively optimizing the weight map W by minimizing the mean absolute error.

[0028] iii) Relighting: In each optimization run, randomly change the lighting direction to obtain different shadow distributions under different lighting directions for the same terrain; ensure that all textures can be fully illuminated; the optimization process for the weight map using relighting is as shown in equation (II) below:

[0029]

[0030] Among them, S rThis represents a user-provided satellite image with shadows removed, where θ and φ represent the zenith angle and azimuth angle, and l represents the illumination parameters other than the direction of illumination.

[0031] iv) Introduce a style loss function, the purpose of which is to converge the sum of the weights of each weight map on any pixel to 1; during the optimization process, negative values ​​appear, and ReLU is used to activate W to remove negative values, and the activated weight map W is normalized before summation to ensure that it conforms to the correct weight map form; the specific optimization process is shown in the following equation (III):

[0032]

[0033] Step 2.2, selecting the multi-layer high-resolution texture map corresponding to the source texture, specifically includes:

[0034] i) Directly select the sample textures used in the weighted graph optimization process;

[0035] ii) Users can choose a texture map similar to the sample texture;

[0036] Of these, only one of the methods described in i) and ii) needs to be selected for operation.

[0037] This invention proposes an innovative method for large-scale terrain landscape generation. This method addresses the problems of cumbersome manual design and the inapplicability of existing modeling methods. Specifically, satellite imagery is treated as the result of terrain scene imaging, and a differentiable, rendering-based inverse optimization algorithm is used to infer the weight map of user-provided source textures on the satellite imagery. In this way, a weight map corresponding to the terrain geometry and appearance is obtained; that is, the method of this invention can automatically infer the weight map corresponding to the user-provided source texture from the satellite imagery. By adding source textures, users can improve the accuracy of the optimization algorithm and enrich the details of the synthesized textures. Furthermore, the weight map optimization process proposed in this invention can be extended by mining features of given terrain and satellite imagery to reduce the requirements for user control. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the weight graph optimization process of the present invention;

[0039] Figure 2 This is a schematic diagram of the differentiable renderer conditions of the present invention;

[0040] Figure 3 This is a schematic diagram of the large-scale terrain landscape generation process of the present invention;

[0041] Figure 4 This is a schematic diagram comparing the terrain and landscape of the present invention;

[0042] Figure 5 This is a schematic diagram showing the layered details of the present invention;

[0043] Figure 6 This is a schematic diagram illustrating the diverse user control of the present invention. Detailed Implementation

[0044] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0045] The method for generating large-scale terrain landscapes using differentiable rendering and texture layering, as described in this invention, specifically includes the following steps:

[0046] I. Weight Graph Optimization

[0047] See Figure 1 Weight graph optimization based on differentiable rendering models specifically includes:

[0048] 1) Obtain topographic maps and satellite images with the same center coordinate matching and scaling level as the targets for weighted map optimization;

[0049] 2) Select sample textures representing different land surface distributions from the satellite image;

[0050] 3) The average color value of the sampled texture is used as the source texture to provide color components for weight map optimization;

[0051] 4) Construct a differentiable renderer to perform weight graph inversion;

[0052] 5) Collect several sets of weighted maps corresponding to different landforms as training datasets for subsequent steps.

[0053] See Figure 2 The inverse process of constructing the differentiable renderer specifically includes:

[0054] a. Differentiable renderer settings

[0055] 1) Model the terrain, setting corresponding vertices for each pixel in the terrain map based on the renderer;

[0056] 2) By weighting and blending the source texture and combining it with lighting conditions, a satellite image of the corresponding terrain is obtained;

[0057] b. Weighted graph inverse calculation

[0058] 1) Using a terrain differentiable renderer, lighting parameters are identified by optimizing the shadow distribution within satellite images to obtain the restored shadowless terrain texture;

[0059] 2) The weight map is inferred from satellite imagery in an end-to-end manner, and the weight map is optimized using the following equation (I):

[0060]

[0061] Where M(M1,M2,M3,...,M) n W(W1,W2,W3,...,W) represents the provided source texture. n () represents the weight map to be optimized, T represents the terrain, L represents the lighting conditions, and S represents the user-provided satellite image. Let represent the synthesized texture, where i represents any one of the n source textures and weights in the weight map, R() represents the differentiable rendering process, and ||......||1 represents the calculation process of the mean absolute error. This represents the process of iteratively optimizing the weight map W by minimizing the mean absolute error.

[0062] Through experiments, this invention found that shadows in satellite images significantly affect the performance of weight map optimization. Therefore, this invention employs a relighting method. In each optimization run, the illumination direction is randomly changed to obtain different shadow distributions. This strategy ensures that all textures can be fully illuminated. Since there is a strong correlation between shadow distribution and terrain, exploring different shadow distributions facilitates the correspondence between the optimized weight map and the terrain. The optimization process for the weight map using relighting is shown in equation (II) below:

[0063]

[0064] Among them, S r This represents a user-provided satellite image with shadows removed, where θ and φ represent the zenith angle and azimuth angle, and l represents the illumination parameters other than the direction of illumination.

[0065] This invention also introduces a commonly used objective function, the purpose of which is to converge the sum of the weights of the weight maps on any pixel to 1. Since negative values ​​may appear during the optimization process, ReLU activation W is used to remove negative values, and the activated weight map W is normalized before summation to ensure it conforms to the correct weight map form. The specific optimization process is shown in equation (III) below:

[0066]

[0067] II. Generation of Large-Scale Topographic Landscapes

[0068] See Figure 3 The process for generating large-scale 3D terrain landscapes is as follows:

[0069] 1) Obtain a set of weighted maps corresponding to large-scale terrain by stitching together the weighted maps.

[0070] 2) The user selects a multi-layer high-definition texture map corresponding to the source texture. Sample textures from the weight map optimization process can be directly selected. Since the weight map represents the distribution of certain source texture elements on the terrain, this invention can also change the visual effect of the composite texture by replacing similar source textures.

[0071] 3) Import the terrain map into a modern engine (such as Unreal Engine or Unity) to generate the corresponding large-scale terrain.

[0072] 4) Import the weight map as the basis for blending multi-layer high-definition texture maps into the modern engine, and import multi-layer high-definition texture maps to generate a blended texture with high definition, thereby obtaining a 3D textured terrain landscape with rich details.

[0073] See Figure 4 Figure a on the left shows a terrain landscape constructed without using this invention, while Figure b on the right shows a terrain landscape constructed using this invention. Figure a uses a texture of the same size as the terrain map, while Figure b uses a weight map of the same size as the terrain map combined with an 8k source texture to model and render a 26km × 13km terrain landscape. The boxed area shows the result of a local magnification of the terrain landscape. The comparison shows that if the terrain texture is the same size as the model, the terrain within the field of view will appear monochrome, severely impacting immersion. This is because in modern engines (such as Unreal Engine and Unity), each rendering unit corresponds to only one pixel. However, even with textures of significantly higher resolution, the number of pixels within each rendering unit remains limited, failing to provide sufficient visual detail. A weight map combined with a specific source texture is an image recording the weights of all vertices of the model. When merged, these weight maps create a total weight map for the entire model. Due to the mechanism of the weight map, for a source texture with a resolution of m × m, as long as its resolution is high enough, even using a 256 × 256 weight map, a texture equivalent to 256m × 256m can be synthesized. This method significantly increases the realism of textures and reduces storage burden. A comparison of Figures a and b shows that the terrain landscape generated using this invention has richer and more realistic details.

[0074] See Figure 5 Figure a at the top shows the terrain landscape generated after weighted graph inversion using the three source textures on the right. Figure b at the bottom shows another result generated using the five source textures on the right. By adding source textures, users can improve the accuracy of the optimization algorithm and enrich the details of the synthesized textures. The comparison shows that Figure a, generated with more source textures, has richer details.

[0075] See Figure 6Figures a, b, c, and d show four different terrain landscapes generated by users selecting different texture maps. This invention uses the average color of the textures instead of the original textures for weight map optimization, and guides the generation of terrain landscapes through this weight map. Compared to satellite imagery, this invention can decouple shadows and change the visual effect of composite textures by replacing similar source textures, thus achieving diverse user control. The four images illustrate how this invention changes the visual effect of composite textures by replacing similar source textures.

[0076] This invention proposes a method for generating large-scale terrain landscapes using differentiable rendering and texture layering. This allows users to access virtual geographic environments with rich detail and reduced storage requirements.

[0077] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for generating large-scale terrain landscapes using differentiable rendering and texture layering, characterized in that, Specifically, the following steps are included: Step 1: Weight graph optimization, specifically including: 1-1: Obtain topographic maps and satellite images with matching center coordinates and consistent scaling levels as the target for weight map optimization; 1-2: Select sample textures representing different land surface distributions from satellite images; 1-3: The average color value of the sampled texture is used as the source texture to provide color components for weight map optimization; 1-4: Construct a differentiable renderer to perform weight graph inversion; 1-5: Collect several sets of weighted maps corresponding to different landforms; Step 2: Large-scale terrain and landscape generation, specifically including: 2-1: Obtain a set of weighted maps corresponding to large-scale terrain through texture stitching; 2-2: Select a multi-layer high-resolution texture map that corresponds to the source texture; 2-3: Import the terrain map into a modern engine such as Unreal or Unity to generate the corresponding large-scale terrain; 2-4: The weight map is imported into a modern engine as the basis for blending multi-layer high-definition texture maps. The blended textures are then used to generate high-resolution textures, resulting in richly detailed 3D textured terrain landscapes. Among these: Steps 1-4 describe the construction of a differentiable renderer and the inverse calculation of the weight graph. The specific process is as follows: 1-41: Settings for Differentiable Renderers i) Model the terrain, setting corresponding vertices for each pixel in the terrain map based on the renderer; ii) By weighting and blending the source texture and combining it with lighting conditions, a texture map of the corresponding terrain is obtained; 1-42: Inverse Calculation of Weight Graph i) Using a terrain differentiable renderer, lighting parameters are identified by optimizing the shadow distribution within satellite images to obtain the restored shadowless terrain texture; ii) Optimize the weight map from satellite imagery in an end-to-end manner using the following equation (Ⅰ): （Ⅰ）； in, ,..., () represents the provided source texture; ,..., () represents the weight map to be optimized; T represents the terrain; L represents the lighting conditions; S represents the satellite image provided by the user; R represents the synthesized texture, where i represents any one of the n source textures and weights in the weight graph; R() represents the differentiable rendering process. The calculation process of the mean absolute error; This represents the process of iteratively optimizing the weight map W by minimizing the mean absolute error. iii) Relighting: In each optimization run, randomly change the lighting direction to obtain different shadow distributions under different lighting directions for the same terrain; ensure that all textures can be fully illuminated; the optimization process for the weight map using relighting is shown in equation (II) below: （Ⅱ） in, This refers to user-provided satellite images with shadows removed. and denoted by zenith angle and azimuth angle, and l represents lighting parameters other than the direction of illumination. iv) Introducing a style loss function aims to converge the sum of the weights of each weight map on any pixel to 1; during the optimization process, negative values ​​appear, and ReLU is used to activate W to remove negative values, and the activated weight map W is normalized before summation to ensure that it conforms to the correct weight map form; the specific optimization process is shown in equation (Ⅲ) below: （Ⅲ）。 2. The method for generating large-scale terrain landscapes according to claim 1, characterized in that, Step 2-2, which involves selecting a multi-layer high-resolution texture map corresponding to the source texture, specifically includes: i) Directly select the sample texture used in the weighted graph optimization process; ii) Users can choose a texture map similar to the sample texture; Only one of i) and ii) is selected for operation.

Images