Method, device and equipment for generating large-scale terrain landscape based on differentiable rendering

Abstract

The application relates to a large-scale terrain landscape generation method, device and equipment based on a differentiable rendering. The method comprises the following steps: respectively preprocessing a plurality of terrain samples, a weight map corresponding to each terrain sample and a real weight map to obtain training samples; constructing a weight map synthesis model; the weight map synthesis model comprises a generator, a differentiable renderer and a discriminator; training the weight map synthesis model according to a first loss function constructed in advance, the training samples and a discrimination result output by the discriminator to obtain a trained generator; generating a multi-layer weight map of each preprocessed terrain in a large-scale terrain through the trained generator, splicing the multi-layer weight map to obtain a large-scale weight map corresponding to the large-scale terrain, and generating a large-scale terrain landscape according to the large-scale weight map. The method can make the weight map support terrains of any size, provide more concise and efficient artistic control for users while having rich visual details.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a method, apparatus and device for generating large-scale terrain landscapes based on differentiable rendering. Background Technology

[0002] Terrain texture modeling is frequently used in various applications such as video, games, and geological research. Existing terrain texture modeling techniques can be categorized into three types: manual modeling, procedural modeling, and deep learning-based methods. In manual methods, artists need to prepare a set of tiled source textures and use a brush to paint them onto the terrain based on the actual landscape. Therefore, texture processing for large-scale terrains is very time-consuming and requires tedious trial-and-error adjustments. To simplify this tedious manual process, procedural modeling parameterizes terrain textures based on geological factors such as terrain height and slope. However, with the diversification of texture styles, the rules become complex, making artistic control difficult for non-professionals. Another rapidly gaining technology is deep learning, which performs texture modeling by mining the correlation between ecological distribution and terrain in multimodal data such as digital elevation models and satellite imagery. However, even with high-resolution textures obtained through super-resolution, as the camera approaches the ground, more details need to be provided to generate high-quality virtual terrain, which current methods still struggle to achieve.

[0003] However, traditional deep learning-based methods cannot provide surface details that are closer to the observer's view with acceptable hardware overhead. How to stitch adjacent weight maps together under limited resources and apply the weight maps to terrains of arbitrary size without producing artifacts has become an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and device for generating large-scale terrain landscapes based on differentiable rendering to address the aforementioned technical problems.

[0005] A method for generating large-scale terrain landscapes based on differentiable rendering, the method comprising:

[0006] The training samples are obtained by preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the real weight map.

[0007] A weighted map synthesis model is constructed. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. The concatenated feature map is then upsampled to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map, respectively. The discriminator is used to output corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map.

[0008] The weighted graph synthesis model is trained based on the pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator to obtain a trained generator.

[0009] A multi-layer weighted map of each preprocessed terrain in a large-scale terrain is generated by a trained generator. The multi-layer weighted maps are then stitched together to obtain a large-scale weighted map corresponding to the large-scale terrain. Large-scale terrain landscape is then generated based on the large-scale weighted map.

[0010] In one embodiment, the weight map synthesis model further includes a style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map.

[0011] In one embodiment, the pre-constructed first loss function is further defined as follows:

[0012] L total =L adv +λ1L W +λ2L R +λ3L style ,

[0013] L W =||G W (T t W s )-W t ||1,

[0014]

[0015]

[0016] Among them, L total Let L be the first loss function corresponding to the weighted graph synthesis model. style Here is the style transfer loss function corresponding to the style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map. λ1, λ2, and λ3 are weighted hyperparameters, and G... WFor the generator model used in weighted graph synthesis, T t For the input terrain sample, W s For the input weight map, W t This is the true weighted graph, where ||·||1 represents the mean absolute error. M i Let i be the source texture corresponding to the weight map of layer i, where i ∈ [1, n] and n is the number of samples. W ti This is the true weight graph of the i-th layer. These are parameters for a differentiable renderer.

[0017] In one embodiment, the method further includes: acquiring several terrain samples; processing each terrain sample using a weighted graph inverse algorithm to obtain a weighted graph corresponding to each terrain sample; randomly cropping the weighted graph and the terrain sample at corresponding positions to obtain a weighted graph patch and a terrain sample patch of size 2k×2k; randomly cropping the weighted graph patch and the terrain sample patch at corresponding positions to obtain a preprocessed weighted graph and a preprocessed terrain sample of size k×k; obtaining a preprocessed true weighted graph based on the weighted graph patch; and obtaining training samples based on the preprocessed weighted graph, the preprocessed terrain sample, and the preprocessed true weighted graph.

[0018] In one embodiment, the generator further includes a first encoder, a second encoder, a feature splicing module, a residual network module, and a decoder; the residual network module includes multiple residual blocks.

[0019] In one embodiment, the method further includes: extracting features from the preprocessed terrain sample using the first encoder to obtain a terrain sample feature map; extracting features from the preprocessed weight map using the second encoder to obtain a weight feature map; stitching the terrain sample feature map and the weight feature map using the feature stitching module to obtain a stitched feature map; inputting the stitched feature map into a residual network module, and upsampling the output of the residual network module using the decoder to obtain a multi-layer weight map.

[0020] In one embodiment, the method further includes: stitching the multi-layer weight map to obtain four adjacent terrains and a stitched weight map corresponding to each adjacent terrain; obtaining a central terrain map and a central weight map based on the four adjacent terrains and their corresponding stitched weight maps; the central terrain map and the central weight map have the same size; the size of the central weight map is the same as the size of the multi-layer weight map; generating a multi-layer weight map corresponding to the central terrain map using the trained weight map synthesis model; inputting the central terrain map, the central weight map, and the multi-layer weight map corresponding to the central terrain map into a pre-trained weight map stitching model to obtain a hybrid mask; the weight map stitching model includes a mask generator; cropping the hybrid mask according to the size of the central weight map; using the cropped hybrid mask to blend the central weight map and the multi-layer weight map corresponding to the central terrain map to obtain a seamless stitched weight map; and obtaining a large-scale weight map corresponding to a large-scale terrain based on the seamless stitched weight map of every four adjacent terrains.

[0021] In one embodiment, the seamless stitching weight map is further defined as follows:

[0022] y′=sm′+t(1-m′),

[0023] m′=(1-c)m+c,

[0024] m = G m (T x ,s,t),

[0025] Where y′ is the seamlessly stitched weight map, s is the center weight map, m′ is the cropped blend mask, t is the multi-layer weight map corresponding to the center topographic map, m is the blend mask, c is the elliptical mask, and G m (·) represents the mask generator, T x Topographic map centered on the topography.

[0026] In one embodiment, the method further includes: training the weighted graph stitching model according to a second loss function to obtain a trained weighted graph stitching model; the second loss function is:

[0027] L′ total =L Madv +λ4L R′ +λ5L M ,

[0028]

[0029]

[0030] L M =||G m (T x,s,t)-c||2,

[0031] Among them, L′ total λ4 and λ5 are the second loss function corresponding to the weighted graph stitching model, and λ4 and λ5 are the weighted hyperparameters. M i Let y′ be the source texture corresponding to the weight map of layer i. i The i-th layer seamlessly stitched weight graph is generated by the model, where i∈[1,n] and n is the number of samples. y i For the i-th layer seamless weighted graph representing real data, Here are the parameters for the differentiable renderer, ||·||1 is the mean absolute error, and ||·||2 is the minimum mean square error.

[0032] A large-scale terrain landscape generation apparatus based on differentiable rendering, the apparatus comprising:

[0033] The training sample acquisition module is used to preprocess several terrain samples, the weight map corresponding to each terrain sample, and the real weight map to obtain training samples.

[0034] A weighted map synthesis model construction module is used to construct a weighted map synthesis model. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. Upsampling is performed on the concatenated feature map to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map. The discriminator is used to output corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map.

[0035] The weighted graph synthesis model training module is used to train the weighted graph synthesis model based on the pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator, so as to obtain a trained generator.

[0036] The terrain landscape generation module is used to generate multi-layer weighted maps of each preprocessed terrain in a large-scale terrain using a trained generator, stitch the multi-layer weighted maps together to obtain a large-scale weighted map corresponding to the large-scale terrain, and generate a large-scale terrain landscape based on the large-scale weighted map.

[0037] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0038] The training samples are obtained by preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the real weight map.

[0039] A weighted map synthesis model is constructed. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. The concatenated feature map is then upsampled to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map, respectively. The discriminator is used to output corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map.

[0040] The weighted graph synthesis model is trained based on the pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator to obtain a trained generator.

[0041] A multi-layer weighted map of each preprocessed terrain in a large-scale terrain is generated by a trained generator. The multi-layer weighted maps are then stitched together to obtain a large-scale weighted map corresponding to the large-scale terrain. Large-scale terrain landscape is then generated based on the large-scale weighted map.

[0042] The aforementioned method, apparatus, and device for generating large-scale terrain landscapes based on differentiable rendering preprocess several terrain samples, the weight map corresponding to each terrain sample, and the real weight map to obtain training samples. A weight map synthesis model is constructed based on a generator, a differentiable renderer, and a discriminator, enabling the weight map of a local area to be expanded to a larger scale. The weight map synthesis model is trained using a pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator to obtain a trained generator. The trained generator is then used to optimize the weight map corresponding to the terrain, enabling the construction of large-scale terrain textures with rich details and strong scalability. This also reduces the deviation between the terrain re-rendered from the synthesized weight map and the rendered result of the real satellite image. In this embodiment, the weight map can support terrain of arbitrary size, providing users with simpler and more efficient artistic control while maintaining rich visual details. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a method for generating large-scale terrain landscapes based on differentiable rendering in one embodiment.

[0044] Figure 2 This is a schematic diagram of the process of training a weighted graph synthesis model in one embodiment;

[0045] Figure 3This is a schematic diagram of the weight graph synthesis and weight graph stitching process in one embodiment;

[0046] Figure 4 This is a schematic diagram of the process for obtaining input data for a weighted graph stitching model in one embodiment;

[0047] Figure 5 This is a schematic diagram of the process for obtaining a seamless stitched weight map in one embodiment;

[0048] Figure 6 This is a structural block diagram of a large-scale terrain landscape generation device based on differentiable rendering in one embodiment;

[0049] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, such as Figure 1 As shown, a method for generating large-scale terrain landscapes based on differentiable rendering is provided, including the following steps:

[0052] Step 102: Preprocess several terrain samples, the weight map corresponding to each terrain sample, and the real weight map to obtain training samples.

[0053] The terrain is modeled using a differentiable renderer, with each pixel in the terrain map corresponding to a vertex in the model, resulting in several terrain samples. A weight map represents the proportion of the source texture at each vertex of the model. The high-resolution texture used to record elements such as rocks and grass is called the source texture, which can be set according to actual needs. By weighting and blending the source textures, the weight map is optimized using a weight map inverse calculation method to obtain the weight map corresponding to the terrain sample. Preprocessing also includes cropping the terrain map and weight map according to pre-set sizes. The cropped results are used as input to the weight map synthesis model, enabling the model to output multi-layered weight maps at the target size. The size of the real weight map is the same as the size of the terrain sample. The weight map synthesis model uses the pre-processed real weight map to determine authenticity, reducing the deviation between the synthesized weight map and the rendered terrain from the real satellite image.

[0054] Step 104: Construct a weighted graph synthesis model.

[0055] The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weight maps, respectively. The feature maps obtained after feature extraction are then concatenated to obtain a concatenated feature map. Upsampling is performed on the concatenated feature map to obtain a multi-layer weight map. The differentiable renderer is used to render the multi-layer weight map and the preprocessed real weight map to obtain a rendered multi-layer weight map and a rendered real weight map, respectively. The discriminator is used to output the corresponding discrimination results based on the rendered multi-layer weight map and the rendered real weight map.

[0056] like Figure 2 The diagram shows the process of weight map synthesis and weight map stitching. The weight map synthesis model also includes a style transferer, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map, so that the rendered multi-layer weight map has a consistent style with the reference satellite image.

[0057] Step 106: Train the weighted graph synthesis model based on the pre-constructed first loss function, training samples, and the discrimination results output by the discriminator to obtain the trained generator.

[0058] Step 108: Generate multi-layer weight maps of each preprocessed terrain in the large-scale terrain using the trained generator, stitch the multi-layer weight maps together to obtain the large-scale weight map corresponding to the large-scale terrain, and generate the large-scale terrain landscape based on the large-scale weight map.

[0059] After obtaining the large-scale weight map corresponding to the large-scale terrain, select the multi-layer high-definition texture map corresponding to the source texture, import the large-scale terrain map into a modern engine (such as Unreal or Unity) to generate the corresponding large-scale terrain, import the large-scale weight map as the basis for blending the multi-layer high-definition texture map into the modern engine, and import the multi-layer high-definition texture map to generate a blended texture with high definition, thereby obtaining a large-scale 3D textured terrain landscape with rich details.

[0060] In the aforementioned method for generating large-scale terrain landscapes based on differentiable rendering, training samples are obtained by preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the real weight map. A weight map synthesis model is constructed based on the generator, differentiable renderer, and discriminator, which enables the weight map of a local area to be expanded to a larger scale. The weight map synthesis model is trained using a pre-constructed first loss function, training samples, and the discrimination results output by the discriminator to obtain a trained generator. The trained generator is then used to optimize the weight map corresponding to the terrain, which can construct large-scale terrain textures with rich details and strong scalability, and reduce the deviation between the terrain re-rendered by the synthesized weight map and the rendering result of the real satellite image. In this embodiment of the invention, the weight map can support terrain of arbitrary size, and while having rich visual details, it can provide users with simpler and more efficient artistic control.

[0061] In one embodiment, the step of preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the ground truth weight map to obtain training samples includes: acquiring several terrain samples; processing each terrain sample using the weight map inverse algorithm to obtain the weight map corresponding to each terrain sample; randomly cropping the weight map and the terrain sample at corresponding positions to obtain a weight map patch and a terrain sample patch of size 2k×2k; randomly cropping the weight map patch and the terrain sample patch at corresponding positions to obtain a preprocessed weight map and a preprocessed terrain sample of size k×k; obtaining the preprocessed ground truth weight map based on the weight map patch; and obtaining the training samples based on the preprocessed weight map, the preprocessed terrain sample, and the preprocessed ground truth weight map. In this embodiment, the weight map inverse calculation includes using a differentiable renderer to identify lighting parameters by optimizing the shadow distribution within the satellite image to obtain the restored shadowless terrain texture; optimizing the weight map from the satellite image in an end-to-end manner; relighting, randomly changing the lighting direction in each optimization run to obtain different shadow distributions under different lighting directions for the same terrain, ensuring that all textures can be fully illuminated; and introducing a style loss function, the purpose of which is to converge the sum of the weights of each weight map on any pixel to 1.

[0062] Specifically, a set of weight maps with corresponding terrain is obtained by inverse weight map calculation. In each training step, a 2k×2k patch is randomly cropped from the weight map and terrain sample, and their positions in the map must match. k×k patches are randomly cropped from the cropped weight map and terrain patch of size 2k×2k as input to the synthetic model. A self-supervised weight map synthesis model is constructed and trained to obtain a series of 2k×2k weight maps corresponding to the target terrain. The model transfers and expands the input weight map to the entire terrain, rather than simply enlarging it to twice its original size.

[0063] In one embodiment, the generator includes a first encoder, a second encoder, a feature stitching module, a residual network module, and a decoder; the residual network module includes multiple residual blocks; the generator is used to extract features from the preprocessed terrain sample and the corresponding preprocessed weight map, respectively, stitch the feature maps obtained after feature extraction to obtain a stitched feature map, and upsample the stitched feature map to obtain a multi-layer weight map, including the following steps: extracting features from the preprocessed terrain sample using the first encoder to obtain a terrain sample feature map; extracting features from the preprocessed weight map using the second encoder to obtain a weight feature map; stitching the terrain sample feature map and the weight feature map using the feature stitching module to obtain a stitched feature map; inputting the stitched feature map into the residual network module, and upsampling the output of the residual network module using the decoder to obtain a multi-layer weight map.

[0064] Specifically, the first encoder E1 is used to extract terrain features by downsampling the terrain map four times, and the second encoder E2 is used to extract weight map features by downsampling the weight map three times. The features from the two encoders are concatenated through channels and input into a bottleneck region consisting of 6 residual blocks. The number of channels for the features is always 768. The decoder is a traditional single-branch structure that upsamples the output of the residual network four times to output a multi-layer weight map. The number of channels in the final output weight map is the same as the number of layers in the input weight map, which is determined by the number of source textures defined by the user.

[0065] In one embodiment, the weight graph synthesis model further includes a style transferor for performing style transfer on the rendered multi-layer weight graph based on the rendered real weight graph.

[0066] In one embodiment, the pre-constructed first loss function is:

[0067] L total =L adv +λ1L W +λ2L R +λ3L style ,

[0068] L W =||G W (T t W s )-W t ||1,

[0069]

[0070]

[0071] Among them, L total Let L be the first loss function corresponding to the weighted graph synthesis model.style Here is the style transfer loss function corresponding to the style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map. λ1, λ2, and λ3 are weighted hyperparameters, and G... W For the generator model used in weighted graph synthesis, T t For the input terrain sample, W s For the input weight map, W t This is the true weighted graph, where ||·||1 represents the mean absolute error. Mi is the source texture corresponding to the weight map of the i-th layer, i∈[1,n], and n is the number of samples. W ti This is the true weight graph of the i-th layer. These are parameters for a differentiable renderer.

[0072] Specifically, such as Figure 2 As shown, a flowchart illustrating the training process of a weighted graph synthesis model is provided. The training process of the self-supervised weighted graph synthesis model specifically includes: calculating L for each weight layer. W Loss, to ensure the quality of weighted graph synthesis:

[0073] L W =||G W (T t W s )-W t ||1,

[0074] Among them, W s and W t These represent the input weight map and its corresponding real data, respectively. The cropped terrain is referred to as T. t The generator model used for weighted graph synthesis is G W To represent. Because there is a mutual dependency between the reconstructed texture and the real data, it can only be represented by L. W The quality of the generated textures cannot be guaranteed. Therefore, this invention uses a differentiable renderer during training and employs a relighting strategy to further improve the visual quality of the synthesized textures and enhance the quality of the weight map.

[0075]

[0076] in, and Use a discriminator to balance the quality of the weight map and terrain texture:

[0077]

[0078] Since the generated weight map may not be three-layered, this invention performs texture blending before calculating the style loss. The Gram matrix is ​​frequently used to calculate style loss in deep learning-based texture synthesis methods. Combining the above loss functions, the first loss function is obtained.

[0079] In one embodiment, such as Figure 3 The diagram illustrates a process for weight map synthesis and stitching, involving stitching together multi-layer weight maps to obtain a large-scale weight map corresponding to a large-scale terrain. The process includes: stitching together multi-layer weight maps to obtain four adjacent terrain features and a stitched weight map corresponding to each adjacent terrain feature; obtaining a central terrain map and a central weight map based on the four adjacent terrain features and their corresponding stitched weight maps; ensuring the central terrain map and central weight map have the same dimensions; ensuring the central weight map has the same dimensions as the multi-layer weight maps; generating a multi-layer weight map corresponding to the central terrain map using a trained weight map synthesis model; inputting the central terrain map, central weight map, and the corresponding multi-layer weight map into a pre-trained weight map stitching model to obtain a blending mask; the weight map stitching model includes a mask generator; cropping the blending mask according to the dimensions of the central weight map; blending the central weight map and the corresponding multi-layer weight map using the cropped blending mask to obtain a seamless stitched weight map; and obtaining a large-scale weight map corresponding to a large-scale terrain based on the seamless stitched weight maps of every four adjacent terrain features.

[0080] Specifically, such as Figure 4 The flowchart shown illustrates the process of obtaining input data for a weighted map stitching model. It creates weighted maps for four adjacent terrain features and extracts their central portions as s(s1,s2,...,s). n This cropped patch corresponds to terrain T. x Its size is the same as the weighted map of the four corners, and s has a clear cross-shaped seam inside, but is continuous with the edge produced by the adjacent terrain; in T x A new weighted graph is generated, denoted as t(t1, t2, ..., t). n The interior of t is continuous, but the edges formed by adjacent terrain are discontinuous. For example... Figure 5 The diagram shows the process of obtaining a seamlessly stitched weight map. A weight map stitching model is constructed and trained. The weight map stitching model includes a mask generator and a conditional multi-scale discriminator. By mixing two sets of weight maps (s and t), the continuity of the edge part is preserved while eliminating the central cross seam of s. The mask generated by the weight map stitching model is used for mixing to obtain a seamlessly stitched weight map.

[0081] In one embodiment, the seamlessly stitched weighted graph is as follows:

[0082] y′=sm′+t(1-m′),

[0083] m′=(1-c)m+c,

[0084] m = G m (T x ,s,t),

[0085] Where y′ is the seamlessly stitched weight map, s is the center weight map, m′ is the cropped blend mask, t is the multi-layer weight map corresponding to the center topographic map, m is the blend mask, c is the elliptical mask, and G m (·) represents the mask generator, T x Topographic map centered on the topography.

[0086] In one embodiment, the step of training the weighted graph stitching model includes: training the weighted graph stitching model according to a second loss function to obtain a trained weighted graph stitching model; the second loss function is:

[0087] L′ total =L Madv +λ4L R′ +λ5L M ,

[0088]

[0089]

[0090] L M =||G m (T x ,s,t)-c||2,

[0091] Among them, L′ total λ4 and λ5 are the second loss function corresponding to the weighted graph stitching model, and λ4 and λ5 are the weighted hyperparameters. M i Let y′ be the source texture corresponding to the weight map of layer i. i The i-th layer seamlessly stitched weight graph is generated by the model, where i∈[1,n] and n is the number of samples. y i For the i-th layer seamless weighted graph representing real data, Here are the parameters for the differentiable renderer, ||·||1 is the mean absolute error, and ||·||2 is the minimum mean square error.

[0092] Specifically, similar to weighted graph synthesis, this invention blends the source texture y′ and uses a discriminator to supervise the rendering of the generated terrain texture I′. Considering the size of the synthesized texture, a conditional multi-scale discriminator D is used. m To balance the global and local generation quality, and to calculate D m Losses:

[0093]

[0094] in, To improve stability during training, the L-value between the rendered composite texture and the re-rendered ground texture is calculated. R′ Loss and the L between the generated blended mask and the clipped mask c M loss:

[0095]

[0096] L M =||G m (T x ,s,t)-c||2,

[0097] in, Combining the above loss functions, we obtain the second loss function.

[0098] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0099] In one embodiment, such as Figure 6 As shown, a large-scale terrain landscape generation device based on differentiable rendering is provided, including: a training sample acquisition module 602, a weighted graph synthesis model construction module 604, a weighted graph synthesis model training module 606, and a terrain landscape generation module 608, wherein:

[0100] The training sample acquisition module 602 is used to preprocess several terrain samples, the weight map corresponding to each terrain sample, and the real weight map to obtain training samples.

[0101] The weighted map synthesis model construction module 604 is used to construct the weighted map synthesis model. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then stitched together to obtain a stitched feature map. Upsampling is performed on the stitched feature map to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map, respectively. The discriminator is used to output the corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map.

[0102] The weighted graph synthesis model training module 606 is used to train the weighted graph synthesis model based on the pre-constructed first loss function, training samples and the discrimination results output by the discriminator, so as to obtain a trained generator.

[0103] The terrain landscape generation module 608 is used to generate multi-layer weight maps of each preprocessed terrain in a large-scale terrain using a trained generator, stitch the multi-layer weight maps together to obtain a large-scale weight map corresponding to the large-scale terrain, and generate a large-scale terrain landscape based on the large-scale weight map.

[0104] In one embodiment, the weight map synthesis model also includes a style transferor for performing style transfer on the rendered multi-layer weight map based on the rendered real weight map.

[0105] In one embodiment, the weight map synthesis model further includes a style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map.

[0106] In one embodiment, the pre-constructed first loss function is further defined as follows:

[0107] L total =L adv +λ1L W +λ2L R +λ3L style ,

[0108] L W =||G W (T t W s )-W t ||1,

[0109]

[0110]

[0111] Among them, L totalLet L be the first loss function corresponding to the weighted graph synthesis model. style Here is the style transfer loss function corresponding to the style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map. λ1, λ2, and λ3 are weighted hyperparameters, and G... W For the generator model used in weighted graph synthesis, T t For the input terrain sample, W s For the input weight map, W t This is the true weighted graph, where ||·||1 represents the mean absolute error. M i Let i be the source texture corresponding to the weight map of layer i, where i ∈ [1, n] and n is the number of samples. W ti This is the true weight graph of the i-th layer. These are parameters for a differentiable renderer.

[0112] In one embodiment, the method further includes: acquiring several terrain samples; processing each terrain sample using a weighted graph inverse algorithm to obtain a weighted graph corresponding to each terrain sample; randomly cropping the weighted graph and the terrain sample at corresponding positions to obtain a weighted graph patch and a terrain sample patch of size 2k×2k; randomly cropping the weighted graph patch and the terrain sample patch at corresponding positions to obtain a preprocessed weighted graph and a preprocessed terrain sample of size k×k; obtaining a preprocessed true weighted graph based on the weighted graph patch; and obtaining training samples based on the preprocessed weighted graph, the preprocessed terrain sample, and the preprocessed true weighted graph.

[0113] In one embodiment, the generator further includes a first encoder, a second encoder, a feature splicing module, a residual network module, and a decoder; the residual network module includes multiple residual blocks.

[0114] In one embodiment, the method further includes: extracting features from the preprocessed terrain sample using the first encoder to obtain a terrain sample feature map; extracting features from the preprocessed weight map using the second encoder to obtain a weight feature map; stitching the terrain sample feature map and the weight feature map using the feature stitching module to obtain a stitched feature map; inputting the stitched feature map into a residual network module, and upsampling the output of the residual network module using the decoder to obtain a multi-layer weight map.

[0115] In one embodiment, the method further includes: stitching the multi-layer weight map to obtain four adjacent terrains and a stitched weight map corresponding to each adjacent terrain; obtaining a central terrain map and a central weight map based on the four adjacent terrains and their corresponding stitched weight maps; the central terrain map and the central weight map have the same size; the size of the central weight map is the same as the size of the multi-layer weight map; generating a multi-layer weight map corresponding to the central terrain map using the trained weight map synthesis model; inputting the central terrain map, the central weight map, and the multi-layer weight map corresponding to the central terrain map into a pre-trained weight map stitching model to obtain a hybrid mask; the weight map stitching model includes a mask generator; cropping the hybrid mask according to the size of the central weight map; using the cropped hybrid mask to blend the central weight map and the multi-layer weight map corresponding to the central terrain map to obtain a seamless stitched weight map; and obtaining a large-scale weight map corresponding to a large-scale terrain based on the seamless stitched weight map of every four adjacent terrains.

[0116] In one embodiment, the seamless stitching weight map is further defined as follows:

[0117] y′=sm′+t(1-m′),

[0118] m′=(1-c)m+c,

[0119] m = G m (T x ,s,t),

[0120] Where y′ is the seamlessly stitched weight map, s is the center weight map, m′ is the cropped blend mask, t is the multi-layer weight map corresponding to the center topographic map, m is the blend mask, c is the elliptical mask, and G m (·) represents the mask generator, T x Topographic map centered on the topography.

[0121] In one embodiment, the method further includes: training the weighted graph stitching model according to a second loss function to obtain a trained weighted graph stitching model; the second loss function is:

[0122] L′ total =L Madv +λ4L R′ +λ5L M ,

[0123]

[0124]

[0125] L M =||G m (T x,s,t)-c||2,

[0126] Among them, L′ total λ4 and λ5 are the second loss function corresponding to the weighted graph stitching model, and λ4 and λ5 are the weighted hyperparameters. M i Let y′ be the source texture corresponding to the weight map of layer i. i The i-th layer seamlessly stitched weight graph is generated by the model, where i∈[1,n] and n is the number of samples. y i For the i-th layer seamless weighted graph representing real data, Here are the parameters for the differentiable renderer, ||·||1 is the mean absolute error, and ||·||2 is the minimum mean square error.

[0127] Specific limitations regarding the apparatus for generating large-scale terrain landscapes based on differentiable rendering can be found in the limitations of the method for generating large-scale terrain landscapes based on differentiable rendering described above, and will not be repeated here. Each module in the aforementioned apparatus for generating large-scale terrain landscapes based on differentiable rendering can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0128] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating large-scale terrain landscapes based on differentiable rendering. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0129] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A method for generating large-scale terrain landscapes based on differentiable rendering, characterized in that, The method includes: The training samples are obtained by preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the real weight map. A weighted map synthesis model is constructed. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. The concatenated feature map is then upsampled to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map, respectively. The discriminator is used to output corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map. The weighted graph synthesis model is trained based on the pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator to obtain a trained generator. A multi-layer weighted map of each preprocessed terrain in a large-scale terrain is generated by a trained generator. The multi-layer weighted maps are then stitched together to obtain a large-scale weighted map corresponding to the large-scale terrain. Large-scale terrain landscape is then generated based on the large-scale weighted map.

2. The method according to claim 1, characterized in that, The pre-constructed first loss function is: L total = L adv + λ1L W + λ2L R + λ3L style , L W =||G W (T t ,W s )-W t ||1, Among them, L total Let L be the first loss function corresponding to the weighted graph synthesis model. style Here is the style transfer loss function corresponding to the style transferor, which is used to perform style transfer on the rendered multi-layer weight map based on the rendered real weight map. λ1, λ2, and λ3 are weighted hyperparameters, and G... W For the generator model used in weighted graph synthesis, T t For the input terrain sample, W s For the input weight map, W t This is the true weighted graph, where ||·||1 represents the mean absolute error. M i Let i be the source texture corresponding to the weight map of layer i, where i ∈ [1, n] and n is the number of samples. W ti This is the true weight graph of the i-th layer. These are parameters for a differentiable renderer.

3. The method according to claim 1, characterized in that, The steps for preprocessing several terrain samples, the weight map corresponding to each terrain sample, and the ground truth weight map to obtain training samples include: Several terrain samples are obtained, and each terrain sample is processed by the weighted graph inverse method to obtain the weighted graph corresponding to each terrain sample. Randomly crop the corresponding positions of the weight map and the terrain sample to obtain a weight map patch and a terrain sample patch with a size of 2k×2k; Random cropping is performed at corresponding positions of the weight map patch and the terrain sample patch to obtain a preprocessed weight map and a preprocessed terrain sample of size k×k. Based on the weight graph patch, the preprocessed true weight graph is obtained; Training samples are obtained based on the preprocessed weight map, the preprocessed terrain samples, and the preprocessed ground truth weight map.

4. The method according to claim 1, characterized in that, The generator includes a first encoder, a second encoder, a feature splicing module, a residual network module, and a decoder; the residual network module includes multiple residual blocks.

5. The method according to claim 4, characterized in that, The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weight maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. Finally, upsampling is performed on the concatenated feature map to obtain a multi-layer weight map. The steps include: The preprocessed terrain sample is used to extract features through the first encoder to obtain a terrain sample feature map. The preprocessed weight map is processed by the second encoder to extract features, resulting in a weight feature map. The feature stitching module stitches the terrain sample feature map and the weighted feature map together to obtain a stitched feature map. The spliced ​​feature map is input into the residual network module, and the output of the residual network module is upsampled by the decoder to obtain a multi-layer weight map.

6. The method according to claim 1, characterized in that, The step of stitching together the multi-layer weighted maps to obtain a large-scale weighted map corresponding to the large-scale terrain includes: The multi-layer weighted graphs are stitched together to obtain four adjacent terrains and the stitched weighted graph corresponding to each adjacent terrain. A central topographic map and a central weighted map are obtained based on four adjacent terrain features and their corresponding stitched weighted maps, respectively; the central topographic map and the central weighted map have the same size; the size of the central weighted map is the same as the size of the multi-layer weighted map. The trained weighted graph synthesis model is used to generate a multi-layered weighted graph corresponding to the central topographic map. The central topographic map, the central weight map, and the corresponding multi-layer weight map are input into a pre-trained weight map stitching model to obtain a hybrid mask; the weight map stitching model includes a mask generator. The hybrid mask is cropped according to the size of the central weight map, and the cropped hybrid mask is used to blend the central weight map and the multi-layer weight map corresponding to the central topographic map to obtain a seamlessly stitched weight map. Based on the seamless stitching weight map of every four adjacent terrain features, a large-scale weight map corresponding to the large-scale terrain is obtained.

7. The method according to claim 6, characterized in that, The seamless stitching weight map is as follows: y′=sm′+t(1-m′), m′=(1-c)m+c, m = G m (T x s, t), Where y′ is the seamlessly stitched weight map, s is the center weight map, m′ is the cropped blend mask, t is the multi-layer weight map corresponding to the center topographic map, m is the blend mask, c is the elliptical mask, and G m (·) represents the mask generator, T x Topographic map centered on the topography.

8. The method according to claim 6, characterized in that, The steps for training the weighted graph stitching model include: The weighted graph stitching model is trained according to the second loss function to obtain a trained weighted graph stitching model; the second loss function is: The total =L Madv +λ4L R′ +λ5L M , L M =||G m (T x ,s,t)-c||2, Among them, L′ total λ4 and λ5 are the second loss function corresponding to the weighted graph stitching model, and λ4 and λ5 are the weighted hyperparameters. M i Let y be the source texture corresponding to the weight map of layer i. i ' is the seamlessly stitched weight graph of the i-th layer generated by the model, where i∈[1,n] and n is the number of samples. y i For the i-th layer seamless weighted graph representing real data, Here are the parameters for the differentiable renderer, ||·||1 is the mean absolute error, and ||·||2 is the minimum mean square error.

9. A large-scale terrain landscape generation device based on differentiable rendering, characterized in that, The device includes: The training sample acquisition module is used to preprocess several terrain samples, the weight map corresponding to each terrain sample, and the real weight map to obtain training samples. A weighted map synthesis model construction module is used to construct a weighted map synthesis model. The weighted map synthesis model includes a generator, a differentiable renderer, and a discriminator. The generator is used to extract features from the preprocessed terrain samples and the corresponding preprocessed weighted maps, respectively. The extracted feature maps are then concatenated to obtain a concatenated feature map. Upsampling is performed on the concatenated feature map to obtain a multi-layer weighted map. The differentiable renderer is used to render the multi-layer weighted map and the preprocessed real weighted map to obtain a rendered multi-layer weighted map and a rendered real weighted map. The discriminator is used to output corresponding discrimination results based on the rendered multi-layer weighted map and the rendered real weighted map. The weighted graph synthesis model training module is used to train the weighted graph synthesis model based on the pre-constructed first loss function, the training samples, and the discrimination results output by the discriminator, so as to obtain a trained generator. The terrain landscape generation module is used to generate multi-layer weighted maps of each preprocessed terrain in a large-scale terrain using a trained generator, stitch the multi-layer weighted maps together to obtain a large-scale weighted map corresponding to the large-scale terrain, and generate a large-scale terrain landscape based on the large-scale weighted map.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images