Scene construction method and system based on micro-renderable multi-source remote sensing data

Abstract

The invention provides a scene construction method and system based on micro-renderable multi-source remote sensing data, and the method comprises the steps: obtaining multi-source remote sensing data, namely, high-resolution remote sensing images and laser radar data; rendering primitive distribution is initialized by reducing a sampling point cloud, a rendering depth constraint function is established by using a point cloud projection depth map, and a primitive space distribution priori field is constructed based on local neighborhood features of the point cloud; a multi-scale progressive training strategy is designed, an image pyramid structure is adopted to realize resolution adaptive optimization, and performance overhead is reduced in combination with an image block asynchronous optimization mechanism; and rendering a scene true shoot and a digital surface model by using orthogonal projection, and synchronously realizing laser radar point cloud color assignment. According to the method, a new multi-modal remote sensing data collaborative modeling mechanism is provided, joint optimization of geometric-spectral features is realized through a micro-renderable architecture, and a complex geographic scene construction task can be dealt with.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision three-dimensional reconstruction, and relates to a scene construction technology based on differentiable rendering, and is mainly applied to the fields of true orthophoto, digital surface model generation and laser radar point cloud coloring. BACKGROUND

[0002] The research content of differentiable rendering is to modify the forward rendering process to support the gradient of the output image of the calculation rendering with respect to the input parameters. Specifically, differentiable rendering simultaneously contains a strictly coupled forward rendering and backward optimization double process. The forward process involves parameterized modeling to numerically characterize scene attributes, and generates a synthetic image based on a physical rendering equation. The difference between the rendering result and the real image is compared by constructing a loss function. The backward process is based on the gradient descent method, combined with the gradient calculation result, and optimizes the scene parameters to minimize the loss function. The scene parameter set is iteratively updated until the loss function converges to a local minimum, and the optimal scene parameter estimation and image rendering are obtained.

[0003] The existing three-dimensional geographic scene construction method based on differentiable rendering can be mainly divided into two categories: one is a scene construction method based on neural implicit representation, which realizes continuous scene representation by combining multi-layer perceptron (MLP) and volumetric rendering (Volumetric Rendering) technology through implicit function coding features; the other is to construct an explicit scene rendering primitive, which gives the primitive color, geometry and other features, and adjusts the parameters and spatial distribution of the rendering primitive in the training process by using a differentiable rasterizer. The current geographic scene construction method based on differentiable rendering only relies on image spectral features, lacks effective integration of geometric priors, and is difficult to cope with complex scene construction tasks. SUMMARY

[0004] The technical problem to be solved by the present application is to avoid the technical bottlenecks of the existing scene construction method, such as lack of spatial geometry and neighborhood information support, and difficulty in balancing high-precision training and low-performance overhead, and to provide a three-dimensional geographic scene construction method fusing multi-source remote sensing data. The present application constructs a laser radar point cloud constrained differentiable rendering optimization framework, combines a multi-scale progressive training strategy and an image block asynchronous optimization mechanism to reduce performance overhead while ensuring training accuracy. Compared with the existing method, the present application can accurately restore scene texture, geometry and neighborhood information in complex geographic scenes, and can effectively support high-precision city modeling, topographic mapping and other geographic information applications. Therefore, the method has important use value and wide application prospect.

[0005] The present application provides a multi-source remote sensing data scene construction method based on differentiable rendering, comprising the following steps: Step 1, acquire multi-source remote sensing data, including high-resolution remote sensing images and laser radar point cloud data; Step 2, construct a point cloud data guided differentiable rendering optimization framework: initialize the rendering primitive distribution by down-sampling the point cloud, establish a rendering depth constraint function by using the point cloud projection depth map, construct the rendering primitive spatial distribution based on the local neighborhood features of the point cloud, and finally obtain the rendering depth and rendering image based on the rendering depth constraint function and the rendering primitive spatial distribution; Step 3, design a multi-scale progressive training strategy for high-resolution remote sensing images to obtain better rendering images, realize resolution adaptive optimization by using image pyramid structure, and reduce performance overhead by combining image block asynchronous optimization mechanism; Step 4, combine the rendering primitive spatial distribution obtained in the previous step, use orthogonal projection to render the real scene projection image and digital surface model, and simultaneously realize laser radar point cloud coloring.

[0006] Further, in step 2, the initial rendering primitive number is determined based on the preset scene complexity, and the farthest point sampling strategy is used to sample the laser radar point cloud to obtain the initial rendering primitive spatial distribution For each rendering primitive , the average distance of the neighborhood primitives is calculated by the k-nearest neighbor algorithm knn, and the rendering primitive scale attribute is initialized accordingly , The calculation formula is as follows:

[0007] Where k represents the positions of the k rendering primitives.

[0008] Further, in step 2, the implementation of obtaining the rendering depth is as follows: Project the laser radar point cloud onto the image plane according to the camera position and attitude parameters to obtain a depth map , and exclude the projection points that do not meet the occlusion relationship according to the point cloud visibility elimination algorithm; use the sparse depth projection points obtained by the visibility algorithm to restore and correct the depth estimation large model based on Transformer, predict the absolute spatial scale of the depth map, and obtain a continuous depth map with real spatial measurement scale ; obtain the rendering result of the rendering primitive depth by volume rendering technology , establish the depth constraint function Minimize the difference between the depth map and the rendering result of the rendering primitive depth , and use to control the loss weight: .

[0009] Further, the depth estimation large model is a depthanything large model.

[0010] Further, in step 2, the implementation of obtaining the rendered image is as follows: A spatial hash grid is constructed for the laser radar point cloud according to a certain grid resolution, and a fast neighborhood query is performed on the rendering primitive according to the hash grid to obtain neighborhood coordinates The neighborhood information is encoded into feature attributes using a neural network The feature attributes are stored in the rendering primitive and are continuously adjusted and optimized following the differentiable rendering process; the rendering primitive feature attributes are used to predict other attributes of the rendering primitive using a multi-layer perception, the rendering primitives in the space are compressed along the imaging optical axis direction according to the projection relationship, two-dimensional rendering primitives are obtained, and the two-dimensional rendering primitives are sorted according to the depth, and finally all the two-dimensional rendering primitive attributes are synthesized by volume rendering technology to obtain the rendered image , and the real image is used for photometric supervision, and is used to control the loss weight:

[0011] The loss function of the final differentiable rendering optimization is composed of a depth loss and a photometric loss: .

[0012] Further, the neural network is a classic PointNet architecture.

[0013] Further, the other attributes of the rendering primitive include color attribute c, spatial rotation parameter, and opacity attribute o.

[0014] Further, in step 3, the process of resolution adaptive optimization is as follows: a multi-level image pyramid structure is constructed, and high-resolution image levels are gradually introduced to participate in the training supervision of the differentiable rendering primitive, low-resolution images are used to constrain the global information of the scene in the early stage, and higher resolution image levels are dynamically loaded as the training process progresses, and the rendering primitive distribution is gradually guided to learn local high-frequency texture features.

[0015] Further, in step 3, the process of image block asynchronous optimization mechanism is as follows: according to the device hardware conditions, the high-resolution image is divided into blocks according to the projection relationship, the block index is , the original image resolution is , the main point coordinates are , the image resolution after block division is , and an additional main point offset is introduced: ​

[0016] The perspective projection matrix is rewritten as:

[0017] where znear refers to the starting point of the view frustum calculation, zfar is the ending point of the view frustum calculation, top, bottom, left and right respectively refer to the upper, lower, left and right edges of the image in the camera coordinate system, = 1; The image is divided in space according to a preset blocking strategy, each block filters the rendering primitives in the view frustum according to the projection relationship, and the rendering primitives are used to render the image, a local optimization objective function is constructed using depth loss and luminosity loss, and decoupled optimization between blocks is realized through an asynchronous parameter update strategy, that is, each time of training optimizes only one block image instead of the complete image, and the performance overhead is significantly reduced during the training process.

[0018] The application also provides a multi-source remote sensing data scene construction system based on differentiable rendering, a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the multi-source remote sensing data scene construction method based on differentiable rendering.

[0019] The application solves three problems of the existing optimization framework based on the rendering primitives of differentiable rendering: unable to effectively utilize high-resolution images, unable to reduce performance overhead while ensuring optimization effect, and unable to supplement the constraints of point cloud information in the differentiable rendering optimization framework. Through the implementation of the several strategies of the application, large-scale remote sensing scene rendering model construction can be completed on consumer-level devices, greatly reducing the hardware threshold for training optimization. BRIEF DESCRIPTION OF DRAWINGS

[0020] The application will be further described below in combination with the drawings and embodiments, and the drawings are as follows: Figure 1 The flowchart of the embodiment of the application is shown in Figure 1. Figure 2 The point cloud local neighborhood feature construction primitive space distribution prior field diagram of the embodiment of the application is shown in Figure 2, where (a) is a scene diagram, (b) is a space hash grid diagram constructed by a certain grid resolution, and (c) is a diagram for quickly querying the neighborhood of the rendering primitive according to the hash grid.

[0021] Figure 3 The true orthographic image (TDOM) and the digital surface model (DSM) results rendered by the differentiable rendering primitives after training optimization are shown in Figure 3, where (a) is a first area survey area, and (b) is a second area survey area. DETAILED DESCRIPTION

[0022] The technical scheme adopted by the present application to solve its technical problems is: a multi-source remote sensing data joint three-dimensional geographic scene construction method based on differentiable rendering, which comprises the following steps: Step 1, obtaining multi-source remote sensing data, including high-resolution remote sensing images and laser radar point cloud data; Step 2, constructing a point cloud data guided differentiable rendering optimization framework: initializing the rendering primitive distribution by down-sampling the point cloud, establishing a rendering depth constraint function by using the point cloud projection depth map, constructing a rendering primitive space distribution prior field based on the local neighborhood features of the point cloud, and finally obtaining the rendering depth and the rendering image based on the rendering depth constraint function and the primitive space distribution prior field; The conventional differentiable rendering optimization framework only needs image data as supervision, and the present method adds the constraint guidance of laser point cloud data to improve the optimization quality and accuracy.

[0023] Step 3, designing a multi-scale progressive training strategy for high-resolution remote sensing images to obtain better rendering images, using an image pyramid structure to realize resolution adaptive optimization, and combining an image block asynchronous optimization mechanism to reduce performance overhead; Step 4, combining the optimized rendering primitive space distribution to render the true orthographic projection image and the digital surface model of the scene, and synchronously realizing laser radar point cloud coloring.

[0024] In step 2 of the above method, the specific construction method of the point cloud constrained differentiable rendering optimization framework is: Step 2.1, determining the initial number of rendering primitives based on the preset scene complexity, using the farthest point sampling strategy to sample the laser radar point cloud to obtain the initial rendering primitive space distribution ; for each rendering primitive , the average distance of the neighborhood primitives is calculated by the k-nearest neighbor (knn) algorithm, and the rendering primitive scale attribute is initialized accordingly , The calculation formula is:

[0025] The scale attribute s of the rendering primitive needs to be set to an estimated value in advance before optimization to prevent unstable convergence. The k nearest rendering primitive positions to the rendering primitive are found by knn, and the average distance is calculated.

[0026] Step 2.2, projecting the laser radar point cloud onto the image plane according to the camera position and attitude parameters to obtain a depth map The projection points that do not conform to the occlusion relationship are excluded according to the point cloud visibility culling algorithm. The absolute spatial scale of the depth map predicted by the Transformer-based depth estimation large model (depthanything large model) is restored and corrected using the sparse depth projection points removed by the visibility algorithm, so as to obtain a continuous depth map with a true spatial metric scale. ; Obtain the depth rendering results of the rendering primitives through volume rendering technology Establish depth constraint function Minimize depth map and depth map Differences in usage Controlling loss weights:

[0027] Step 2.3: Construct a spatial hash grid for the LiDAR point cloud according to a certain grid resolution, such as... Figure 2 As shown in (a) in the figure; a fast neighborhood lookup of the rendering primitives is performed based on the hash grid to obtain the results. neighborhood coordinates ( Figure 2 (b) in the example uses a neural network to utilize neighborhood information. Encoding as feature attributes Stored in the rendering primitives, and continuously adjusted and optimized in the differentiable rendering process, such as... Figure 2 (c) in:

[0028] In this embodiment of the invention, the classic PointNet architecture is used to construct a K*C tensor (where C is the dimension of each coordinate) from the K coordinates obtained by neighborhood query, and this tensor is directly input into the network to obtain the features of the rendering primitive. In addition to possessing feature attributes... In addition to the scale attribute s, it also has the color attribute c and spatial rotation parameters. And the opacity property o. Utilizing rendering primitive feature properties. Other attributes of the rendering primitives can be predicted using a multilayer perceptron (MLP). By compressing the rendering primitives in space according to their projection relationship along the imaging optical axis, two-dimensional rendering primitives are obtained. These primitives are then sorted according to their depth. Finally, all the attributes of the two-dimensional rendering primitives are synthesized using volumetric rendering techniques to obtain the rendered image. and using real images Perform photometric monitoring, using Controlling loss weights:

[0029] The final loss function consists of depth loss and photometric loss:

[0030] In the above method step 3, the multi-scale progressive training strategy is implemented as follows: Step 3.1, a multi-level image pyramid structure is constructed, and high-resolution image levels are introduced step by step to participate in the training supervision of the differentiable rendering primitive. Low-resolution images are used to constrain global information in the scene at the beginning, and higher resolution image levels are dynamically loaded as the training progresses, gradually guiding the rendering primitive distribution to learn local high-frequency texture features.

[0031] Step 3.2, implement an image block asynchronous optimization mechanism: combined with the device hardware conditions, the high-resolution image is divided into blocks according to the projection relationship, and the block index is . The original image resolution is The principal point coordinates are , the resolution of the image after blocking is , and an additional principal point offset is introduced:

[0032] The perspective projection matrix is rewritten as:

[0033] Where znear refers to the starting point of the view frustum calculation, zfar is the end point of the view frustum calculation. top, bottom, left, right respectively refer to the upper, lower, left and right edges of the image in the camera coordinate system, = 1.

[0034] The image is divided in space according to the preset blocking strategy, each block selects the rendering primitive in the view frustum according to the projection relationship, and renders the image using these rendering primitives, and uses depth loss and luminosity loss to construct a local optimization objective function. Decoupled optimization between blocks is achieved through an asynchronous parameter update strategy, that is, only one block image is optimized instead of the entire image each time, significantly reducing performance overhead during training.

[0035] On the other hand, the embodiment of the present application also provides a multi-source remote sensing data scene construction system based on differentiable rendering, a processor and a memory, the memory is used to store program instructions, and the processor is used to call the storage instructions in the memory to execute the multi-source remote sensing data scene construction method based on differentiable rendering as described in the above technical solution.

[0036] The computer hardware environment of the experiment of the present application is 13th Gen Intel i9-13900K, GeForce RTX3090Ti GPU, the software environment is linux, the deep learning framework uses PyTorch, and rendering acceleration is carried out based on CUDA. In the training process, the model is trained for 40k rounds. The present application is verified on two groups of remote sensing image data sets, which are the first area and the second area data sets. The present application adopts peak signal to noise ratio (PSNR), structural similarity index (SSIM) and learned perceptual image patch similarity (LPIPS) as evaluation indexes, and the results are shown in Table 1.

[0037] Table 1 Test results of different data sets

[0038] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or replace them with similar ways, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A method for constructing multi-source remote sensing data scenes based on differentiable rendering, characterized in that, Includes the following steps: Step 1: Acquire multi-source remote sensing data, including high-resolution remote sensing images and lidar point cloud data; Step 2: Construct a differentiable rendering optimization framework guided by point cloud data: Initialize the rendering primitive distribution by downsampling point cloud, establish a rendering depth constraint function using point cloud projection depth map, construct the rendering primitive spatial distribution based on local neighborhood features of point cloud, and finally obtain the rendering depth and rendering image based on the rendering depth constraint function and the rendering primitive spatial distribution. Step 3: Design a multi-scale progressive training strategy for high-resolution remote sensing images to obtain better rendered images. Use an image pyramid structure to achieve resolution adaptive optimization and combine an image block asynchronous optimization mechanism to reduce performance overhead. Step 4: Combining the previously obtained rendering primitive spatial distribution, use orthogonal projection to render the true orthogonal image of the scene and the digital surface model, and simultaneously achieve colorization of the LiDAR point cloud.

2. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 1, characterized in that: In step 2, the initial number of rendering primitives is determined based on the preset scene complexity, and the initial spatial distribution of rendering primitives is obtained by sampling in the LiDAR point cloud using the farthest point sampling strategy. For each rendering primitive The average spacing of neighborhood primitives is calculated using the k-nearest neighbor algorithm (kNN), and the rendering primitive scale attribute is initialized accordingly. , The calculation formula is: Where k represents the k rendering primitive positions.

3. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 1, characterized in that: In step 2, the rendering depth is obtained as follows: The depth map is obtained by projecting the LiDAR point cloud onto the image plane according to the camera position and attitude parameters. The projection points that do not conform to the occlusion relationship are excluded according to the point cloud visibility culling algorithm; the sparse depth projection points after the visibility algorithm culling are used to recover and correct the large depth estimation model based on Transformer, predict the absolute spatial scale of the depth map, and obtain a continuous depth map with a true spatial metric scale. ; Obtain the depth rendering results of the rendering primitives through volume rendering technology Establish depth constraint function Minimize depth map and render primitive depth rendering results Differences in usage Controlling loss weights: 。 4. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 3, characterized in that: The large-scale depth estimation model is the depthanything large-scale model.

5. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 3, characterized in that: In step 2, the method for obtaining the rendered image is as follows: A spatial hash grid is constructed for the LiDAR point cloud according to a certain grid resolution. The rendering primitives are then obtained by performing a fast neighborhood lookup based on the hash grid. neighborhood coordinates Using neural networks to transfer neighborhood information Encoding as feature attributes Stored in rendering primitives, and continuously adjusted and optimized in the differentiable rendering process; utilizing the feature attributes of rendering primitives. Other attributes of the rendering primitives are predicted using a multilayer perceptron. Two-dimensional rendering primitives are obtained by compressing them along the imaging optical axis according to their projection relationship. These primitives are then sorted according to their depth. Finally, all attributes of the two-dimensional rendering primitives are synthesized using volumetric rendering techniques to obtain the rendered image. and using real images Photometric monitoring was performed using... Controlling loss weights: The final loss function for differentiable rendering optimization consists of depth loss and photometric loss: 。 6. The method for constructing a scene from multi-source remote sensing data based on differentiable rendering as described in claim 5, characterized in that: Neural Networks It uses the classic PointNet architecture.

7. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 5, characterized in that: Other properties of the rendering primitive include the color property c, the spatial rotation parameter, and the opacity property o.

8. The method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in claim 1, characterized in that: In step 3, the process of achieving resolution adaptive optimization is as follows: construct a multi-level image pyramid structure, introduce high-resolution image layers to participate in the training supervision of differentiable rendering primitives step by step, initially use low-resolution images to constrain the global information of the scene, dynamically load higher resolution image layers as the training process progresses, and progressively guide the distribution of rendering primitives to learn local high-frequency texture features.

9. The method for constructing a scene based on differentiable rendering of multi-source remote sensing data as described in claim 1, characterized in that: In step 3, the asynchronous optimization mechanism for image segmentation proceeds as follows: Based on the device hardware conditions, the high-resolution image is segmented according to the projection relationship, with the segmentation index being... The original image resolution is Like the principal point coordinates are The resolution of the image after segmentation is Introducing additional principal point offset : The perspective projection matrix is ​​rewritten as: Where znear refers to the starting point of the view frustum calculation, zfar is the ending point of the view frustum calculation, and top, bottom, left, and right refer to the top, bottom, left, and right coordinates of the image in the camera coordinate system, respectively. =1; The image is divided into spatial domains according to a preset block segmentation strategy. Each block selects rendering primitives within the view frustum based on projection relationships and uses these rendering primitives to render the image. A local optimization objective function is constructed using depth loss and photometric loss. Decoupling optimization between blocks is achieved through an asynchronous parameter update strategy, that is, each training session only optimizes one block of the image instead of the entire image, which significantly reduces performance overhead during training.

10. A multi-source remote sensing data scene construction system based on differentiable rendering, characterized in that: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the method for constructing a multi-source remote sensing data scene based on differentiable rendering as described in any one of claims 1-9.

Images