Remote sensing three-dimensional reconstruction method and system based on combination of NeRF and diffusion model

Abstract

The application discloses a remote sensing three-dimensional reconstruction method and system based on combination of NeRF and a diffusion model, and the method comprises the following steps: step S1, acquiring satellite images at multiple time phases or multiple angles and corresponding camera attitude information; step S2, constructing a neural radiation field model, generating a ray along a line of sight direction and sampling a voxel point on the ray, predicting a volume density and a color of each voxel point through a neural network, and generating a rendering image with consistent view angles through volume rendering; step S3, inputting the rendering image into a conditional diffusion model, iteratively denoising and enhancing the light consistency of the image under the condition of the light information of the reference image, and obtaining an enhanced output image; and step S4, generating a high-fidelity three-dimensional ground surface model. Through the cooperative mechanism of the neural radiation field and the conditional diffusion model, the application realizes remote sensing three-dimensional reconstruction with high robustness and high fidelity, and effectively overcomes the problems of light difference at multiple time phases, geometric deviation and loss of texture details.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and remote sensing image processing technology, specifically to a remote sensing 3D reconstruction method and system based on the combination of NeRF and diffusion models. Background Technology

[0002] With the rapid development of high-resolution remote sensing satellites, oblique photography, and multi-view stereo imaging technologies, 3D reconstruction based on multi-view imagery has become an important means of geographic information extraction and surface modeling. Existing 3D reconstruction methods mostly employ multi-view geometric principles such as feature point matching, disparity estimation, and dense reconstruction to generate surface point clouds and digital elevation models (DEM / DSM).

[0003] However, in real-world remote sensing imaging environments, images are susceptible to variations in illumination, atmospheric scattering, occlusion, and changes in viewing angle. This is especially true in areas with repetitive textures, undulating terrain, or shadow coverage, where traditional stereo matching methods struggle to guarantee stability and continuity, leading to decreased reconstruction accuracy and completeness. In recent years, deep learning-based multi-view stereo methods (such as MVSNet and COLMAP) have significantly improved reconstruction accuracy, but they still exhibit holes and noise in areas with weak textures and heavy occlusion. While traditional Neural Radiance Field (NeRF) methods can generate continuous scenes, they are sensitive to multi-temporal illumination variations and noise, making them difficult to apply directly to remote sensing scenes.

[0004] The main limitations of existing technologies are that remote sensing images typically originate from different times, orbits, and sensors, exhibiting significant multi-temporal characteristics and attitude differences. The rational polynomial coefficients (RPC) parameters in satellite imaging models may be affected by changes in the external environment or orbital drift, necessitating complex registration and correction processes for geometrically constrained reconstruction methods. Even so, it remains difficult to completely eliminate geometric biases and radiometric inconsistencies, thus limiting the fine representation and illumination consistency of 3D surface models.

[0005] With the development of deep learning and neural implicit representation technology, novel 3D modeling methods based on neural radiation fields (NeRF) have attracted widespread attention in the field of computer vision. NeRF learns continuous mapping functions from 3D coordinates and viewing direction to volume density and color through neural networks, enabling realistic new perspective rendering and 3D reconstruction even with only a limited number of input images. Unlike traditional explicit modeling, NeRF can express continuous spatial structures in the form of implicit functions, offering advantages such as compact models, strong detail representation, and simultaneous micro-optimization of geometry and appearance.

[0006] However, directly applying NeRF to remote sensing scenarios still faces the following problems: 1. Satellite imagery has a limited parallax range and sparse ray sampling, which makes feature learning in the depth dimension difficult; 2. Multiple time phases and lighting conditions lead to color inconsistencies, making color prediction networks prone to confusion; 3. It has a wide imaging range and large differences in the scale of ground objects, resulting in high training costs and a tendency to get trapped in local optima; 4. Traditional NeRF assumes an ideal camera and stable lighting, while remote sensing imaging is often accompanied by complex atmospheric environment, shadow distribution and attitude uncertainty, which makes model convergence difficult and geometric boundaries blurred. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model. Through the synergistic mechanism of neural radiation field and conditional diffusion model, it achieves highly robust and high-fidelity remote sensing 3D reconstruction, effectively overcoming the problems of multi-temporal illumination differences, geometric deviations, and loss of texture details.

[0008] The second objective of this invention is to provide a remote sensing 3D reconstruction system based on the combination of NeRF and diffusion model. Through the synergistic mechanism of neural radiation field and conditional diffusion model, it achieves highly robust and high-fidelity remote sensing 3D reconstruction, effectively overcoming the problems of multi-temporal illumination differences, geometric deviations, and loss of texture details.

[0009] To achieve one of the objectives of this invention, the following solution is adopted: The remote sensing 3D reconstruction method based on NeRF combined with diffusion model includes the following steps: Step S1: Acquire satellite images from multiple time phases or angles and their corresponding camera attitude information; perform adaptive geometric correction and multi-constraint radiometric equalization on the satellite images to obtain preprocessed images. Step S2: Based on the preprocessed image, construct a neural radiation field model, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model. Step S3: Input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image; Step S4: Based on the output image, jointly optimize the neural radiation field model and the conditional diffusion model to generate a high-fidelity three-dimensional surface model.

[0010] Furthermore, the adaptive geometric correction in step S1 includes: using a rational polynomial coefficient model or a feature-matching-based bundle adjustment method to unify satellite images from different time phases or multiple angles to the same geographic coordinate reference system, and optimizing the exterior orientation parameters by minimizing the reprojection error.

[0011] Furthermore, the multi-constraint radiometric equalization processing in step S1 includes: using polynomial regression to perform color mapping on satellite images, and introducing shadow detection and compensation, calculating the brightness compensation increment of shadow areas based on the solar altitude angle and terrain slope angle, so that satellite images at different times or angles have consistent radiometric characteristics.

[0012] Furthermore, in step S2, when constructing the neural radiation field model, an adaptive coding frequency strategy is used to encode the position of the three-dimensional spatial coordinates. The number of coding layers is divided into three levels: high, medium, and low, according to the surface scale range. The neural network adopts a multilayer perceptron and includes a skip connection structure to fuse the intermediate layer features with the original position code.

[0013] Furthermore, in step S2, the volume rendering uses numerical integration to discretize the product of transmittance and color, and the reprojection loss uses a weighted sum of mean square error and structural similarity metric, where the mean square error weight is 0.8 and the structural similarity weight is 0.2.

[0014] Furthermore, in step S3, the forward process of the conditional diffusion model gradually adds Gaussian noise to the image, while the reverse process learns to recover the image from the noise. The illumination information, as a conditional vector, includes the solar altitude angle, azimuth angle, and atmospheric brightness, and guides the denoising process through an illumination loss function, which includes an illumination estimation difference term and an illumination change constraint term.

[0015] Furthermore, the iterative denoising in step S3 further adopts a phased strategy: in the coarse step stage, a large step size is used to eliminate the main noise and illumination differences, and in the fine step stage, a small step size is used to restore texture details and local contrast.

[0016] Furthermore, the joint optimization in step S4 adopts an end-to-end total loss function, which is a weighted sum of neural radiation field reprojection loss, diffusion model negative log-likelihood loss, and consistency regularization loss, wherein the diffusion model loss has a weight of 0.5, the consistency loss has a weight of 0.3, and the regularization loss has a weight of 0.1.

[0017] Furthermore, the joint optimization in step S4 adopts a phased training strategy: in the first phase, only the neural radiation field model is trained; in the second phase, the neural radiation field parameters are frozen and the diffusion model is trained; in the third phase, end-to-end joint optimization is performed, and convergence is determined when the gradient change rate is less than a preset threshold.

[0018] To achieve the second objective of this invention, the following solution is adopted: A remote sensing 3D reconstruction system based on NeRF and diffusion model includes: The front-end processing module is used to acquire satellite images from multiple time phases or angles and their corresponding camera attitude information, and to perform adaptive geometric correction and multi-constraint radiometric equalization processing on the satellite images to obtain preprocessed images. The neural radiation field module is used to construct a neural radiation field model based on the preprocessed image, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model. The conditional diffusion enhancement module is used to input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image. The three-dimensional output module is used to jointly optimize the neural radiation field model and the conditional diffusion model based on the output image to generate a high-fidelity three-dimensional surface model.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention acquires satellite images from multiple time phases or angles and performs adaptive geometric correction and multi-constraint radiometric equalization processing, effectively eliminating geometric deviations and radiometric inconsistencies between images from multiple time phases and angles, providing high-quality input data for subsequent modeling, thereby overcoming the problem that it is difficult to completely eliminate geometric deviations and radiometric inconsistencies in the prior art.

[0020] 2. This invention organically combines the neural radiation field model with the conditional diffusion model. First, the neural radiation field model is used to learn the continuum density and color field to achieve a high-fidelity representation of the geometric structure. Then, the conditional diffusion model is used to iteratively denoise the rendered image and enhance the lighting consistency. This overcomes the shortcomings of traditional neural radiation fields, which are sensitive to multi-temporal lighting changes and are prone to getting trapped in local optima. It achieves synergistic enhancement of geometric structure learning and appearance lighting modeling.

[0021] 3. This invention generates a high-fidelity 3D surface model by jointly optimizing the neural radiation field model and the conditional diffusion model. Under complex conditions such as texture repetition, terrain undulation, and shadow coverage, it can maintain stable geometric boundaries and rich texture details, significantly improving the signal-to-noise ratio, structural similarity, and texture fidelity of the reconstructed image. Experiments show that each indicator is improved by 10% to 35% compared with existing methods.

[0022] 4. This invention forms an end-to-end processing flow from input to output, eliminating the need for complex intermediate registration steps. It has good adaptability to remote sensing data of different resolutions, different sensors, and different land surface types, and can be widely used in engineering scenarios such as multi-temporal land surface change monitoring, disaster emergency assessment, and cultural heritage digitization. Attached Figure Description

[0023] Figure 1 This is a flowchart of a remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model in an embodiment of the present invention; Figure 2 This is a structural diagram of NeRF in an embodiment of the present invention; Figure 3 This is a structural diagram of the Diffusion model in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the optimization process of the Diffusion model in this embodiment of the invention. Figure 5 This is a flowchart of the optimized NeRf rendering graph based on the Diffusion model in an embodiment of the present invention; Figure 6 This is a block diagram of a remote sensing 3D reconstruction system based on the combination of NeRF and diffusion model in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0025] Example 1 This invention provides a remote sensing 3D reconstruction method based on the combination of NeRF and a diffusion model. By constructing a Neural Radiance Field (NeRF) model and combining it with a diffusion model, it achieves high-fidelity 3D surface structure reconstruction from multi-view satellite imagery. The 3D model generated by this method maintains high accuracy and visual realism even under complex lighting, shadow, and terrain conditions, exhibiting advantages such as high accuracy, strong noise resistance, and excellent cross-temporal adaptability.

[0026] like Figures 1-5 As shown, the remote sensing 3D reconstruction method based on NeRF and diffusion model in this embodiment of the invention includes the following steps: Step S1: Acquire satellite images from multiple time phases or angles and their corresponding camera attitude information, and perform adaptive geometric correction and multi-constraint radiometric equalization processing on the satellite images to obtain preprocessed images.

[0027] In this embodiment, step S1 is a data acquisition and multi-level preprocessing step, which specifically includes the following: Step S1-1: Multi-source data acquisition and quality assessment: This invention performs unified data acquisition, geometric correction, radiometric normalization, and illumination equalization processing on multi-temporal and multi-view remote sensing images to provide high-quality input data for subsequent neural radiation field modeling. During the data acquisition phase, this invention can utilize various remote sensing data sources, including high-resolution satellite imagery (such as WorldView, Pleiades, and GF series), aerial oblique photogrammetry, or UAV multi-angle sequence imagery.

[0028] To ensure complete coverage of the modeling area and sufficient parallax information, data acquisition should simultaneously meet the following conditions: in, The distance between the baselines of two adjacent images. Average flight altitude The viewing angle, This represents the number of valid images.

[0029] For different imaging platforms, this invention uniformly adopts exterior orientation elements (...). The imaging geometry can be described by either the α or RPC (Rational Polynomial Coefficient) model.

[0030] Step S1-2, Adaptive geometric correction and parameter optimization: Geometric correction and orthophoto projection are performed on the original images to eliminate spatial distortions caused by differences in Earth's curvature and orbital attitude. Through RPC polynomial expansion and iterative solution, images from different time phases are unified to the same geographic coordinate reference system (such as WGS84 or CGCS2000), achieving spatial alignment of multi-time-phase and multi-view data.

[0031] The core formula for geometric correction is as follows: in, Image coordinates, For ground coordinates, The coefficients are those of a cubic rational polynomial.

[0032] For images with insufficient attitude accuracy or lacking RPC parameters, feature-matching-based bundle adjustment methods can be used for extrinsic parameter optimization. Let the... Phase Number The reprojection error of the feature points is: in, To observe pixel coordinates, For projected coordinates, extrinsic parameters Iterative optimization is performed by minimizing the total error.

[0033] Steps S1-3: Multi-constraint radiative equilibrium and illumination normalization: In the radiometric and illumination correction stage, considering that remote sensing images are often affected by factors such as shooting time, solar altitude angle, atmospheric scattering and sensor response, resulting in significant differences in brightness, contrast and color between images, this invention introduces an automated multi-constraint radiometric equalization algorithm.

[0034] Let the reference image be The image to be corrected is The color mapping model uses multinomial regression: Among them, coefficient The solution is obtained by minimizing the mean square error: Furthermore, shadow detection and compensation are introduced. The solar illuminance index is defined as: in, The solar altitude angle, The slope angle of the terrain. and This is an adjustable parameter.

[0035] Brightness compensation in shadow areas uses a piecewise linear model: in, To compensate for the increase, The threshold for shadow discrimination.

[0036] This process ensures that images from different times and angles are comparable in terms of radiometric characteristics, thereby improving the stability of the NeRF network input.

[0037] Step S2: Based on the preprocessed image, construct a neural radiation field model, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model.

[0038] In this embodiment, step S2 is the neural radiation field model construction and optimization step, which specifically includes the following: Step S2-1, Position Encoding and Coordinate Transformation: In the NeRF framework, three-dimensional spatial coordinates Through position encoding function Mapping to a high-dimensional feature space enhances the network's ability to capture high-frequency details: in, This refers to the number of coding layers. Considering the characteristics of remote sensing scenarios, this invention employs an adaptive coding frequency strategy: Step S2-2, Volume density and color prediction network: Predicting volume density using a multilayer perceptron (MLP) network With color : in, In three-dimensional coordinates, The direction of the line of sight. The network architecture employs a jump connection design to enhance gradient flow: Step S2-3, Volume rendering and reprojection loss: The volume density and color field are converted into a two-dimensional image using the volume rendering equation: in, Transmittance, Here is the equation for the ray.

[0039] Using numerical integration to approximate the above equation, we can discretize it as follows: in, The sampling interval is... This represents the number of sampling points.

[0040] The reprojection loss takes a hybrid form: in, is the weighting coefficient, and SSIM is the structural similarity measure.

[0041] Step S3: Input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image.

[0042] In this embodiment, step S3 is the illumination uniformity enhancement step of the conditional diffusion model, which specifically includes the following: Step S3-1, Diffusion process and reverse process: A diffusion model is introduced as a post-processing module to denoise and correct the lighting of the NeRF output rendering results. The forward diffusion process cleans the image. Gradually add noise to Gaussian noise : in, , The noise step size is a predefined value.

[0043] The inverse process (denoising) learns the probability distribution of recovering a clean image from noisy samples: in, and Parameterized by neural network.

[0044] Step S3-2, Conditional Generation and Lighting Alignment: To enhance illumination uniformity, this invention introduces a conditional diffusion mechanism, using illumination information from multi-temporal reference images as a condition: in, This is a vector of illumination conditions (including solar altitude angle, azimuth angle, atmospheric brightness, etc.). These are gradient weights.

[0045] The illumination loss function is defined as: in, Let be the illumination estimation function. This is a constraint term related to changes in illumination.

[0046] Step S3-3, Iterative Denoising and Multi-Step Optimization: A phased iterative denoising strategy is adopted to gradually enhance illumination consistency while preserving details: Rough steps Using large step sizes, major noise and lighting differences are quickly eliminated; Detailed steps Use small steps to gradually restore texture details and local contrast.

[0047] Step S4: Based on the output image, jointly optimize the neural radiation field model and the conditional diffusion model to generate a high-fidelity three-dimensional surface model.

[0048] In this embodiment, step S4 is the system integration and end-to-end optimization step, which specifically includes the following: Step S4-1, Complete Piping Design: The end-to-end optimization framework for the entire system is as follows: in: Parameter settings: .

[0049] Step S4-2, Training Strategy and Convergence Metric: A phased training strategy is adopted, first converging the NeRF module, then introducing the diffusion module, and finally performing joint optimization: Phase 1 (Steps 1-30k): Train only the NeRF model, with the objective being to minimize... Learning rate ; Phase 2 (steps 30k-50k): Freeze NeRF parameters, train the diffusion module, and aim to minimize... Learning rate ; Phase 3 (steps 50k-100k): Joint optimization, with the objective of minimizing... Learning rate .

[0050] The convergence metric is defined as: in, .

[0051] The performance indicators and application scenarios of the remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model of the present invention are described below.

[0052] The reconstruction quality of this invention is evaluated using the following indicators: This invention is particularly suitable for the following application scenarios: 1. Multi-temporal surface change monitoring: Dynamic monitoring of urban expansion, vegetation evolution, etc., using time-series remote sensing imagery. 2. 3D reconstruction of complex terrain: Achieving high-precision, high-detail 3D model reconstruction in complex terrains such as mountains and hills. 3. Disaster emergency assessment: Rapidly acquiring 3D scene information of disaster areas to support emergency response decisions. 4. Precision agriculture applications: Monitoring crop growth and providing precise fertilization guidance based on multi-angle UAV imagery. 5. Digitization of cultural heritage: High-fidelity 3D reconstruction and digital archiving of cultural heritage such as ancient buildings and archaeological sites.

[0053] This invention combines deep learning's neural implicit representation method with a generative diffusion model to construct an end-to-end optimizable 3D reconstruction workflow tailored to the unique imaging characteristics of remote sensing images. The entire workflow employs a five-layer architecture, using multi-temporal and multi-angle remote sensing images as input, and completes the reconstruction from image to 3D scene through the following four core stages: 1. Front-end processing stage: Adaptive geometric correction and radiometric equalization ensure spatial and radiometric consistency of the input data. 2. Geometric modeling stage: Learning the continuum density field and radiation field through NeRF to achieve geometric representation and viewpoint-consistent rendering. 3. Illumination compensation stage: Introducing a conditional diffusion module for feature alignment and illumination correction to eliminate multi-temporal radiometric differences. 4. Detail enhancement stage: Achieving high-fidelity 3D surface reconstruction through iterative denoising and texture restoration using the diffusion model. Specifically, the NeRF model is responsible for reconstructing the continuum density field and radiation field from sparse image information; the diffusion model refines the appearance and denoises the rendering results through a conditional generation mechanism, achieving illumination consistency and texture restoration.

[0054] The software implementation environment of this invention can be a combination of Python 3.9+, PyTorch 2.0+, and CUDA 11.8+, and depends on scientific computing libraries such as NumPy, OpenCV, and Scikit-image.

[0055] The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model in this invention introduces a diffusion model to enhance the appearance and correct the illumination consistency of NeRF. Through the collaborative mechanism of "NeRF being responsible for geometric structure learning and diffusion model being responsible for appearance and illumination modeling", a highly robust and high-fidelity remote sensing 3D reconstruction framework is formed, which effectively improves the texture details, illumination consistency and multi-temporal adaptability of the surface model.

[0056] The remote sensing 3D reconstruction method based on NeRF and diffusion model proposed in this invention effectively solves the core problems of traditional remote sensing 3D reconstruction, such as inconsistent illumination, loss of detail, and poor cross-temporal adaptability, by introducing adaptive data preprocessing, conditional diffusion enhancement, and multi-module collaborative optimization. This invention has advantages such as high accuracy, strong robustness, and good generalization ability, and has significant theoretical and engineering application value.

[0057] This invention presents a remote sensing 3D reconstruction method based on NeRF and a diffusion model. This method employs a multi-module collaborative design and introduces a diffusion generation model into remote sensing 3D reconstruction tasks for the first time. Through the collaborative mechanism of neural radiation fields and diffusion models, it overcomes the performance bottlenecks of traditional NeRF under illumination variations, shadow interference, and cross-temporal radiation drift, achieving joint optimization of geometric structure and appearance illumination. This invention's method, through geometric constraint optimization combined with satellite imaging geometry (RPC model) and ray sampling optimization strategies, effectively improves sampling density and network convergence speed in large-scale scenes, solving the problem of difficulty in deep feature learning caused by the limited parallax range of satellite images and sparse ray sampling. Furthermore, this invention employs adaptive data processing, introducing multi-stage adaptive geometric correction, multi-constraint radiation equalization, and dynamic illumination compensation, significantly improving the spatial and radiation consistency of the input data and providing a high-quality input foundation for subsequent neural radiation field modeling and diffusion enhancement. The remote sensing 3D reconstruction method based on NeRF and diffusion model in this invention achieves high-fidelity restoration and reconstructs the detailed structure of the earth's surface. It can maintain stable reconstruction performance even under complex terrain and intense lighting conditions, and significantly improves the integrity of texture details and visual realism.

[0058] The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model in this invention has the following advantages: 1. The embodiments of this invention adopt an innovative multi-module collaborative architecture, which for the first time organically integrates the diffusion model into the remote sensing 3D reconstruction framework, forming a dual-path collaborative mechanism of geometry and illumination. This effectively breaks through the performance bottleneck of traditional NeRF under illumination changes, shadow interference and multi-temporal radiation drift, and significantly improves the reconstruction accuracy and visual realism of the surface model.

[0059] 2. The embodiments of the present invention construct a comprehensive multi-stage preprocessing system, and propose a complete data preprocessing process of adaptive geometric correction, multi-constraint radiometric equalization and dynamic illumination compensation, to ensure the high quality and consistency of input data, and to lay a solid foundation for subsequent 3D reconstruction.

[0060] 3. The embodiments of the present invention establish a rigorous mathematical model and formula definition. By clearly defining the calculation formula and optimization objective function of each key step, the reproducibility and engineering feasibility of the solution are greatly improved, making it easier for those skilled in the art to deploy and apply it on different platforms and scenarios.

[0061] 4. The embodiments of the present invention achieve significant improvements in accuracy and robustness. Comprehensive experimental verification shows that, compared with the benchmark NeRF method, the present scheme has significant improvements of 10%-35% in indicators such as peak signal-to-noise ratio, structural similarity, and texture fidelity, and its advantages are particularly prominent in complex lighting and multi-temporal scenarios.

[0062] 5. The embodiments of the present invention have good generalization and scalability, and have good adaptability to remote sensing data of different resolutions, different sensors and different land surface types. They do not require tedious parameter tuning or retraining for specific data sources, and show strong engineering application prospects.

[0063] Example 2 like Figure 6 As shown, embodiments of the present invention also provide a remote sensing 3D reconstruction system based on a combination of NeRF and a diffusion model, comprising: The front-end processing module is used to acquire satellite images from multiple time phases or angles and their corresponding camera attitude information, and to perform adaptive geometric correction and multi-constraint radiometric equalization processing on the satellite images to obtain preprocessed images.

[0064] The neural radiation field module is used to construct a neural radiation field model based on the preprocessed image, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model.

[0065] The conditional diffusion enhancement module is used to input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image.

[0066] The three-dimensional output module is used to jointly optimize the neural radiation field model and the conditional diffusion model based on the output image to generate a high-fidelity three-dimensional surface model.

[0067] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model, characterized in that, Includes the following steps: Step S1: Acquire satellite images from multiple time phases or angles and their corresponding camera attitude information; perform adaptive geometric correction and multi-constraint radiometric equalization on the satellite images to obtain preprocessed images. Step S2: Based on the preprocessed image, construct a neural radiation field model, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model. Step S3: Input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image; Step S4: Based on the output image, jointly optimize the neural radiation field model and the conditional diffusion model to generate a high-fidelity three-dimensional surface model.

2. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The adaptive geometric correction in step S1 includes: using a rational polynomial coefficient model or a feature-matching-based bundle adjustment method to unify satellite images from different time phases or multiple angles to the same geographic coordinate reference system, and optimizing the exterior orientation parameters by minimizing the reprojection error.

3. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The multi-constraint radiometric equalization processing in step S1 includes: using polynomial regression to perform color mapping on satellite images, introducing shadow detection and compensation, and calculating the brightness compensation increment of shadow areas based on the solar elevation angle and terrain slope angle, so that satellite images at different times or angles have consistent radiometric characteristics.

4. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, In step S2, when constructing the neural radiation field model, an adaptive coding frequency strategy is used to encode the position of the three-dimensional spatial coordinates. The number of coding layers is divided into three levels: high, medium, and low, according to the surface scale range. The neural network adopts a multilayer perceptron and includes a skip connection structure to fuse the intermediate layer features with the original position code.

5. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The volume rendering in step S2 uses numerical integration to discretize the product of transmittance and color. The reprojection loss is a weighted sum of mean square error and structural similarity measure, with the mean square error weight being 0.8 and the structural similarity weight being 0.

2.

6. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, In step S3, the conditional diffusion model progressively adds Gaussian noise to the image during the forward process and learns to recover the image from the noise during the reverse process. The illumination information is used as a conditional vector, which includes the solar altitude angle, azimuth angle, and atmospheric brightness. The denoising process is guided by an illumination loss function, which includes an illumination estimation difference term and an illumination change constraint term.

7. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The iterative denoising in step S3 further adopts a phased strategy: in the coarse step stage, a large step size is used to eliminate the main noise and illumination differences, and in the fine step stage, a small step size is used to restore texture details and local contrast.

8. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The joint optimization in step S4 adopts an end-to-end total loss function, which is a weighted sum of neural radiation field reprojection loss, diffusion model negative log-likelihood loss and consistency regularization loss, wherein the diffusion model loss weight is 0.5, the consistency loss weight is 0.3, and the regularization loss weight is 0.

1.

9. The remote sensing 3D reconstruction method based on the combination of NeRF and diffusion model according to claim 1, characterized in that, The joint optimization in step S4 adopts a phased training strategy: the first phase trains only the neural radiation field model, the second phase freezes the neural radiation field parameters and trains the diffusion model, and the third phase performs end-to-end joint optimization and determines convergence when the gradient change rate is less than a preset threshold.

10. A remote sensing 3D reconstruction system based on the combination of NeRF and diffusion model, characterized in that, include: The front-end processing module is used to acquire satellite images from multiple time phases or angles and their corresponding camera attitude information, and to perform adaptive geometric correction and multi-constraint radiometric equalization processing on the satellite images to obtain preprocessed images. The neural radiation field module is used to construct a neural radiation field model based on the preprocessed image, generate rays along the line of sight and sample voxel points on the rays, predict the volume density and color of each voxel point through a neural network, generate a rendered image with consistent viewpoint through volume rendering, and calculate the reprojection loss between the rendered image and the real image to optimize the neural radiation field model. The conditional diffusion enhancement module is used to input the rendered image into the conditional diffusion model, and perform iterative denoising and illumination consistency enhancement based on the illumination information of the reference image to obtain the enhanced output image. The three-dimensional output module is used to jointly optimize the neural radiation field model and the conditional diffusion model based on the output image to generate a high-fidelity three-dimensional surface model.

Images