Three-dimensional reconstruction method based on neural radiation field

Abstract

The invention provides a three-dimensional reconstruction method based on a neural radiation field. The method comprises the following steps: S1, screening out a plurality of primary common-view images; s2, carrying out super-resolution processing on each primary common-view image to form a plurality of groups of advanced common-view image pairs; s3, performing feature extraction and matching on each group of advanced common-view image pairs to obtain multiple groups of homonymy point pairs and three-dimensional coordinates; s4, repairing the original RPC parameters of each advanced common-view image by using the optimized RPC model to obtain a plurality of repaired RPC parameters; s5, acquiring a camera ray of each pixel in each advanced common-view image based on the repaired RPC parameters, and determining a 3D point sequence of each camera ray; s6, inputting each group of 3D point sequence set into the trained Sat-Nerf model, predicting the color value and volume density of each 3D point, and constructing a sigma model; and realizing three-dimensional reconstruction of the target area based on a sigma model. According to the method, the Sat-Nerf model is adopted for three-dimensional reconstruction, the volume density, the reflected light, the shadow gray value, the ambient light and other multi-dimensional information of each 3D point are predicted, and the reconstruction capability of a complex scene is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a three-dimensional reconstruction method based on neural radiation fields. Background Art

[0002] The core goal of Neural Radiance Fields (NeRF) is to solve the problem of view synthesis in complex scenes. It employs a multi-layer perceptron (MLP) network to approximate a continuous 5D scene representation. View synthesis is achieved by projecting the color and density information output by the network onto the image using a large number of input views and known camera poses using classic volume rendering techniques. However, when applied to satellite imagery, the camera model differs significantly from that of traditional close-range imagery. The imaging process is more complex, involving multiple factors such as the satellite's orbit, pose, and sensor characteristics. The approximate pinhole camera model commonly used in NeRF cannot accurately describe the imaging geometry of satellite cameras, resulting in significant errors when processing satellite imagery. Furthermore, satellite imagery typically covers a wide area, with a significant distance between the camera and the scene. This results in relatively low resolution and significant geometric distortion of objects in the image. NeRF performs poorly in processing such large-scale, low-resolution, and highly distorted scenes.

[0003] Sat-NeRF is a method tailored for multi-view satellite imagery. Taking into account the characteristics of satellite imagery, it uses rational polynomial coefficients (RPC) functions to describe the camera's imaging process. This model accurately reflects the complex geometric relationships of satellite cameras, giving Sat-NeRF a natural advantage in processing satellite imagery. The RPC model can better adapt to the imaging characteristics of complex terrain and buildings, accurately calculating the propagation paths and intersection locations of rays in the scene. Sat-NeRF can also effectively address the radiometric inconsistencies found in satellite imagery.

[0004] However, network reconstruction based only on sparse rays sampled from two images will result in low accuracy in recovering the structure and scale information of a single building target, lack of capturing details of the remote sensing scene, and its three-dimensional reconstruction accuracy needs to be improved. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a three-dimensional reconstruction method based on neural radiation field, which includes:

[0006] S1 obtains multiple primary images and ground information data of the target area in multiple time series and multiple perspectives, and selects multiple primary common view images that meet the common view conditions in the target area;

[0007] S2 uses the trained super-resolution model to perform super-resolution processing on each primary common view image in turn to obtain multiple high-level common view images and form multiple groups of high-level common view image pairs;

[0008] S3 extracts and matches features of each set of advanced common view image pairs in turn to obtain multiple sets of homonymous point pairs and first pixel coordinates; based on the multiple sets of homonymous point pairs and ground signal data, the first pixel coordinates are converted into multiple sets of three-dimensional coordinates in turn;

[0009] S4 optimizes the initial RPC model based on multiple sets of homonymous point pairs and 3D coordinates using a combined L1 and L2 regularization method to obtain an optimized RPC model. The optimized RPC model is then used to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters.

[0010] S5 obtains the camera ray corresponding to each pixel in each advanced common view image according to the imaging view angle and multiple restoration RPC parameters to obtain multiple camera rays; and sequentially determines the 3D point sequence of each camera ray according to a preset step size to obtain a 3D point sequence set;

[0011] S6 inputs the 3D point sequence set and geospatial data into the trained Sat-Nerf model, predicts the color value and volume density of each 3D point, and constructs the sigma model and implicit function of the entire target area; based on the sigma model and implicit function, it realizes the three-dimensional reconstruction of the target area.

[0012] Specifically, the Sat-Nerf model in step S6 includes a network base network, a static network, a transient network and an auxiliary network; wherein the base network includes an 8-layer MLP neural network, each layer of MLP has 512 neurons, and each layer of MLP performs nonlinear transformation through an activation function. The base network is used to predict the volume density and reflected light of each 3D point; the static network includes a 3-layer MLP neural network, each layer of MLP has 256 neurons, and the static network is used to predict the shadow grayscale of each 3D point; the transient network includes a 1-layer MLP neural network with 256 neurons in the layer, which is used to predict the embedding vector of the changing object; the auxiliary network includes a 1-layer MLP neural network with 256 neurons in the layer, which is used to predict the ambient light of each 3D point.

[0013] Specifically, the color value of each 3D point is calculated by the reflected light, ambient light and shadow grayscale, and the specific formula is:

[0014] s i =α i ·(β i +(1-β i )·μ i )

[0015] Among them, s iis the color value of the i-th 3D point, β i is the shadow grayscale of the i-th 3D point, μ i is the ambient light of the i-th 3D point, α i is the reflected light of the i-th 3D point.

[0016] Specifically, the method for obtaining the trained Sat-Nerf model in step S6 includes:

[0017] Acquire multiple sets of sample image pairs and sample data in multiple time series and multiple perspectives; the sample data includes multiple sets of same-name point pairs and their corresponding multiple sets of three-dimensional sample coordinates, the true color value of each sample image, and RPC sample parameters;

[0018] Obtain a camera sample ray corresponding to each pixel in each sample image according to the imaging view angle and multiple RPC sample parameters to obtain multiple camera sample rays; and determine the sample point sequence of each camera sample ray in sequence according to a preset step size to obtain a sample point sequence set;

[0019] Input the sample point sequence into the initial Sat-Nerf model, adjust the model parameters based on the sample data according to the comprehensive loss function, and generate the trained Sat-Nerf model when the model parameters converge;

[0020] Specifically, the comprehensive loss function is:

[0021] L=L RGB (R)+δ SC L SC (R SC )+δ DS L DS (R DS )

[0022] Among them, L RGB (R) is the color loss function, L SC (R SC ) is the sunlight correction loss function, L DS (R DS ) is the depth estimation supervision loss function; δ SC and δ DS are the weights of the sunlight correction loss function and the depth estimation supervision loss function, respectively.

[0023] Specifically, step S6 further includes:

[0024] S61 inputs the 3D point sequence set into the trained Sat-Nerf model and predicts the volume density value and color value corresponding to each 3D point sequence;

[0025] S62 constructs a sigma model and an implicit function of the entire target area, and converts the volume density values ​​based on all 3D points into a three-dimensional point cloud based on the sigma model and the implicit function;

[0026] Based on the 3D point cloud, S63 uses a surface reconstruction algorithm to build a digital surface model of the target area, maps color values ​​to the digital surface model, and realizes 3D reconstruction of the target area.

[0027] Specifically, the common viewing conditions are that the satellite zenith angle of each primary image is less than 30°, the angle between any two primary images is less than 5°, and the shadow area of ​​each primary image accounts for less than 5%.

[0028] Specifically, step S3 further includes:

[0029] S31 uses the SIFT algorithm to extract and match features of each set of advanced common view image pairs in turn, obtaining multiple sets of same-name point pairs and corresponding first pixel coordinates;

[0030] S32 converts the first pixel coordinate into a geographic coordinate in sequence based on the multiple sets of same-name point pairs and the geospatial data to obtain the longitude and latitude coordinates of the multiple sets of same-name point pairs;

[0031] S33 interpolates the longitude and latitude coordinates of multiple sets of same-name point pairs according to the geospatial data to obtain multiple sets of three-dimensional coordinates.

[0032] Specifically, step S4 further includes:

[0033] S41 uses the same-name point pairs corresponding to the three-dimensional coordinates as control points to construct an RPC model, and performs positive calculations on the original RPC parameters of each advanced common view image and the three-dimensional coordinates to obtain the second pixel coordinates on the advanced common view image corresponding to the three-dimensional coordinates;

[0034] S42 calculates the error distance between the first pixel coordinate and the second pixel coordinate, adopts a regularization method combining L1 and L2 based on the least squares adjustment principle, introduces a set linear constraint condition, and obtains an optimized RPC model;

[0035] S43 repairs the original RPC parameters of each advanced common view image using the optimized RPC model according to the error distance to obtain a plurality of repaired RPC parameters;

[0036] S42 uses the optimized RPC model to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters.

[0037] Specifically, the super-resolution model in step S2 includes a residual hybrid attention network and an edge enhancement network; each primary common view image is super-reconstructed by the residual hybrid attention network to obtain multiple intermediate common view images; the edge of each intermediate common view image is optimized by the edge enhancement network to obtain multiple high-level common view images.

[0038] The beneficial effects of the present invention include at least:

[0039] (1) The present invention uses the Sat-Nerf model for three-dimensional reconstruction, which can predict multi-dimensional information such as volume density, reflected light, shadow grayscale value, ambient light and uncertainty coefficient of each 3D point, so that the model can more comprehensively describe the scene characteristics of the target area, thereby enhancing the reconstruction capability of complex scenes;

[0040] (2) The present invention improves upon the NeRF model by utilizing the sun rays and moving object embedding vector models in geospatial data as input. This makes the model more adaptable and robust to images taken at different times and under different lighting conditions, and can better ignore the interference caused by transient objects, thereby improving the accuracy of 3D reconstruction.

[0041] (3) The present invention filters multi-time series and multi-view images through common view conditions and processes them using a super-resolution model. The generative adversarial network makes the images sharper and clearer. The residual hybrid attention network can expand the available range of pixels and restore more high-frequency information to obtain high-quality advanced common view images.

[0042] (4) The present invention adopts steps such as super-resolution processing and feature matching, and combines them with the optimization of the RPC model to eliminate problems such as the reduction of geometric positioning accuracy caused by model errors in the RPC model. It can obtain the geometric information of the scene more accurately, thereby effectively improving the accuracy of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a flow chart of the three-dimensional reconstruction method based on neural radiation field in Example 1 of the present invention;

[0045] Figure 2 Schematic diagram of the Sat-Nerf model structure in Example 1 of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention are within the scope of protection of the present invention.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] See also Figure 1 , Figure 1 This is a flow chart of a three-dimensional reconstruction method based on neural radiation fields in Example 1 of the present invention, which includes:

[0049] S1 obtains multiple primary images and ground information data of the target area in multiple time series and multiple perspectives, and selects multiple primary common view images that meet the common view conditions in the target area;

[0050] In an embodiment of the present invention, the common viewing conditions in step S1 are that the satellite zenith angle of each primary image is less than 30°, the angle between any two primary images is less than 5°, and the shadow area of ​​each primary image accounts for less than 5%, and 80 primary images from different perspectives of the target area are obtained.

[0051] S2 uses the trained super-resolution model to perform super-resolution processing on each primary common view image in turn to obtain multiple high-level common view images and form multiple groups of high-level common view image pairs;

[0052] In an embodiment of the present invention, the super-resolution model in step S2 includes a residual hybrid attention network and an edge enhancement network; each primary common view image is super-reconstructed by the residual hybrid attention network to obtain multiple intermediate common view images; the edge of each intermediate common view image is optimized by the edge enhancement network to obtain multiple high-level common view images. The residual hybrid attention network includes a shallow feature extraction module, a deep feature extraction module and an image reconstruction module, wherein the deep feature extraction module includes n cascaded residual hybrid attention groups and a first convolution layer; wherein the residual hybrid attention group includes m cascaded hybrid attention convolution blocks, overlapping cross attention blocks and a second convolution layer with residual connections, where m is a positive integer; wherein the hybrid attention convolution block includes a multi-head self-attention block and a channel attention block arranged in parallel, and a deep convolution layer connected after the multi-head self-attention block and the channel attention block; wherein the edge enhancement network includes a Laplacian operator, strided convolution, a residual dense subnetwork and sub-pixel convolution, wherein the residual dense subnetwork includes multiple residual nested residual dense blocks, the residual nested residual dense blocks include multiple residual dense blocks, the residual dense blocks are connected as residual connections, and each residual operation of the residual nested residual dense blocks is set with a weighting coefficient.

[0053] Specifically, it includes: downsampling multiple primary common view images in sequence to obtain downsampled common view images; super-reconstructing the downsampled common view images through the residual hybrid attention network to obtain multiple intermediate common view images; and optimizing the edges of the intermediate common view images through the edge enhancement network to obtain multiple high-level common view images.

[0054] Multi-time series and multi-view images are screened through common view conditions and processed using a super-resolution model. The images are made sharper and clearer through generative adversarial networks. The residual hybrid attention network can expand the available pixel range and restore more high-frequency information to obtain high-quality advanced common view images.

[0055] S3 extracts and matches features of each set of advanced common view image pairs in turn to obtain multiple sets of homonymous point pairs and first pixel coordinates; based on the multiple sets of homonymous point pairs and ground signal data, the first pixel coordinates are converted into multiple sets of three-dimensional coordinates in turn;

[0056] In this embodiment of the present invention, step S3 further includes:

[0057] S31 uses the SIFT algorithm to extract and match features of each set of advanced common view image pairs in turn, obtaining multiple sets of same-name point pairs and corresponding first pixel coordinates;

[0058] S32 converts the first pixel coordinate into a geographic coordinate in sequence based on the multiple sets of same-name point pairs and the geospatial data to obtain the longitude and latitude coordinates of the multiple sets of same-name point pairs;

[0059] S33 interpolates the longitude and latitude coordinates of multiple sets of same-name point pairs according to the geospatial data to obtain multiple sets of three-dimensional coordinates.

[0060] In an embodiment of the present invention, the SIFT algorithm is used to extract features from each set of advanced common view image pairs in turn to obtain multiple feature points. The feature points are then homogenized using a greedy algorithm to obtain a final uniformly distributed pair of matching points, thereby obtaining the first pixel coordinates (x1, y1) corresponding to each matching point in each advanced common view image. The first pixel coordinates of the matching points corresponding to each advanced common view image are converted into latitude and longitude coordinates (B, L) based on geospatial data.

[0061] In an embodiment of the present invention, the affine transformation matrix of the orthophoto image is obtained based on the geospatial data to achieve the conversion between the first pixel coordinates and the latitude and longitude coordinates of the image, as shown in the following formula:

[0062] X=Ax1+By1+C

[0063] Y=Dx1+Ey1+F

[0064] Calculate the longitude and latitude coordinates of the points with the same name according to the above formula. In the formula, (x1, y1) is the first pixel coordinate, (X, Y) is the longitude and latitude coordinates, The high-level common view image parameters in the geospatial data are then used to interpolate the elevation H corresponding to the feature point in the DEM using raster data interpolation methods such as bilinear interpolation and bicubic convolution interpolation to obtain the three-dimensional coordinates (X, Y, H).

[0065] S4 optimizes the initial RPC model based on multiple sets of homonymous point pairs and 3D coordinates using a combined L1 and L2 regularization method to obtain an optimized RPC model. The optimized RPC model is then used to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters.

[0066] In this embodiment of the present invention, step S4 further includes:

[0067] S41 uses the same-name point pairs corresponding to the three-dimensional coordinates as control points to construct an RPC model, and performs positive calculations on the original RPC parameters of each advanced common view image and the three-dimensional coordinates to obtain the second pixel coordinates on the advanced common view image corresponding to the three-dimensional coordinates;

[0068] S42 calculates the error distance between the first pixel coordinate and the second pixel coordinate, adopts a regularization method combining L1 and L2 based on the least squares adjustment principle, introduces a set linear constraint condition, and obtains an optimized RPC model;

[0069] S43 repairs the original RPC parameters of each advanced common view image using the optimized RPC model according to the error distance to obtain a plurality of repaired RPC parameters;

[0070] S42 uses the optimized RPC model to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters.

[0071] In an embodiment of the present invention, an RPC model is first constructed, and the original RPC parameters of each advanced common view image are subjected to RPC positive calculation with the three-dimensional coordinates to obtain the second pixel coordinates corresponding to the three-dimensional coordinates on the advanced common view image, and the mean of the error distances of the corresponding first pixel coordinates and the second pixel coordinates are respectively calculated; on the basis of the least squares adjustment principle, an L1 and L2 regularization optimization method is used to introduce a set linear constraint condition to obtain an optimized RPC model; a regularization method combining L1 and L2 is proposed, and a sparse model is generated based on the robust L1, L2 norm estimation principle. L1 regularization can produce a sparse solution, while L2 regularization can prevent overfitting. The combination of the two is to achieve a balance between sparsity and robustness, and can then be used for feature selection. By introducing the set linear constraint condition, the RPC model is optimized.

[0072] In the embodiments of the present invention, super-resolution processing and feature matching are adopted, and the RPC model is optimized to eliminate problems such as reduced geometric positioning accuracy caused by model errors in the RPC model. The geometric information of the scene can be obtained more accurately, thereby effectively improving the accuracy of three-dimensional reconstruction.

[0073] S5 obtains the camera ray corresponding to each pixel in each advanced common view image according to the imaging perspective and multiple restoration RPC parameters to obtain multiple camera rays; and determines the 3D point sequence of each camera ray in sequence according to the preset step size to obtain a 3D point sequence set.

[0074] Based on the restored RPC parameters from multiple advanced common view images, a camera ray is constructed for each pixel in the advanced common view image, based on the pixel's 3D coordinates and the metadata of the remote sensing image it resides in. After determining the 3D point sequence based on a preset step size, the server then uses the trained neural radiation field model to adjust the transparency of the camera ray output, increasing the weights between points where the camera ray transmittance changes and decreasing the weights between 3D point sequences where the transmittance remains unchanged. This process is repeated to obtain more effective 3D point sequences for each camera ray.

[0075] S6 inputs the 3D point sequence set and geospatial data into the trained Sat-Nerf model, predicts the color value and volume density of each 3D point, and constructs the sigma model and implicit function of the entire target area; based on the sigma model and implicit function, it realizes the three-dimensional reconstruction of the target area.

[0076] In an embodiment of the present invention, the Sat-Nerf model in step S6 includes a network base network, a static network, a transient network and an auxiliary network; wherein the base network includes an 8-layer MLP neural network, each layer of MLP has 512 neurons, and each layer of MLP performs nonlinear transformation through an activation function. The base network is used to predict the volume density and reflected light of each 3D point; the static network includes a 3-layer MLP neural network, each layer of MLP has 256 neurons, and the static network is used to predict the shadow grayscale of each 3D point; the transient network includes a 1-layer MLP neural network, with 256 neurons in the layer, for predicting the embedding vector of the changing object; the auxiliary network includes a 1-layer MLP neural network, with 256 neurons in the layer, for predicting the ambient light of each 3D point.

[0077] In an embodiment of the present invention, step S6 further includes:

[0078] S61 inputs the 3D point sequence set into the trained Sat-Nerf model to predict the volume density value and color value corresponding to each 3D point.

[0079] For each 3D point three-dimensional coordinate, the position encoding function is used to encode the three-dimensional coordinates, and the three-dimensional coordinates encoded in the 3D point sequence are input into the trained neural radiation field model. The input of the trained neural radiation field model is the three-dimensional coordinates of each 3D point in the 3D point sequence M = (M X ,M Y ,M H )∈R 3 And the direction of sunlight in the ground information data W=(W X ,W Y ,W H )∈R 3 and the embedding vector ρ of the changing object, where R 3 Represents 3D, and the output is the volume density value σ∈R for each 3D point + , reflected light α=(α r ,α g ,α b )∈R 3 , shadow grayscale β∈R + And the ambient light μ=(μ r ,μ g ,μ b )∈R 3 ;

[0080] The embedding vector ρ for changing objects refers to local features that undergo sudden changes between multiple time series images. The type and number of some features in images of the same area captured at different times can change, causing image grayscale variations. This variation cannot be explained by the sun's position and surface albedo, so an embedding vector ρ for changing objects is artificially introduced. When ignoring changing objects, the algorithm must express the grayscale variations caused by changing objects using shadow grayscale s and volume density σ. However, introducing ρ allows the algorithm to somewhat ignore the grayscale perturbations caused by changing objects, improving depth estimation accuracy.

[0081] The color value s of each 3D point i By reflected light α i , ambient light μ i and shadow grayscale β i The specific formula is calculated as follows:

[0082] s i =α i ·(β i +(1-β i )·μ i )

[0083] S62 constructs a sigma model and implicit function of the entire target area, and converts the volume density values ​​based on all 3D points into a three-dimensional point cloud based on the sigma model.

[0084] Based on the 3D point cloud, S63 uses a surface reconstruction algorithm to build a digital surface model of the target area, maps color values ​​to the digital surface model, and realizes 3D reconstruction of the target area.

[0085] Obtain the volume density value of each 3D point, combine the volume density values ​​of all 3D points into a voxel grid, build a sigma model, and convert the volume density values ​​based on all 3D points into a three-dimensional point cloud to represent the volume density distribution of the target area.

[0086] Under Sat-NeRF, the implicit function is described as:

[0087] F: (M, W, T) → F: (σ, α, β, μ).

[0088] Implicit functions are used to predict the color and other attributes of each 3D point, and the color values ​​are mapped to the digital surface model to generate realistic texture maps.

[0089] In an embodiment of the present invention, improvements are made on the basis of the NeRF model, and the sunlight and changing object embedding vectors in the geospatial data are used as model inputs. This makes the model more adaptable and robust when facing images at different times and under different lighting conditions, and can better ignore the interference caused by transient objects, thereby improving the accuracy of three-dimensional reconstruction; the Sat-Nerf model is used for three-dimensional reconstruction, which can predict multi-dimensional information such as the volume density, reflected light, shadow grayscale value, ambient light and embedding vector of changing objects of each 3D point, so that the model can more comprehensively describe the scene characteristics of the target area, thereby enhancing the reconstruction capability of complex scenes.

[0090] In an embodiment of the present invention, the method for obtaining the trained Sat-Nerf model in step S6 includes:

[0091] Acquire multiple sets of sample image pairs and sample data in multiple time series and multiple perspectives; the sample data includes multiple sets of same-name point pairs and their corresponding multiple sets of three-dimensional sample coordinates, the true color value of each sample image, and RPC sample parameters;

[0092] Obtain a camera sample ray corresponding to each pixel in each sample image according to the imaging view angle and multiple RPC sample parameters to obtain multiple camera sample rays; and determine the sample point sequence of each camera sample ray in sequence according to a preset step size to obtain a sample point sequence set;

[0093] Input the sample point sequence into the initial Sat-Nerf model, adjust the model parameters based on the sample data according to the comprehensive loss function, and generate the trained Sat-Nerf model when the model parameters converge;

[0094] In an embodiment of the present invention, the comprehensive loss function is:

[0095] L=L RGB (R)+δ SC L SC (R SC )+δ DS L DS (R DS )

[0096] Among them, L RGB (R) is the color loss function, L SC (R SC ) is the sunlight correction loss function, L DS (R DS ) is the depth estimation supervision loss function; δ SC and δ DS are the weights of the sunlight correction loss function and the depth estimation supervision loss function, which are 0.1 / 3 and 1000 / 3 respectively.

[0097] The color loss is used to measure the difference between the predicted color value of the output sample image and the actual color value of the sample image. By minimizing this difference, the color prediction ability of the model is optimized to ensure that the 3D reconstruction result is as consistent as possible with the sample image in color. The color loss function is:

[0098]

[0099] in, e(u) is the predicted color value of the u-th sample, e GT (u) is the true color value of the sample image of the u-th sample, ρ′(u) is the embedding vector of the changing object of the u-th sample, and ω is a constant.

[0100] The sunlight regularization loss is used to ensure the consistency of shadows and optimize the shadow prediction ability of the model by comparing the difference between the predicted shadow scalar and the transmittance. The sunlight correction loss function is:

[0101]

[0102] Among them, T i is the transmittance of the i-th sample point, S i is the shadow scalar of the i-th sample point, N SC is the number of sample points, R SC is the number of sample images.

[0103] The depth supervision loss is used to supervise the neural network's prediction of depth, and optimizes the model's depth prediction ability by comparing the difference between the predicted depth value and the actual depth value. The depth estimation supervision loss function is:

[0104]

[0105] Where d(u) is the predicted depth value of the u-th sample image, ||i(u)-o(u)||2 is the true depth value of the u-th sample image, τ(u) is the contribution weight of i(u) to the depth supervision information, which is calculated by the reprojection error at point i(u) during the adjustment process, and i(u) is the 3D point with the i-th known three-dimensional coordinates in the u-th sample image.

[0106] In an embodiment of the present invention, the initial Sat-NeRF is trained using a comprehensive loss function that combines a color loss, a sunlight correction loss, and a depth estimation supervision loss. This multi-objective optimization strategy enables the model to simultaneously consider color consistency, shadow consistency, and depth accuracy during training. This more comprehensively constrains the model's learning process, accelerates model convergence, and improves training efficiency. This enables Sat-NeRF to generate high-quality 3D reconstruction results and adjust the network size and complexity to meet different needs.

[0107] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A three-dimensional reconstruction method based on neural radiation field, characterized in that: The method comprises: S1 obtains multiple primary images and ground information data of the target area in multiple time series and multiple perspectives, and selects multiple primary common view images that meet the common view conditions in the target area; S2 uses the trained super-resolution model to perform super-resolution processing on each primary common view image in turn to obtain multiple high-level common view images and form multiple groups of high-level common view image pairs; S3 extracts and matches features of each set of advanced common view image pairs in turn to obtain multiple sets of homonymous point pairs and first pixel coordinates; based on the multiple sets of homonymous point pairs and ground signal data, the first pixel coordinates are converted into multiple sets of three-dimensional coordinates in turn; S4 optimizes the initial RPC model based on multiple sets of homonymous point pairs and 3D coordinates using a combined L1 and L2 regularization method to obtain an optimized RPC model. The optimized RPC model is then used to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters. S5 obtains the camera ray corresponding to each pixel in each advanced common view image according to the imaging view angle and multiple restoration RPC parameters to obtain multiple camera rays; and sequentially determines the 3D point sequence of each camera ray according to a preset step size to obtain a 3D point sequence set; S6 inputs the 3D point sequence set and geospatial data into the trained Sat-Nerf model, predicts the color value and volume density of each 3D point, and constructs the sigma model and implicit function of the entire target area; based on the sigma model and implicit function, it realizes the three-dimensional reconstruction of the target area.

2. The method according to claim 1, characterized in that The Sat-Nerf model in step S6 includes a basic network, a static network, a transient network, and an auxiliary network; wherein the basic network includes an 8-layer MLP neural network, each layer of which has 512 neurons, and each layer of MLP performs nonlinear transformation through an activation function. The basic network is used to predict the volume density and reflected light of each 3D point; the static network includes a 3-layer MLP neural network, each layer of which has 256 neurons, and the static network is used to predict the shadow grayscale of each 3D point; the transient network includes a 1-layer MLP neural network with 256 neurons in the layer, which is used to predict the embedding vector of the changing object; the auxiliary network includes a 1-layer MLP neural network with 256 neurons in the layer, which is used to predict the ambient light of each 3D point.

3. The method according to claim 2, characterized in that The color value of each 3D point is calculated by the reflected light, ambient light and shadow grayscale. The specific formula is: s i =a i ·(b i +(1-β i )·m i ) Among them, s i is the color value of the i-th 3D point, β i is the shadow grayscale of the i-th 3D point, μ i is the ambient light of the i-th 3D point, α i is the reflected light of the i-th 3D point.

4. The method according to claim 1, wherein The method for obtaining the trained Sat-Nerf model in step S6 includes: Acquire multiple sets of sample image pairs and sample data in multiple time series and multiple perspectives; the sample data includes multiple sets of same-name point pairs and their corresponding multiple sets of three-dimensional sample coordinates, the true color value of each sample image, and RPC sample parameters; Obtain a camera sample ray corresponding to each pixel in each sample image according to the imaging view angle and multiple RPC sample parameters to obtain multiple camera sample rays; and determine the sample point sequence of each camera sample ray in sequence according to a preset step size to obtain a sample point sequence set; The sample point sequence is input into the initial Sat-Nerf model, and the model parameters are adjusted according to the comprehensive loss function based on the sample data. When the model parameters converge, the trained Sat-Nerf model is generated.

5. The method according to claim 4, characterized in that The comprehensive loss function is: L=L RGB (R)+d SC L SC (R SC )+d DS L DS (R DS ) Among them, L RGB (R) is the color loss function, L SC (R SC ) is the sunlight correction loss function, L DS (R DS ) is the depth estimation supervision loss function; δ SC and δ DS are the weights of the sunlight correction loss function and the depth estimation supervision loss function, respectively.

6. The method according to claim 1, characterized in that Step S6 further comprises: S61 inputs the 3D point sequence set into the trained Sat-Nerf model and predicts the volume density value and color value corresponding to each 3D point sequence; S62 constructs a sigma model and an implicit function of the entire target area, and converts the volume density values ​​based on all 3D points into a three-dimensional point cloud based on the sigma model and the implicit function; Based on the 3D point cloud, S63 uses a surface reconstruction algorithm to build a digital surface model of the target area, maps color values ​​to the digital surface model, and realizes 3D reconstruction of the target area.

7. The method according to claim 1, characterized in that The common viewing conditions are that the satellite zenith angle of each primary image is less than 30°, the angle between any two primary images is less than 5°, and the shadow area of ​​each primary image accounts for less than 5%.

8. The method according to claim 1, characterized in that Step S3 further comprises: S31 uses the SIFT algorithm to extract and match features of each set of advanced common view image pairs in turn, obtaining multiple sets of same-name point pairs and corresponding first pixel coordinates; S32 converts the first pixel coordinate into a geographic coordinate in sequence based on the multiple sets of same-name point pairs and the geospatial data to obtain the longitude and latitude coordinates of the multiple sets of same-name point pairs; S33 interpolates the longitude and latitude coordinates of multiple sets of same-name point pairs according to the geospatial data to obtain multiple sets of three-dimensional coordinates.

9. The method according to claim 1, characterized in that Step S4 further comprises: S41 uses the same-name point pairs corresponding to the three-dimensional coordinates as control points to construct an RPC model, and performs positive calculations on the original RPC parameters of each advanced common view image and the three-dimensional coordinates to obtain the second pixel coordinates on the advanced common view image corresponding to the three-dimensional coordinates; S42 calculates the error distance between the first pixel coordinate and the second pixel coordinate, adopts a regularization method combining L1 and L2 based on the least squares adjustment principle, introduces a set linear constraint condition, and obtains an optimized RPC model; S43 repairs the original RPC parameters of each advanced common view image using the optimized RPC model according to the error distance to obtain a plurality of repaired RPC parameters; S42 uses the optimized RPC model to repair the original RPC parameters of each advanced common view image to obtain multiple repaired RPC parameters.

10. The method according to claim 1, characterized in that The super-resolution model described in step S2 includes a residual hybrid attention network and an edge enhancement network; each primary common view image is super-reconstructed by the residual hybrid attention network to obtain multiple intermediate common view images; the edge of each intermediate common view image is optimized by the edge enhancement network to obtain multiple high-level common view images.

Images