Shadow perception satellite image nerve radiation field surface three-dimensional reconstruction method

Abstract

The invention provides a shadow perception satellite image nerve radiation field surface three-dimensional reconstruction method, which comprises the steps of optimizing an RPC model based on a multi-temporal satellite image, generating a sparse point cloud, and constructing a unified coordinate system through outer bounding sphere normalization; three-dimensional points are sampled in the pixel line-of-sight direction, and a training set is constructed; the SDF network predicts that spatial points have symbolic distance values, the color network integrates SDF features and sight line direction output colors, and the shadow color network combines the sun direction to generate shadow vectors; the light color is predicted through volume rendering integration, the shadow probability is calculated based on secondary ray casting in the sun direction, and a loss function is constructed in combination with an irradiance error to jointly optimize network parameters; and generating a three-dimensional grid by using sampling points of the trained SDF network in a unified coordinate system and adopting a Marching Cubes algorithm. According to the invention, the convergence stability of the neural radiation field model on satellite data and the success rate and precision of scene three-dimensional surface reconstruction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a shadow-aware satellite image neural radiation field surface three-dimensional reconstruction method. Background Art

[0002] Satellite imagery, with its global coverage, long-term monitoring, and large imaging range, is an important means of obtaining information about the Earth's surface. Utilizing multi-temporal data from historical image libraries to form observations from different perspectives enables comprehensive reconstruction of three-dimensional surface information, providing a new solution for large-scale real-world three-dimensional construction. In recent years, the development of neural radiance field technology has brought new possibilities for three-dimensional reconstruction of satellite imagery. This method captures and learns implicit representations of scenes from multi-temporal satellite imagery, avoiding the complex processes and intermediate representations of traditional methods, demonstrating powerful fitting capabilities for complex scenes, and capable of outputting three-dimensional geometric information of scenes. The original neural radiance field method was based on static ground data and did not consider the complexity and uncertainty of multi-temporal satellite data.

[0003] When using multi-temporal satellite images for three-dimensional reconstruction, factors such as radiation inconsistency, seasonal changes, and dynamic changes lead to uncertainty in pixel color information. The neural radiation field method is sensitive to radiation information. Inconsistent color information will cause the network to fit the wrong geometric structure, causing the optimization process to fall into a local optimal solution, which limits the application of neural radiation field technology on satellite data.

[0004] Shadow information is ubiquitous in satellite imagery. Since there is only one independent light source, the sun, shadows are not complex. The nearly parallel nature of sunlight makes it easier to estimate its direction. Furthermore, the solar altitude and azimuth angles in satellite metadata can accurately calculate the sun's direction at the time the sensor collects data. Therefore, shadow information can serve as an important clue for estimating the geometry of land features.

[0005] Existing methods such as S-NeRF and Sat-NeRF attempt to utilize shadow information to optimize neural radiance fields, but numerous issues remain. First, insufficient shadow generalization: While S-NeRF uses a neural network to learn and fit shadow effects in satellite data, it does not explicitly establish a functional relationship between shadows and scene geometry. This results in poor generalization of shadows to new perspectives, making it difficult to accurately predict shadow distributions under different perspectives. Second, transient object interference: Sat-NeRF improves upon S-NeRF by estimating neural network uncertainty to predict transient objects (such as moving vehicles or vegetation changes). However, shadow information may be misclassified as transient, leading to geometric errors and poor reconstruction accuracy. Finally, the volume density representation lacks constraints: EO-NeRF infers scene shadows based on the spatial distribution of volume density. While this improves shadow generalization to some extent, its volume density representation lacks effective spatial constraints, making it difficult to accurately obtain isosurfaces, resulting in inaccurate output mesh models.

[0006] Neural radiometric technology is an effective method for reconstructing three-dimensional geometry. It learns and predicts the three-dimensional geometry of a scene based on the consistent radiometric field between multi-view images. This technology works well for static ground scenes, but when the input images are multi-temporal satellite images acquired at different times, factors such as inconsistency in radiometric information between images, seasonal changes, and dynamic changes can cause the network to fit incorrect geometric structures, and even cause the entire optimization process to fall into a local optimal solution, making it impossible to achieve stable and effective convergence. This limits the three-dimensional reconstruction effect of neural radiometric field technology on satellite data. Summary of the Invention

[0007] In view of this, the present invention proposes a shadow-aware satellite image neural radiation field reconstruction method, which optimizes the scene SDF based on additional satellite image shadow information, improves the convergence stability of the neural radiation field model on satellite data, and improves the success rate and accuracy of scene three-dimensional surface reconstruction.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A shadow-aware satellite image neural radiometric site surface 3D reconstruction method comprises the following steps:

[0010] Step S1: Based on the multi-temporal satellite image dataset of the area of ​​interest, the sun direction vector of each image is calculated using the sun azimuth and altitude angles. Image cropping, 16-bit to 8-bit downscaling, and panchromatic and multispectral image fusion are performed. The RPC model is optimized using bundle adjustment to improve RPC accuracy, and a sparse point cloud is generated.

[0011] Step S2: Computing a bounding sphere outside the sparse point cloud scene and normalizing the point cloud to a unit sphere space, constructing a unified scale coordinate system so that the coordinate range of the scene to be reconstructed is between -1 and 1;

[0012] Step S3: Based on the unified scale coordinate system constructed in step S2, randomly select image pixels and generate light along the line of sight, sampling three-dimensional points on the light , associated 3D points The training sample set is constructed by using the coordinates, sight direction and pixel color of the pixel;

[0013] Step S4: Construct a neural radiation field network model and a shadow color network model, wherein the neural radiation field network model includes an SDF network and a color network, wherein the SDF network predicts the signed distance SDF value of a spatial point , the color network fuses the SDF features and the sight direction to output the color value of the spatial point , the shadow color network generates the color vector of the shadow based on the sun direction obtained in step S1 ;

[0014] Step S5: using the signed distance SDF value of the spatial point output by the SDF network, the color value of the pixel light is predicted based on the volume rendering integral formula;

[0015] Step S6: Based on the sun direction vector obtained in step S1, a secondary reflected ray is emitted along the sun direction, and the shadow probability of the surface point is calculated according to the transmittance.

[0016] Step S7: Calculate the irradiance of the pixel light by combining the shadow probability and irradiance model of step S6, and construct a minimization loss function based on the error between the irradiance and the color value of the pixel light to reversely optimize the parameters of the SDF network, color network and shadow color network, where the signed distance SDF value of the spatial point obtained in step S4 is , the color value of the spatial point and shadow color vector As input to the irradiance model;

[0017] Step S8: After training is completed, based on the unified scale coordinate system of step S2, spatial points are uniformly sampled in the coordinate range of -1 to 1 for each coordinate axis (x, y, z), and the SDF value of each point is predicted by the trained SDF network. The Marching Cubes algorithm is used to generate a three-dimensional mesh model.

[0018] Furthermore, step S1 specifically includes:

[0019] Step S1.1: Read the sun azimuth from the satellite metadata file for each image and the sun's altitude angle Information, calculate the sun direction vector :

[0020] ;

[0021] Step S1.2: Determine the UTM coordinate range of the area of ​​interest and convert it to WGS-84 geographic coordinates; use the RPC parameters of the satellite image to project the area of ​​interest onto the panchromatic and multispectral images and crop them according to the minimum bounding rectangle; after cropping, subtract the starting row / column number of the target area in the image from the original offset to update the row offset and column offset of the RPC parameters;

[0022] Step S1.3: Use the maximum and minimum value stretching method to convert the 16-bit data into 8-bit data. First, traverse the entire 16-bit image data and determine its minimum grayscale value. and the maximum grayscale value , then using the formula

[0023] ;

[0024] Convert the 16-bit grayscale value of each pixel to Mapping to 8-bit grayscale values ;

[0025] Step S1.4: Extract blue, green, and red bands from the multispectral image and use the Brovey pan-sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image to generate a color image with both high spatial resolution and rich spectral information.

[0026] Step S1.5: Use the feature extraction, feature matching, and bundle adjustment methods in the COLMAP software to correct the RPC parameters and generate a sparse point cloud.

[0027] Furthermore, step S2 specifically includes: normalizing the scene, determining the geometric boundary of the point cloud based on the outer bounding sphere of the sparse point cloud computing scene generated by RPC optimization, and translating the center of gravity of the sparse point cloud to the origin of the coordinate system to achieve center alignment; then, scaling the point cloud according to the radius of the outer bounding sphere so that all sparse points fall within a standardized sphere with a radius of 1 centered at the origin, ensuring that the coordinate value range is controlled between [-1, 1].

[0028] Furthermore, step S5 specifically includes:

[0029] All sampling point positions on a pixel ray Gaze direction and gaze direction Input the SDF network and output the SDF and color values ​​of all points. The predicted color of the pixel light is calculated by the volume rendering formula:

[0030] ;

[0031] is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, represents opacity, that is, the probability of each point blocking the propagation of light; is the color weight, which indicates the contribution of each sampling point to the final predicted color. and By sampling point Volume density at calculate:

[0032] ;

[0033] in, Represents the distance between two adjacent sampling points on the ray: ; Introduce the unimodal S-density function To model the relationship between surface distance SDF and color weight:

[0034] ;

[0035] in Represents the SDF value of a spatial point, is the Sigmoid function The derivative of

[0036] ; It is a learnable parameter that indicates the concentration of color weight near the surface of the object. The unimodal Sigmoid function constrains the color weight to increase as it approaches the surface and to reach its maximum value on the surface. After derivation, the opacity density is obtained. Relationship with SDF:

[0037] ;

[0038] Cumulative opacity and predicted color Further changed to:

[0039] ;

[0040] .

[0041] Furthermore, step S6 specifically includes:

[0042] The scene is represented as an implicit SDF field, and the scene surface is determined by the zero level set:

[0043] ;

[0044] The volume rendering framework learns the SDF and color of the scene by emitting countless pixel rays. Find surface points where the SDF is zero : Sampling on pixel rays 3D coordinate points , the SDF value of each point is expressed as ;set up for The function of , a ray passing through the surface of an object will have two adjacent sampling points, one inside the object with an SDF value less than 0, and the other outside the object with an SDF value greater than 0. The surface points are located at positions where the SDF is alternating between positive and negative:

[0045] ;

[0046] is a set of sampling points that meet the conditions; the surface point position is estimated by linear interpolation based on the sampling points:

[0047] ;

[0048] In determining pixel light Intersection with the 3D scene surface Afterwards, For direction, The secondary ray is emitted towards the sun from the starting point ; Sample far enough away on this ray points

[0049] , the SDF value of each point is , then the opacity density of each point is for:

[0050] ;

[0051] Transparent density Integrate and calculate the light Cumulative transparency of:

[0052] ;

[0053] Sunlight Previous point The probability of being occluded is determined by the cumulative transparency Indicates that in the volume rendering framework, when light is emitted from the surface of an object to the sun, the probability that the point is visible to the light source is the transmittance, which is the cumulative transparency of the last sampling point emitted from the surface of the object to the light source. Indicates transmittance :

[0054] ;

[0055] Since then, the transmittance estimation of satellite images has been realized based on the geometric relationship between implicit SDF and the direction of the sun. Indicates the emission of light The probability that a pixel is in the shadow is also the probability that the light The probability that the intersection position with the scene is in shadow.

[0056] Furthermore, the irradiance model in step S7 is expressed as:

[0057] ;

[0058] in, is the irradiance received by the satellite image, is the color of the scene itself, is the solar transmittance, is the shadow color;

[0059] Corresponding to the scene color in the irradiance model , then the irradiance model based on volume rendering is expressed as:

[0060] ;

[0061] For light The irradiance, The predicted color for volume rendering, For light The solar transmittance is determined by the scene SDF and the sun direction, and the shadow color Related to the sun direction, by the shadow color network Estimation; Through the irradiance model, the color of the pixel is represented by direct and indirect lighting, and the shadow information is also decoupled and included in the color prediction. The loss function is updated as follows:

[0062] ;

[0063] in For light The irradiance, For light The corresponding pixel color of the input satellite image.

[0064] The present invention has the following beneficial effects:

[0065] (1) This paper uses a volume rendering framework based on a signed distance function (SDF) and a geometry-aware shadow detection method to accurately predict scene shadow information and optimize scene geometry to produce shadow effects consistent with the input image. The introduction of sunlight enables the network to infer the geometry of the ground objects that produce shadows based on image shadows, thereby demonstrating greater robustness in the reconstruction of vegetation-covered areas and highly reflective surfaces (such as glass curtain walls).

[0066] (2) Neural radiance field technology learns scene geometry through color. In the early stages of iteration, the network prioritizes black and white information with the strongest contrast. Shadows, as black areas, can provide the network with rough geometric information in the early stages of iteration, allowing the network to obtain relatively accurate initial geometric values. This mechanism effectively prevents the network from falling into local optimal solutions, improves convergence stability, and enhances the robustness of neural radiance field technology on satellite data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of a method for three-dimensional reconstruction of a satellite image neural radiometric site surface using shadow perception according to an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of three-dimensional scene normalization according to an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the architecture of a shadow-aware satellite image neural radiometric site surface 3D reconstruction method according to an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of the network structure of an embodiment of the present invention;

[0071] Figure 5 Schematic diagram of shadow detection based on secondary reflection according to an embodiment of the present invention;

[0072] Figure 6 This is an example of the shadow prediction result of an embodiment of the present invention;

[0073] Figure 7 Results of different methods on the DFC2019 dataset;

[0074] Figure 8 Schematic diagram of the effect of shadow information on 3D reconstruction results in an ablation experiment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0076] See also Figure 1 The embodiment of the present invention provides a shadow-aware satellite image neural radiation site surface three-dimensional reconstruction method, comprising the following steps:

[0077] Step S1.1: Read the sun azimuth from the satellite metadata file for each image and the sun's altitude angle Information, calculate the sun direction vector :

[0078]

[0079] Step S1.2: Satellite imagery typically covers a vast area, with a single image potentially containing hundreds of kilometers of ground information. However, in practice, the focus is often limited to a specific, smaller area of ​​interest (AOI). Therefore, the satellite imagery needs to be cropped to extract valid data within the AOI. To effectively crop large imagery, the UTM coordinate range of the AOI is first determined and converted to WGS-84 geographic coordinates. Next, using the satellite imagery's RPC parameters, the AOI is projected onto panchromatic and multispectral imagery and cropped based on a minimum bounding rectangle. After cropping, the row and column offsets of the RPC parameters must be updated. This involves subtracting the starting row / column number of the target area in the image from the original offset to ensure that the cropped image remains accurately geolocated under the new RPC model.

[0080] Step S1.3: Use the maximum and minimum value stretching method to convert the 16-bit data into 8-bit data. Specifically, first traverse the entire 16-bit image data and determine its minimum grayscale value. and the maximum grayscale value , then, using the formula

[0081]

[0082] Convert the 16-bit grayscale value of each pixel to Mapping to 8-bit grayscale values ;

[0083] Step S1.4: In order to obtain a high-resolution color image, the present invention extracts blue, green, and red bands from the multispectral image, and uses the Brovey pan-sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image to generate a high-resolution color image.

[0084] Step S1.5: Use the feature extraction, feature matching, and bundle adjustment methods in the COLMAP software to correct the RPC parameters and generate a sparse point cloud.

[0085] Step S2: Figure 2 As shown in the figure, the scene is normalized and the bounding sphere of the scene is calculated based on the sparse point cloud generated by RPC optimization to determine the geometric boundaries of the point cloud. The center of gravity of the point cloud is translated to the origin of the coordinate system to achieve center alignment. Then, the point cloud is scaled according to the radius of the bounding sphere so that all sparse points fall within a normalized sphere with a radius of 1 centered at the origin, ensuring that the coordinate values ​​are within the range [-1, 1].

[0086] Step S3: Randomly select a satellite image and randomly sample on the image pixels, for each sampled pixel, a line of sight is formed from the center of the satellite sensor to the pixel. Pixel rays (called pixel rays ), each ray is emitted from the center of the satellite and intersects the normalized sphere at and , between these two points, points can be sampled by distance:

[0087]

[0088] Represents the direction of light, is the number of sampling points, Is the distance constant. Record the position of each sampling point , sight direction , the image to which it belongs and pixel color , forming a data sample, and establishing multiple Pixel rays are sampled to obtain a sample set. Take 512, Take 64.

[0089] Step S4: Figure 3 As shown, the method of the present invention uses three-dimensional coordinates , sight direction and the direction of the sun As input, and use multi-layer perceptron (MLP) to predict the output one-dimensional SDF, three-dimensional object color and shadow color The network architecture includes SDF network , Color Network Each network consists of an input layer, multiple hidden layers and an output layer, using Softplus as the activation function. Specifically, Figure 4 As shown, SDF network It consists of eight hidden layers. The input layer is a three-dimensional position vector , while the output layer generates a 256-dimensional feature vector and a one-dimensional SDF. In addition, a link is established between the input layer and the output of the fourth layer to facilitate the flow of information. It consists of four hidden layers, the output layer is the color of the three-dimensional sampling point, and the input layer includes three vectors: the three-dimensional sampling point observation direction, the three-dimensional sampling point position and a 256-dimensional feature vector. Predicted shadow color , which consists of a 256-dimensional feature vector, and the input is the three-dimensional direction of the sun In addition, in order to improve the representation ability of low-dimensional vectors, the present invention uses vector and They are position-encoded before entering the network. Specifically, six different frequency components are used to encode the position Encode and expand its dimension from 3 to 39. Similarly, the direction of the light at the sampling point Four different frequency components are encoded, and their dimension is expanded from 3 to 27.

[0090] Step S5: Predict the color value of pixel light based on the volume rendering method of SDF. Gaze direction and gaze direction Input the SDF network and output the SDF and color values ​​of all points. Then the predicted color of the pixel light can be calculated by the volume rendering formula:

[0091]

[0092] is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, Represents opacity, that is, the probability of each point blocking the propagation of light. is the color weight, which represents the contribution of each sampling point to the final predicted color. and By sampling point Volume density at calculate:

[0093]

[0094] in, Represents the distance between two adjacent sampling points on the ray: Color Weight Indirectly by volume density The distribution of volume density in space is more flexible and has no additional constraints. However, ideally, the color weight at the surface of the object should be the largest, and the farther away from the object, the smaller the color weight should be. That is, the maximum value of the color weight should be distributed unbiased on the surface of the object. In order to achieve such a constraint, the present invention introduces a single-peak S-density function To model the relationship between surface distance SDF and color weight:

[0095]

[0096] in Represents the SDF value of a spatial point, is the Sigmoid function The derivative of . is a learnable parameter that represents the concentration of color weight near the surface of the object. The unimodal Sigmoid function constrains the color weight to increase as it approaches the surface and to reach its maximum value on the surface. After derivation, the opacity density can be obtained Relationship with SDF:

[0097]

[0098] Cumulative opacity and predicted color Can be further changed to:

[0099]

[0100]

[0101] Step S6: Calculate the probability of a surface point being in the shadow based on the secondary reflection of the light. Different from the shadow detection method based on image recognition, the present invention determines the shadow based on the relationship between the implicit geometric SDF of the scene and the visible relationship of the sun. Figure 5 As shown, based on the sensor's position, the image pixel sends a ray to query the surface intersection in the scene Position, then the intersection Toward the sun Shoot another ray If the ray is not blocked by the scene geometry, it means Point is visible to the Sun and is not in shadow (see Figure 5 in ). On the contrary, if is occluded by scene geometry, it means Points are shaded (see Figure 5 in ). Thus, shadow extraction can be achieved based on scene geometry. Furthermore, in volume rendering theory, light may be blocked by the spatial particles it passes through. Therefore, visibility is not determined as a binary 0 or 1, but rather as a visibility probability. Therefore, this paper discusses how to implement shadow detection within the framework of SDF-based volume rendering.

[0102] The method of the present invention uses an implicit SDF to represent the three-dimensional space and estimates the probability that the scene surface is visible to the sun by emitting secondary rays, that is, the solar transmittance. To calculate the solar transmittance of an image ,First, a ray is emitted from the pixel to the estimated surface point in the scene where the ray intersects, and then a secondary ray is emitted from the surface point toward the sun, calculating the probability of being occluded by its own geometry.

[0103] Specifically, the scene is represented as an implicit SDF field, and the scene surface can be determined by the zero-level set:

[0104]

[0105] The volume rendering framework learns the SDF and color of the scene by emitting countless pixel rays. Similarly, it can be used to Find surface points where the SDF is zero : Sampling on pixel rays 3D coordinate points , the SDF value of each point is expressed as For simplicity, let for The function of , a ray passing through the surface of an object will always have two adjacent sampling points, one inside the object with an SDF value less than 0, and the other outside the object with an SDF value greater than 0. The surface point is exactly at the position where the SDF alternates between positive and negative:

[0106]

[0107] Is a set of sampling points that meet the conditions. Based on these discrete sampling points, the position of the surface point can be estimated by linear interpolation:

[0108]

[0109] For a ray, the scene point it intersects with There may be multiple, the present invention selects The smallest one corresponds to the outermost surface point that is least likely to be occluded. Intersection with the 3D scene surface Afterwards, For direction, The secondary ray is emitted towards the sun from the starting point Similarly, sampling on this ray is far enough away points , the SDF value of each point is , then the opacity density of each point is for:

[0110]

[0111] Finally, for transparent density Integrate and calculate the light Cumulative transparency of:

[0112]

[0113] Sunlight Previous point The probability of being occluded is determined by the cumulative transparency In the volume rendering framework, when emitting light from the surface of an object to the sun, the probability that the point is visible to the light source is the transmittance, so the cumulative transparency of the last sampling point emitted from the surface of the object to the light source is Indicates transmittance .

[0114]

[0115] Since then, the transmittance estimation of satellite images has been realized based on the geometric relationship between implicit SDF and the direction of the sun. Indicates the emission of light The probability that a pixel is in the shadow is also the probability that the light The probability that the intersection point with the scene is in the shadow is predicted as follows Figure 6 shown.

[0116] Step S7: Calculate the color loss of the pixel based on the irradiance model and reversely optimize the network parameters. Traditional volume rendering models do not take shadow effects into account and cannot handle inconsistent shadows in multi-temporal satellite images. The present invention introduces an irradiance model to decouple direct illumination and indirect shadows: first, the color of the pixel light is regarded as the color of the object itself, representing the appearance of the object under shadow-free conditions; then, the irradiance of the pixel light is jointly calculated in combination with the shadow probability to simulate the combined effect of illumination and shadow. Next, the calculated irradiance is compared with the actual color value of the pixel light, the difference between the two is calculated, and the difference is used as a loss function for training the network. By minimizing the loss function, the parameters of the SDF network, color network, and shadow color network are optimized, so that the model can more accurately predict the geometric shape, appearance attributes, and lighting and shadow information of the scene.

[0117] In classic volume rendering methods, radiation information is completely determined by the color of scene particles, and shadow phenomena are also coupled to color. This assumption is satisfied in static scenes with a constant light source. However, for multi-temporal satellite data, each image has different sun illumination directions and shadows, resulting in the basic radiation model no longer being suitable for practical problems. Therefore, based on volume rendering theory, this paper uses an irradiance model to decouple color, direct light, and indirect light:

[0118]

[0119] is the irradiance received by the satellite image, which is composed of direct sunlight and indirect lighting Direct illumination refers to the color of sunlight directly hitting the ground and reflected by the ground to the pixels of satellite images. Indirect illumination involves a more complex physical process, which describes the color of sunlight that reaches the blocked area after being blocked by objects on the ground through multiple reflections. This part of the color is mainly obtained by secondary reflection from the atmosphere, so it is called indirect illumination. In order to model these processes, the solar transmittance is introduced. , which represents the probability that an object is directly visible from the sun, so direct illumination The color of the scene itself Solar transmittance Multiplication calculation, while indirect lighting The color of the scene itself , transmittance and shadow color Therefore, the irradiance model is expressed as:

[0120]

[0121] Classical volume rendering assumes that the color of the scene is constant, which corresponds to the color of the scene itself in the irradiance model. , then the irradiance model based on volume rendering can be expressed as:

[0122]

[0123] For light The irradiance, The predicted color for volume rendering, For light The solar transmittance is determined by the scene SDF and the sun direction, and the shadow color Related to the sun direction, by the shadow color network Estimation. Through the irradiance model, the color of the pixel is represented by direct and indirect lighting, and the shadow information is also decoupled and included in the color prediction, so the loss function can be updated as:

[0124]

[0125] in For light The irradiance, For light The corresponding pixel color of the input satellite image.

[0126] Step S8: After iterating steps S3 to S7 300,000 times, network training is completed. Uniform sampling prediction is performed within the 3D scene space to obtain the SDF values ​​of all sampling points. These values ​​are then input into the Marching Cubes method to generate a 3D mesh model. Specifically, the Marching Cubes algorithm is used to process the SDF values ​​of all sampling points, extracting isosurfaces with SDF values ​​of 0 to generate a continuous and smooth 3D mesh surface, and outputting the 3D mesh model. The number of samples within the 3D scene space can be 128*128*128.

[0127] Example:

[0128] This paper uses the DFC2019 dataset to evaluate different methods, including four scenarios: JAX_004, JAX_068, and JAX_260. Figure 7The mesh reconstruction results of our method and comparison methods (S2P, S-NeRF, Sat-NeRF, and VisSat) are presented. Overall, the results of S2P, S-NeRF, and Sat-NeRF exhibit significant noise, especially in the areas of transient objects (such as vehicles). This is likely due to a lack of geometric regularization. Our method, by regularizing the SDF field, suppresses the drastic surface geometry changes caused by noise, resulting in smoother and more detailed reconstructions. For building outlines, volume rendering-based techniques (S-NeRF, Sat-NeRF, and our method) are able to learn sharp edges from color variations, outperforming the blurry results of VisSat and S2P. For vegetation reconstruction, our method utilizes shadow information to infer geometry, avoiding the influence of color variations and achieving more complete vegetation reconstructions, while other methods (such as VisSat and S2P) exhibit blurring or loss of vegetation.

[0129] Figure 8 Ablation experiment results are presented, comparing the reconstruction performance of a baseline method (which does not use shadow information) with that of the proposed method (which incorporates a shadow-aware module). The baseline method suffers from unstable reconstruction results, resulting in geometric errors and holes, due to seasonal variations in vegetation and the highly reflective nature of glass curtain walls in multi-temporal satellite imagery. For example, in scenes JAX_004, JAX_068, and JAX_260, pixel color variations in vegetation areas cause the network to fall into a local optimum, affecting global geometric accuracy. In scene JAX_167, the strong reflective nature of the glass curtain wall causes reconstruction failures based on color consistency. In contrast, the proposed method, through the shadow-aware module, leverages shadow information to infer geometric details, significantly improving the stability and integrity of the reconstruction. For example, in scene JAX_214, the proposed method recovers roof details using shadow information, while in scene JAX_167, the shadow-based geometric-aware method successfully reconstructs the complete structure of the glass curtain wall. Experiments demonstrate that the inclusion of shadow information effectively resolves convergence instability caused by vegetation and reflective surfaces in multi-temporal imagery, improving the geometric accuracy and integrity of the reconstruction results.

[0130] This paper introduces an irradiance model to decouple shadow information. By simulating the imaging process, the predicted color is decomposed into the object's own color, direct illumination, and indirect illumination. A geometrically aware shadow detection method is proposed. Based on the spatial representation of the signed distance function (SDF), the secondary projection process of sunlight is simulated to achieve accurate prediction of scene shadows. Finally, the sun's direction information is used to infer the geometric shape of the shadow, providing geometric guidance for the learning of neural implicit surfaces and improving the convergence stability of the neural radiation field model in complex satellite data.

[0131] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A shadow-aware satellite image neural radiometric site surface 3D reconstruction method, characterized in that: The steps include: Step S1: Based on the multi-temporal satellite image dataset of the area of ​​interest, the sun direction vector of each image is calculated using the sun azimuth and altitude angles. Image cropping, 16-bit to 8-bit downscaling, and panchromatic and multispectral image fusion are performed. The RPC model is optimized using bundle adjustment to improve RPC accuracy, and a sparse point cloud is generated. Step S2: Computing a bounding sphere outside the sparse point cloud scene and normalizing the point cloud to a unit sphere space, constructing a unified scale coordinate system so that the coordinate range of the scene to be reconstructed is between -1 and 1; Step S3: Based on the unified scale coordinate system constructed in step S2, randomly select image pixels and generate light along the line of sight, sampling three-dimensional points on the light , associated 3D points The training sample set is constructed by using the coordinates, sight direction and pixel color of the pixel; Step S4: Construct a neural radiation field network model and a shadow color network model, wherein the neural radiation field network model includes an SDF network and a color network, wherein the SDF network predicts the signed distance SDF value of a spatial point , the color network fuses the SDF features and the sight direction to output the color value of the spatial point , the shadow color network generates the color vector of the shadow based on the sun direction obtained in step S1 ; Step S5: using the signed distance SDF value of the spatial point output by the SDF network, the color value of the pixel light is predicted based on the volume rendering integral formula; Step S6: Based on the sun direction vector obtained in step S1, a secondary reflected ray is emitted along the sun direction, and the shadow probability of the surface point is calculated according to the transmittance. Step S7: Calculate the irradiance of the pixel light by combining the shadow probability and irradiance model of step S6, and construct a minimization loss function based on the error between the irradiance and the color value of the pixel light to reversely optimize the parameters of the SDF network, color network and shadow color network, where the signed distance SDF value of the spatial point obtained in step S4 is , the color value of the spatial point and shadow color vector As input to the irradiance model; Step S8: After training is completed, based on the unified scale coordinate system of step S2, spatial points are uniformly sampled in the coordinate range of -1 to 1 for each coordinate axis (x, y, z), and the SDF value of each point is predicted by the trained SDF network. The Marching Cubes algorithm is used to generate a three-dimensional mesh model.

2. The shadow-aware satellite image neural radiometric site surface 3D reconstruction method according to claim 1, characterized in that: Step S1 specifically includes: Step S1.1: Read the sun azimuth from the satellite metadata file for each image and the sun's altitude angle Information, calculate the sun direction vector : ； Step S1.2: Determine the UTM coordinate range of the area of ​​interest and convert it to WGS-84 geographic coordinates; use the RPC parameters of the satellite image to project the area of ​​interest onto the panchromatic and multispectral images and crop them according to the minimum bounding rectangle; after cropping, subtract the starting row / column number of the target area in the image from the original offset to update the row offset and column offset of the RPC parameters; Step S1.3: Use the maximum and minimum value stretching method to convert the 16-bit data into 8-bit data. First, traverse the entire 16-bit image data and determine its minimum grayscale value. and the maximum grayscale value , then using the formula ； Convert the 16-bit grayscale value of each pixel to Mapping to 8-bit grayscale values ; Step S1.4: Extract blue, green, and red bands from the multispectral image and use the Brovey pan-sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image to generate a color image with both high spatial resolution and rich spectral information. Step S1.5: Use the feature extraction, feature matching, and bundle adjustment methods in the COLMAP software to correct the RPC parameters and generate a sparse point cloud.

3. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S2 specifically includes: normalizing the scene, determining the geometric boundary of the point cloud based on the bounding sphere of the sparse point cloud computing scene generated by RPC optimization, and translating the center of gravity of the sparse point cloud to the origin of the coordinate system to achieve center alignment; then, scaling the point cloud according to the radius of the bounding sphere so that all sparse points fall within a standardized sphere with a radius of 1 centered at the origin, ensuring that the coordinate value range is controlled between [-1, 1].

4. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S5 specifically includes: All sampling point positions on a pixel ray Gaze direction and gaze direction Input the SDF network and output the SDF and color values ​​of all points. The predicted color of the pixel light is calculated by the volume rendering formula: ； is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, represents opacity, that is, the probability of each point blocking the propagation of light; is the color weight, which indicates the contribution of each sampling point to the final predicted color. and By sampling point Volume density at calculate: ； in, Represents the distance between two adjacent sampling points on the ray: ; Introduce the unimodal S-density function To model the relationship between surface distance SDF and color weight: ； in Represents the SDF value of a spatial point, is the Sigmoid function The derivative of ; It is a learnable parameter that indicates the concentration of color weight near the surface of the object. The unimodal Sigmoid function constrains the color weight to increase as it approaches the surface and to reach its maximum value on the surface. After derivation, the opacity density is obtained. Relationship with SDF: ； Cumulative opacity and predicted color Further changed to: ； 。 5. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S6 specifically includes: The scene is represented as an implicit SDF field, and the scene surface is determined by the zero level set: ； The volume rendering framework learns the SDF and color of the scene by emitting countless pixel rays. Find surface points where the SDF is zero : Sampling on pixel rays 3D coordinate points , the SDF value of each point is expressed as ;set up for The function of , a ray passing through the surface of an object will have two adjacent sampling points, one inside the object with an SDF value less than 0, and the other outside the object with an SDF value greater than 0. The surface points are located at positions where the SDF is alternating between positive and negative: ； is a set of sampling points that meet the conditions; the surface point position is estimated by linear interpolation based on the sampling points: ； In determining pixel light Intersection with the 3D scene surface Afterwards, For direction, The secondary ray is emitted towards the sun from the starting point ; Sample far enough away on this ray points , the SDF value of each point is , then the opacity density of each point is for: ； Transparent density Integrate and calculate the light Cumulative transparency of: ； Sunlight Previous point The probability of being occluded is determined by the cumulative transparency Indicates that in the volume rendering framework, when light is emitted from the surface of an object to the sun, the probability that the point is visible to the light source is the transmittance, which is the cumulative transparency of the last sampling point emitted from the surface of the object to the light source. Indicates transmittance : ； Since then, the transmittance estimation of satellite images has been realized based on the geometric relationship between implicit SDF and the direction of the sun. Indicates the emission of light The probability that a pixel is in the shadow is also the probability that the light The probability that the intersection position with the scene is in shadow.

6. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 5, characterized in that: The irradiance model in step S7 is expressed as: ； in, is the irradiance received by the satellite image, is the color of the scene itself, is the solar transmittance, is the shadow color; Corresponding to the scene color in the irradiance model , then the irradiance model based on volume rendering is expressed as: ； For light The irradiance, The predicted color for volume rendering, For light The solar transmittance is determined by the scene SDF and the sun direction, and the shadow color Related to the sun direction, by the shadow color network Estimation; Through the irradiance model, the color of the pixel is represented by direct and indirect lighting, and the shadow information is also decoupled and included in the color prediction. The loss function is updated as follows: ； in For light The irradiance, For light The corresponding pixel color of the input satellite image.

Images