Shadow-aware satellite image neural radiance field terrestrial surface 3d reconstruction method

Abstract

The application provides a shadow-aware satellite image neural radiance field ground surface three-dimensional reconstruction method, comprising: optimizing an RPC model based on multi-temporal satellite images and generating a sparse point cloud, and constructing a unified coordinate system through an outer enclosing sphere normalization; sampling three-dimensional points along a pixel line-of-sight direction and constructing a training set; an SDF network predicts a signed distance value of a spatial point, a color network fuses SDF features and a line-of-sight direction to output color, and a shadow color network generates a shadow vector in combination with a sun direction; a ray color is predicted through volume rendering integration, a shadow probability is calculated based on twice ray casting of the sun direction, a loss function is constructed in combination with an irradiance error to jointly optimize network parameters; and a three-dimensional grid is generated by using a Marching Cubes algorithm after sampling points in the unified coordinate system by using the trained SDF network. The application can improve the convergence stability of a neural radiance field model on satellite data and the success rate and precision of scene three-dimensional surface reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a shadow-aware satellite image neural radiance field ground surface three-dimensional reconstruction method. BACKGROUND

[0002] Satellite images have the characteristics of global coverage, long-term monitoring and large imaging range, and are an important means of obtaining information about the earth's surface. Using multi-temporal data in the historical image library to form different viewing angles can achieve comprehensive reconstruction of three-dimensional information of the ground surface, and provides a new solution for large-scale real scene three-dimensional construction. In recent years, the development of neural radiance field technology has brought new possibilities for satellite image three-dimensional reconstruction. This method captures and learns the implicit representation of the scene from multi-temporal satellite images, avoiding the complex process and intermediate representation in traditional methods, and showing strong fitting ability for complex scenes. It can output the three-dimensional geometric information of the scene. The original neural radiance field method is based on static ground data, and does not consider the complexity and uncertainty of multi-temporal satellite data.

[0003] When using multi-temporal satellite images for three-dimensional reconstruction, factors such as inconsistency of radiometric information, seasonal changes and dynamic changes cause uncertainty in pixel color information. The neural radiance field method is sensitive to radiometric information, and inconsistent color information can cause the network to fit incorrect geometric structures, causing the optimization process to fall into a local optimal solution, which limits the application of neural radiance field technology to satellite data.

[0004] Shadow information is ubiquitous in satellite images. Since there is only one independent light source, the sun, the shadow is not complex. The characteristic that the sun's rays are nearly parallel makes it easier to estimate the direction of the sun's rays. At the same time, the sun elevation and azimuth information in the satellite metadata can accurately calculate the direction of the sun at the time when the sensor collects data. Therefore, shadow information can be used as an important clue to estimate the geometry of ground objects.

[0005] In the prior art, methods such as S-NeRF and Sat-NeRF attempt to optimize the neural radiance field by utilizing shadow information, but still have many problems. First, the shadow generalization ability is insufficient: S-NeRF learns and fits the shadow effect in satellite data through a neural network, but does not explicitly establish a functional relationship between the shadow and the scene geometry, resulting in poor generalization of the shadow at new viewing angles and difficulty in accurately predicting the shadow distribution at different viewing angles. Second, transient object interference: Sat-NeRF improves on S-NeRF by estimating the uncertainty of the neural network to predict transient objects (such as moving vehicles or vegetation changes), but shadow information may be misjudged as a transient object, resulting in incorrect geometry estimation and affecting the reconstruction accuracy. Finally, the representation of volume density lacks constraints: EO-NeRF derives the scene shadow based on the spatial distribution of volume density, which improves the generalization effect of the shadow to some extent, but the representation of volume density lacks effective spatial constraints, making it difficult to achieve accurate isosurface acquisition and resulting in insufficient accuracy of the output mesh model.

[0006] Neural radiance technology is an effective method for reconstructing three-dimensional geometry, which learns and predicts the three-dimensional geometry of a scene based on the radiance consistency field between multiple-view images. This technology has good results in static ground scenes, but when the input images are multi-temporal satellite images taken at different times, factors such as inconsistency of radiance information between images, seasonal changes, and dynamic changes can cause the network to fit incorrect geometric structures, and even cause the entire optimization process to fall into a local optimal solution, making it impossible to achieve stable and effective convergence, limiting the three-dimensional reconstruction effect of neural radiance field technology on satellite data. SUMMARY

[0007] Therefore, the present application proposes a shadow-aware satellite image neural radiance field reconstruction method, which optimizes the scene SDF based on additional satellite image shadow information, improves the convergence stability of the neural radiance field model on satellite data, and improves the success rate and accuracy of three-dimensional surface reconstruction of the scene.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] A shadow-aware satellite image neural radiance field surface three-dimensional reconstruction method, comprising the following steps:

[0010] Step S1: Based on the multi-temporal satellite image dataset of the region of interest, the sun direction vector of each image is calculated by the sun azimuth and elevation angle, image cropping, 16-bit to 8-bit bit reduction conversion, panchromatic and multispectral image fusion processing are performed, and the RPC model is optimized by bundle adjustment to improve the RPC accuracy, and a sparse point cloud is generated at the same time;

[0011] Step S2: according to the sparse point cloud, a scene enclosing sphere is calculated and the point cloud is normalized to a unit sphere space, a unified scale coordinate system is constructed, so that the coordinate range of the scene to be reconstructed is between -1 and 1;

[0012] Step S3: based on the unified scale coordinate system constructed in step S2, randomly selecting an image pixel and generating a light ray along the line of sight, sampling a three-dimensional point on the light ray , and constructing a training sample set by associating the coordinates of the three-dimensional point , the line of sight direction and the pixel color;

[0013] Step S4: constructing a neural radiance field network model and a shadow color network model, the neural radiance field network model includes an SDF network and a color network, wherein the SDF network predicts the signed distance SDF value of a spatial point , the color network fuses the SDF feature and the line of sight direction to output the color value of the spatial point , and the shadow color network generates the color vector of the shadow according to the sun direction obtained in step S1 ;

[0014] Step S5: based on the signed distance SDF value of the spatial point output by the SDF network, the color value of the pixel light ray is predicted based on the volume rendering integral formula;

[0015] Step S6: based on the sun direction vector in step S1, a secondary reflection light ray is emitted along the sun direction, and the shadow probability of the surface point is calculated according to the transmittance;

[0016] Step S7: combining the shadow probability in step S6 and the irradiance model to calculate the irradiance of the pixel light ray, and constructing a minimum loss function according to the error between the irradiance and the color value of the pixel light ray to optimize the parameters of the SDF network, the color network and the shadow color network in reverse, wherein the signed distance SDF value of the spatial point , the color value of the spatial point and the color vector of the shadow are input to the irradiance model;

[0017] Step S8: after the training is completed, based on the unified scale coordinate system in step S2, the spatial points are uniformly sampled in the coordinate range of -1 to 1 for each coordinate axis (x, y, z), and the SDF value of each point is predicted by inputting the spatial points into the trained SDF network, and a three-dimensional grid model is generated by using the Marching Cubes algorithm.

[0018] Further, step S1 specifically includes:

[0019] Step S1.1: reading the sun azimuth and the sun elevation Information, calculating the sun direction vector :

[0020] ;

[0021] Step S1.2: Determine the UTM coordinate range of the region of interest and convert it to WGS-84 geographic coordinates; project the region of interest onto the panchromatic and multispectral images using the RPC parameters of the satellite images, and cut according to the minimum bounding rectangle; after cutting, subtract the starting row / column number of the target region in the image from the original offset to update the row offset and column offset of the RPC parameters;

[0022] Step S1.3: Convert 16-bit data to 8-bit data using the maximum and minimum value stretching method, first traverse the entire 16-bit image data to determine the minimum gray value and the maximum gray value , then use the formula

[0023] ;

[0024] Map the 16-bit gray value of each pixel to the 8-bit gray value ;

[0025] Step S1.4: Extract the blue, green, and red bands from the multispectral image, and use the Brovey panchromatic sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image, generating a color image with high spatial resolution and rich spectral information.

[0026] Step S1.5: Use the feature extraction, feature matching, and bundle adjustment methods in the COLMAP software to correct the RPC parameters and generate sparse point clouds.

[0027] Further, step S2 specifically includes: normalizing the scene, calculating the outer bounding sphere of the scene based on the sparse point cloud generated by RPC optimization, determining the geometric boundary of the point cloud, and translating the center of gravity of the sparse point cloud to the origin of the coordinate system to achieve center alignment; then, scale the point cloud according to the radius of the outer bounding sphere, so that all sparse points fall within a standardized sphere with the origin as the center and a radius of 1, ensuring that the coordinate value range is controlled within [-1, 1].

[0028] Further, step S5 specifically includes:

[0029] Input the position of all sampling points on a pixel ray, the line-of-sight direction , and the line-of-sight direction to the SDF network, output the SDF and color values of all points, and then calculate the predicted color of the pixel ray using the volume rendering formula:

[0030] ;

[0031] is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, represents opacity, that is, the probability of each point blocking the propagation of light; is the color weight, which indicates the contribution of each sampling point to the final predicted color. and By sampling point Volume density at calculate:

[0032] ;

[0033] in, Represents the distance between two adjacent sampling points on the ray: ; Introduce the unimodal S-density function To model the relationship between surface distance SDF and color weight:

[0034] ;

[0035] in Represents the SDF value of a spatial point, is the Sigmoid function The derivative of

[0036] ; It is a learnable parameter that indicates the concentration of color weight near the surface of the object. The unimodal Sigmoid function constrains the color weight to increase as it approaches the surface and to reach its maximum value on the surface. After derivation, the opacity density is obtained. Relationship with SDF:

[0037] ;

[0038] Cumulative opacity and predicted color Further changed to:

[0039] ;

[0040] .

[0041] Furthermore, step S6 specifically includes:

[0042] The scene is represented as an implicit SDF field, and the scene surface is determined by the zero level set:

[0043] ;

[0044] The volume rendering framework learns the SDF and color of the scene by emitting countless pixel rays. Find surface points where the SDF is zero : Sampling on pixel rays 3D coordinate points , the SDF value of each point is expressed as ;set up for The function of , a ray passing through the surface of an object will have two adjacent sampling points, one inside the object with an SDF value less than 0, and the other outside the object with an SDF value greater than 0. The surface points are located at positions where the SDF is alternating between positive and negative:

[0045] ;

[0046] is a set of sampling points that meet the conditions; the surface point position is estimated by linear interpolation based on the sampling points:

[0047] ;

[0048] In determining pixel light Intersection with the 3D scene surface Afterwards, For direction, The secondary ray is emitted towards the sun from the starting point ; Sample far enough away on this ray points

[0049] , the SDF value of each point is , then the opacity density of each point is for:

[0050] ;

[0051] Transparent density Integrate and calculate the light Cumulative transparency of:

[0052] ;

[0053] Sunlight Previous point The probability of being occluded is determined by the cumulative transparency denotes the transmittance, in the volume rendering framework, the probability that a point is visible by the light source when a ray is emitted from the object surface to the sun, the cumulative transparency of the last sample point emitted from the object surface to the light source denotes the transmittance

[0054]

[0055] From this, based on the geometric relationship between the implicit SDF and the sun direction, the transmittance estimation of satellite imagery is realized, denotes the probability that a pixel is in the shadow, which is also the probability that a ray is in the shadow at the intersection position with the scene.

[0056] Further, the irradiance model in step S7 is represented as:

[0057]

[0058] wherein is the irradiance received by the satellite imagery, is the color of the scene itself, is the sun transmittance, is the shadow color;

[0059] corresponding to the color of the scene itself in the irradiance model , the irradiance model based on volume rendering is represented as:

[0060]

[0061] is the irradiance of the ray , is the color predicted by volume rendering, is the sun transmittance of the ray , determined by the scene SDF and the sun direction, the shadow color related to the sun direction, estimated by the shadow color network ; through the irradiance model, the color of the pixel is represented by direct illumination and indirect illumination, and the shadow information is also decoupled and included in the color prediction, and the loss function is updated as:

[0062]

[0063] wherein is the irradiance of the ray , is the color of the corresponding input satellite imagery pixel.

[0064] ​​​​​​​​​The present invention has the following beneficial effects:

[0065] (1) This paper uses a volume rendering framework based on a signed distance function (SDF) and a geometry-aware shadow detection method to accurately predict scene shadow information and optimize scene geometry to produce shadow effects consistent with the input image. The introduction of sunlight enables the network to infer the geometry of the ground objects that produce shadows based on image shadows, thereby demonstrating greater robustness in the reconstruction of vegetation-covered areas and highly reflective surfaces (such as glass curtain walls).

[0066] (2) Neural radiance field technology learns scene geometry through color. In the early stages of iteration, the network prioritizes black and white information with the strongest contrast. Shadows, as black areas, can provide the network with rough geometric information in the early stages of iteration, allowing the network to obtain relatively accurate initial geometric values. This mechanism effectively prevents the network from falling into local optimal solutions, improves convergence stability, and enhances the robustness of neural radiance field technology on satellite data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of a method for three-dimensional reconstruction of a satellite image neural radiometric site surface using shadow perception according to an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of three-dimensional scene normalization according to an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the architecture of a shadow-aware satellite image neural radiometric site surface 3D reconstruction method according to an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of the network structure of an embodiment of the present invention;

[0071] Figure 5 Schematic diagram of shadow detection based on secondary reflection according to an embodiment of the present invention;

[0072] Figure 6 This is an example of the shadow prediction result of an embodiment of the present invention;

[0073] Figure 7 Results of different methods on the DFC2019 dataset;

[0074] Figure 8 Schematic diagram of the effect of shadow information on 3D reconstruction results in an ablation experiment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0076] See also Figure 1 The embodiment of the present invention provides a shadow-aware satellite image neural radiation site surface three-dimensional reconstruction method, comprising the following steps:

[0077] Step S1.1: Read the sun azimuth from the satellite metadata file for each image and the sun's altitude angle Information, calculate the sun direction vector :

[0078]

[0079] Step S1.2: Satellite imagery typically covers a vast area, with a single image potentially containing hundreds of kilometers of ground information. However, in practice, the focus is often limited to a specific, smaller area of ​​interest (AOI). Therefore, the satellite imagery needs to be cropped to extract valid data within the AOI. To effectively crop large imagery, the UTM coordinate range of the AOI is first determined and converted to WGS-84 geographic coordinates. Next, using the satellite imagery's RPC parameters, the AOI is projected onto panchromatic and multispectral imagery and cropped based on a minimum bounding rectangle. After cropping, the row and column offsets of the RPC parameters must be updated. This involves subtracting the starting row / column number of the target area in the image from the original offset to ensure that the cropped image remains accurately geolocated under the new RPC model.

[0080] Step S1.3: Use the maximum and minimum value stretching method to convert the 16-bit data into 8-bit data. Specifically, first traverse the entire 16-bit image data and determine its minimum grayscale value. and the maximum grayscale value , then, using the formula

[0081]

[0082] Convert the 16-bit grayscale value of each pixel to Mapping to 8-bit grayscale values ;

[0083] Step S1.4: In order to obtain a high-resolution color image, the present application extracts blue, green and red bands from the multispectral image, and uses the Brovey panchromatic sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image, to generate a high-resolution color image.

[0084] Step S1.5: The feature extraction, feature matching and bundle adjustment method in the COLMAP software are used to correct the RPC parameters, while generating a sparse point cloud.

[0085] Step S2: As shown in Figure 2 , the scene is normalized, and based on the sparse point cloud generated by RPC optimization, the outer bounding sphere of the scene is calculated to determine the geometric boundary of the point cloud. The center of gravity of the point cloud is moved to the origin of the coordinate system to realize center alignment; then, the point cloud is scaled according to the radius of the outer bounding sphere, so that all sparse points fall within a standardized sphere with the origin as the center and a radius of 1, ensuring that the range of coordinate values is controlled within [-1, 1].

[0086] Step S3: A satellite image is randomly selected, and pixels are randomly sampled on the image. For each sampled pixel, a pixel ray (referred to as pixel light ) is formed from the center of the satellite sensor to the pixel, with a direction . Each light is emitted from the satellite center and intersects the normalized sphere at and . Between these two points, points can be sampled by distance:

[0087]

[0088] represent the light direction, is the number of sampling points, is the distance constant. The position of each sampling point, the line of sight direction , the image it belongs to and the pixel color are recorded to form a data sample. Multiple pixel lights are established and sampled to obtain a sample set. In this embodiment, 512 and 64 are taken.

[0089] Step S4: As shown in Figure 3 , the method of the present application uses three-dimensional coordinates , line of sight direction and sun direction As input, and use multi-layer perceptron (MLP) to predict the output one-dimensional SDF, three-dimensional object color and shadow color The network architecture includes SDF network , Color Network Each network consists of an input layer, multiple hidden layers and an output layer, using Softplus as the activation function. Specifically, Figure 4 As shown, SDF network It consists of eight hidden layers. The input layer is a three-dimensional position vector , while the output layer generates a 256-dimensional feature vector and a one-dimensional SDF. In addition, a link is established between the input layer and the output of the fourth layer to facilitate the flow of information. It consists of four hidden layers, the output layer is the color of the three-dimensional sampling point, and the input layer includes three vectors: the three-dimensional sampling point observation direction, the three-dimensional sampling point position and a 256-dimensional feature vector. Predicted shadow color , which consists of a 256-dimensional feature vector, and the input is the three-dimensional direction of the sun In addition, in order to improve the representation ability of low-dimensional vectors, the present invention uses vector and They are position-encoded before entering the network. Specifically, six different frequency components are used to encode the position Encode and expand its dimension from 3 to 39. Similarly, the direction of the light at the sampling point Four different frequency components are encoded, and their dimension is expanded from 3 to 27.

[0090] Step S5: Predict the color value of pixel light based on the volume rendering method of SDF. Gaze direction and gaze direction Input the SDF network and output the SDF and color values ​​of all points. Then the predicted color of the pixel light can be calculated by the volume rendering formula:

[0091]

[0092] is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, Represents opacity, that is, the probability of each point blocking the propagation of light. is the color weight, which represents the contribution of each sampling point to the final predicted color. and The body density at the sampling point is calculated as:

[0093]

[0094] where, represents the distance between two adjacent sampling points on the light ray: The color weight is indirectly determined by the body density , which is flexible in space without additional constraints. Ideally, the color weight at the object surface should be the maximum, and the farther away from the object, the smaller the color weight should be, i.e. the maximum value of the color weight should be distributed on the object surface without bias. To achieve such a constraint, the present application introduces a unimodal S-density function to model the relationship between the surface distance SDF and the color weight:

[0095]

[0096] where represents the SDF value of the spatial point, is the derivative of the Sigmoid function . . is a learnable parameter, indicating the concentration of the color weight near the object surface. The unimodal Sigmoid function constrains the color weight to increase near the surface and reach the maximum value on the surface. Through derivation, the relationship between the opacity density and the SDF can be obtained:

[0097]

[0098] The cumulative opacity and the predicted color can be further changed to:

[0099]

[0100]

[0101] Step S6: Calculate the probability of the surface point being in the shadow based on the secondary reflection of the light ray. Unlike the shadow detection method based on image recognition, the present application determines the shadow according to the visible relationship between the implicit geometry SDF of the scene and the sun. As shown in Figure 5 , according to the pose of the sensor, an image pixel emits a light ray to query the intersection position of the surface in the scene, and then another light ray is emitted from the intersection to the direction of the sun ​​, if the ray is not occluded by the scene's own geometry, then it means that the point is visible by the sun and in non-shadow (see Figure 5 ) in . Conversely, if is occluded by the scene's geometry, then it means Figure 5 that the point is in shadow (see ) in . Thus, shadow extraction can be achieved based on the scene's geometry. In addition, in volume rendering theory, a ray can be occluded by the space particles it passes through, so the visibility judgment is not non-zero or one, but the visibility probability, so the present application discusses how to achieve shadow detection under the framework of SDF-based volume rendering.

[0102] The method of the present application represents a three-dimensional space with an implicit SDF, estimates the probability that the surface of the scene is visible to the sun, i.e., the sun transmittance , by emitting a secondary ray. In order to calculate the sun transmittance of an image, first, a ray is emitted from a pixel to the scene to estimate the surface point intersected by the ray, and then a secondary ray is emitted from the surface point to the sun direction to calculate the probability of being occluded by its own geometry.

[0103] Specifically, the scene is represented as an implicit SDF field, and the surface of the scene can be determined by a zero-level set (Zero-level Set):

[0104]

[0105] The volume rendering framework learns the SDF and color in the scene by emitting countless pixel rays, and similarly, the surface point where the SDF is equal to zero can be found along the pixel ray : sample three-dimensional coordinate points on the pixel ray , and the SDF value of each point is represented as . In order to simplify, let be a function of , i.e. , the ray that passes through the surface of the object always has two adjacent sampling points, one inside the object, with an SDF value less than 0, and the other outside the object, with an SDF value greater than 0, and the surface point is exactly located at the position where the SDF alternates from positive to negative:

[0106]

[0107] is a set of sampling points that meet the condition. According to these discrete sampling points, the surface point position can be estimated by linear interpolation:

[0108]

[0109] For a ray, the scene point intersected by it There can be multiple, the invention chooses The smallest one, corresponding to the outermost surface point least likely to be occluded. In determining the pixel ray Intersection with three-dimensional scene surface After that Direction, The secondary ray is emitted from the starting point towards the sun Similarly, sample enough points On this ray Each point has an SDF value The opacity density of each point is :

[0110]

[0111] Finally, integrate the transparent density To calculate the cumulative transparency of the ray :

[0112]

[0113] The probability of the point on the sun ray Being occluded is represented by the cumulative transparency In the volume rendering framework, when a ray is emitted from the object surface to the sun, the probability of the point being visible to the light source is the transmittance, so the cumulative transparency of the last sampling point emitted from the object surface to the light source Indicates the transmittance . .

[0114]

[0115] From this, based on the geometric relationship between the implicit SDF and the sun direction, the transmittance estimation of satellite images is realized, Indicates the probability of the pixel emitting the ray Being in the shadow, which is also the probability of the intersection position of the ray With the scene being in the shadow, and the prediction result is as shown in Figure 6 .

[0116] Step S7: Calculate the color loss of the pixel based on the irradiance model, and optimize the network parameters in reverse. Traditional volume rendering models do not consider shadow effects and cannot handle inconsistent shadows in multi-temporal satellite images. The present application introduces an irradiance model to decouple direct light and indirect shadow: first, the color of the pixel light is regarded as the color of the object itself, representing the appearance of the object under the condition of no shadow; then, the irradiance of the pixel light is calculated jointly with the shadow probability to simulate the combined effect of light and shadow. Then, the calculated irradiance is compared with the actual color value of the pixel light, the difference between the two is calculated, and the difference is used as a loss function for training the network. By minimizing the loss function, the parameters of the SDF network, the color network and the shadow color network are optimized, so that the model can more accurately predict the geometric shape, appearance attribute and light and shadow information of the scene.

[0117] In the classical volume rendering method, the radiation information is determined by the color of the scene particles, and the shadow phenomenon is coupled in the color. Such assumption is satisfied in static scenes where the light source is unchanged, but for multi-temporal satellite data, each image has different sun illumination direction and shadow, which leads to the fact that the basic radiation model no longer meets the actual problem. Therefore, the present application uses an irradiance model to decouple color based on the theory of volume rendering:

[0118]

[0119] The irradiance received by the satellite image is composed of direct light and indirect light . Among them, the direct light refers to the color of the sunlight directly irradiating to the ground and being reflected to the satellite image pixel by the ground. The indirect light involves more complex physical processes, which describes the color of the light that reaches the blocked area through multiple reflections after being blocked by the object on the ground. This part of the color is mainly obtained by atmospheric secondary reflection, and is therefore called indirect light. In order to model these processes, the solar transmittance is introduced, which represents the probability that the object is directly visible to the sun, so the direct light can be calculated by multiplying the color of the scene itself and the solar transmittance . The indirect light is determined by the color of the scene itself , the transmittance and the shadow color . Therefore, the irradiance model is represented as:

[0120]

[0121] Classical volume rendering assumes that the color of the scene is constant, which corresponds to the color of the scene itself in the irradiance model. , then the irradiance model based on volume rendering can be expressed as:

[0122]

[0123] For light The irradiance, The predicted color for volume rendering, For light The solar transmittance is determined by the scene SDF and the sun direction, and the shadow color Related to the sun direction, by the shadow color network Estimation. Through the irradiance model, the color of the pixel is represented by direct and indirect lighting, and the shadow information is also decoupled and included in the color prediction, so the loss function can be updated as:

[0124]

[0125] in For light The irradiance, For light The corresponding pixel color of the input satellite image.

[0126] Step S8: After iterating steps S3 to S7 300,000 times, network training is completed. Uniform sampling prediction is performed within the 3D scene space to obtain the SDF values ​​of all sampling points. These values ​​are then input into the Marching Cubes method to generate a 3D mesh model. Specifically, the Marching Cubes algorithm is used to process the SDF values ​​of all sampling points, extracting isosurfaces with SDF values ​​of 0 to generate a continuous and smooth 3D mesh surface, and outputting the 3D mesh model. The number of samples within the 3D scene space can be 128*128*128.

[0127] Example:

[0128] This paper uses the DFC2019 dataset to evaluate different methods, including four scenarios: JAX_004, JAX_068, and JAX_260. Figure 7The grid reconstruction results of the method of the application and comparative methods (S2P, S-NeRF, Sat-NeRF and VisSat) are shown. Overall, there are obvious noise points in the results of S2P, S-NeRF and Sat-NeRF, especially in the area of transient objects (such as vehicles), which may be due to the lack of geometric regularization. The method of the application suppresses the surface geometry changes caused by noise by regularizing the SDF field, and the reconstruction result is smoother and more detailed. In terms of building contours, the techniques based on volume rendering (S-NeRF, Sat-NeRF and the method of the application) can learn sharp edges from color changes, which are superior to the blurred results of VisSat and S2P. In vegetation reconstruction, the method of the application uses shadow information to infer geometry, avoiding the influence of color changes, and obtaining more complete vegetation reconstruction results, while other methods (such as VisSat and S2P) have blurred or missing phenomena.

[0129] Figure 8 The ablation experiment results are shown, and the reconstruction effects of the basic method (without using shadow information) and the method of the application (adding a shadow perception module) are compared. The basic method in multi-temporal satellite images leads to unstable reconstruction results, geometric errors and void phenomena due to seasonal differences in vegetation and high reflectivity of glass curtain walls. For example, in the JAX_004, JAX_068 and JAX_260 scenes, the color change of the pixels in the vegetation area causes the network to fall into a local optimal solution, affecting the global geometric accuracy; while in the JAX_167 scene, the strong reflective properties of the glass curtain wall cause the color consistency-based method to fail to reconstruct. In contrast, the method of the application uses the shadow perception module to infer geometric details using shadow information, significantly improving the stability and completeness of the reconstruction. For example, in the JAX_214 scene, the method of the application restores the details of the roof through shadow information; in the JAX_167 scene, the shadow-based geometric perception method successfully reconstructs the complete structure of the glass curtain wall. Experiments show that the introduction of shadow information effectively solves the convergence instability problem caused by vegetation and reflective surfaces in multi-temporal images, and improves the geometric accuracy and completeness of the reconstruction results.

[0130] The application introduces an irradiance model to decouple shadow information, decomposes the predicted color into the color of the object itself, direct light and indirect light by simulating the imaging process, and proposes a shadow detection method for geometric perception, based on the spatial representation of the signed distance function (SDF), simulates the secondary projection process of sunlight, and realizes accurate prediction of the shadow of the scene; finally, using the sun direction information, the geometric shape that produces the shadow is inferred, providing geometric guidance for the learning of neural implicit surfaces, and improving the convergence stability of the neural radiance field model in complex satellite data.

[0131] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements within the technical scope disclosed by the present application, which can be easily thought by any person skilled in the art, should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A shadow-aware satellite image neural radiometric site surface 3D reconstruction method, characterized in that: The steps include: Step S1: Based on the multi-temporal satellite image dataset of the area of ​​interest, the sun direction vector of each image is calculated using the sun azimuth and altitude angles. Image cropping, 16-bit to 8-bit downscaling, and panchromatic and multispectral image fusion are performed. The RPC model is optimized using bundle adjustment to improve RPC accuracy, and a sparse point cloud is generated. Step S2: Computing a bounding sphere outside the sparse point cloud scene and normalizing the point cloud to a unit sphere space, constructing a unified scale coordinate system so that the coordinate range of the scene to be reconstructed is between -1 and 1; Step S3: Based on the unified scale coordinate system constructed in step S2, randomly select image pixels and generate light along the line of sight, sampling three-dimensional points on the light , associated 3D points The training sample set is constructed by using the coordinates, sight direction and pixel color of the pixel; Step S4: Construct a neural radiation field network model and a shadow color network model, wherein the neural radiation field network model includes an SDF network and a color network, wherein the SDF network predicts the signed distance SDF value of a spatial point , the color network fuses the SDF features and the sight direction to output the color value of the spatial point , the shadow color network generates the shadow color vector according to the sun direction obtained in step S1 ; Step S5: using the signed distance SDF value of the spatial point output by the SDF network, the color value of the pixel light is predicted based on the volume rendering integral formula; Step S6: Based on the sun direction vector obtained in step S1, a secondary reflected ray is emitted along the sun direction, and the shadow probability of the surface point is calculated according to the transmittance. Step S7: Calculate the irradiance of the pixel light by combining the shadow probability and irradiance model of step S6, and construct a minimization loss function based on the error between the irradiance and the color value of the pixel light to reversely optimize the parameters of the SDF network, color network and shadow color network, where the signed distance SDF value of the spatial point obtained in step S4 is , the color value of the spatial point and shadow color vector As input to the irradiance model; Step S8: After training is completed, based on the unified scale coordinate system of step S2, spatial points are uniformly sampled in the coordinate range of -1 to 1 for each coordinate axis (x, y, z), and the SDF value of each point is predicted by the trained SDF network. The Marching Cubes algorithm is used to generate a three-dimensional mesh model.

2. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S1 specifically includes: Step S1.1: Read the sun azimuth from the satellite metadata file for each image and the sun's altitude angle Information, calculate the sun direction vector : ； Step S1.2: Determine the UTM coordinate range of the area of ​​interest and convert it to WGS-84 geographic coordinates; use the RPC parameters of the satellite image to project the area of ​​interest onto the panchromatic and multispectral images and crop them according to the minimum bounding rectangle; after cropping, subtract the starting row / column number of the target area in the image from the original offset to update the row offset and column offset of the RPC parameters; Step S1.3: Use the maximum and minimum value stretching method to convert the 16-bit data into 8-bit data. First, traverse the entire 16-bit image data and determine its minimum grayscale value. and the maximum grayscale value , then using the formula ； Convert the 16-bit grayscale value of each pixel to Mapping to 8-bit grayscale values ; Step S1.4: Extract blue, green, and red bands from the multispectral image and use the Brovey pan-sharpening algorithm to fuse the high spatial resolution of the panchromatic image with the color information of the multispectral image to generate a color image with both high spatial resolution and rich spectral information. Step S1.5: Use the feature extraction, feature matching, and bundle adjustment methods in the COLMAP software to correct the RPC parameters and generate a sparse point cloud.

3. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S2 specifically includes: normalizing the scene, determining the geometric boundary of the point cloud based on the bounding sphere of the sparse point cloud computing scene generated by RPC optimization, and translating the center of gravity of the sparse point cloud to the origin of the coordinate system to achieve center alignment; then, scaling the point cloud according to the radius of the bounding sphere so that all sparse points fall within a standardized sphere with a radius of 1 centered at the origin, ensuring that the coordinate value range is controlled between [-1, 1].

4. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S5 specifically includes: All sampling point positions on a pixel ray Gaze direction and gaze direction Input the SDF network and output the SDF and color values ​​of all points. The predicted color of the pixel light is calculated by the volume rendering formula: ； is the cumulative perspective, which means the angle along the ray from the starting point arrive The probability of not encountering any obstacles in the process, represents opacity, that is, the probability of each point blocking the propagation of light; is the color weight, which indicates the contribution of each sampling point to the final predicted color. and By sampling point Volume density at calculate: ； in, Represents the distance between two adjacent sampling points on the ray: ; Introduce the unimodal S-density function To model the relationship between surface distance SDF and color weight: ； in Represents the SDF value of a spatial point, is the Sigmoid function The derivative of ; It is a learnable parameter that indicates the concentration of color weight near the surface of the object. The unimodal Sigmoid function constrains the color weight to increase as it approaches the surface and to reach its maximum value on the surface. After derivation, the opacity density is obtained. Relationship with SDF: ； Cumulative opacity and predicted color Further changed to: ； 。 5. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 1, characterized in that: Step S6 specifically includes: The scene is represented as an implicit SDF field, and the scene surface is determined by the zero level set: ； The volume rendering framework learns the SDF and color of the scene by emitting countless pixel rays. Find surface points where the SDF is zero : Sampling on pixel rays 3D coordinate points , the SDF value of each point is expressed as ;set up for The function of , a ray passing through the surface of an object will have two adjacent sampling points, one inside the object with an SDF value less than 0, and the other outside the object with an SDF value greater than 0. The surface points are located at positions where the SDF is alternating between positive and negative: ； is a set of sampling points that meet the conditions; the surface point position is estimated by linear interpolation based on the sampling points: ； In determining pixel light Intersection with the 3D scene surface Afterwards, For direction, The secondary ray is emitted towards the sun from the starting point ; Sample far enough away on this ray points , the SDF value of each point is , then the opacity density of each point is for: ； Transparent density Integrate and calculate the light Cumulative transparency of: ； Sunlight Previous point The probability of being occluded is determined by the cumulative transparency Indicates that in the volume rendering framework, when light is emitted from the surface of an object to the sun, the probability that the point is visible to the light source is the transmittance, which is the cumulative transparency of the last sampling point emitted from the surface of the object to the light source. Indicates transmittance : ； Since then, the transmittance estimation of satellite images has been realized based on the geometric relationship between implicit SDF and the direction of the sun. Indicates the emission of light The probability that a pixel is in the shadow is also the probability that the light The probability that the intersection position with the scene is in shadow.

6. The shadow-aware satellite image neural radiation site surface 3D reconstruction method according to claim 5, characterized in that: The irradiance model in step S7 is expressed as: ； in, is the irradiance received by the satellite image, is the color of the scene itself, is the solar transmittance, is the shadow color; Corresponding to the scene color in the irradiance model , then the irradiance model based on volume rendering is expressed as: ； For light The irradiance, The predicted color for volume rendering, For light The solar transmittance is determined by the scene SDF and the sun direction, and the shadow color Related to the sun direction, by the shadow color network Estimation; Through the irradiance model, the color of the pixel is represented by direct and indirect lighting, and the shadow information is also decoupled and included in the color prediction. The loss function is updated as follows: ； in For light The irradiance, For light The corresponding pixel color of the input satellite image.

Images