Method and system for digital surface modeling based on training a neural radiance field model

The neural radiance field model addresses the inefficiencies of traditional digital surface modeling by predicting volume density and albedo color from multi-date satellite images, achieving superior performance in digital surface modeling and novel view synthesis, particularly by rendering geometrically consistent shadows and handling raw satellite images.

WO2024246575A1PCT designated stage expired Publication Date: 2024-12-05KAYRROS
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
PCT/IB2023/000495
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Current digital surface modeling from remote sensing images relies heavily on stereo or tri-stereo image products, which are expensive and inefficient, while neural rendering methods, such as NeRF, have shown promise but struggle to perform effectively with multi-date satellite imagery, resulting in inferior performance compared to traditional Multi-View Stereo approaches.

Method used

A computer-implemented method using a neural radiance field model to predict volume density and albedo color from multi-date remote sensing images, incorporating a base portion for geometry prediction, a static content portion for albedo color prediction, and a transient content portion for handling transient phenomena, with a loss function that compares predicted radiances to ground truth, allowing for improved 3D reconstruction and digital surface modeling.

Benefits of technology

The method achieves state-of-the-art performance in novel view synthesis and digital surface modeling, particularly by rendering geometrically consistent shadows and handling raw satellite images, improving upon previous NeRF variants by generalizing across different solar directions and refining geometry, thus overcoming the limitations of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure IB2023000495_05122024_PF_FP_ABST
    Figure IB2023000495_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method, comprising: obtaining multi-date images of a ground area of interest; using the multi-date images, perform machine-learning of a neural radiance field model to predict a volume density (ex) and an albedo color (cn) at a 3D location (x); and determining a digital surface model of the ground area of interest based on volume densities predicted by the machine-learned neural radiance field model. Performing machine-learning comprises: for each camera ray intersecting a surface point of the ground area of interest: o determining a transmittance of a solar ray which intersects the surface point based on volume densities predicted by the neural radiance field model at 3D locations along the solar ray; o determining a radiance of the camera ray by computing the product of the albedo color (cn) predicted by the neural radiance field model at the surface point and an irradiance function weighted by the determined transmittance; evaluating a loss function that compares the radiances determined for each camera ray with ground truth radiances derived from the multi-date images; modifying the neural radiance field model based on the evaluated loss function.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method and system for digital surface modeling based on training a neural radiance field model

[0002] TECHNICAL FIELD

[0003] The field of the invention is that of methods and systems for constructing volumetric (3D) models of the Earth's surface based on remote sensing observations. The invention more particularly uses methods from the computer vision field, namely, Neural Radiance Fields, to perform photogrammetry using multi-date remote sensing images of a ground area of interest.

[0004] DESCRIPTION OF RELATED ART

[0005] Many Earth observation satellites acquire images periodically over the same areas, contributing to large multidate collections of satellite images. These collections offer the opportunity to estimate the topography of the Earth's land cover, which knowledge can be relevant for several remote sensing applications, including terrain mapping for flood risk mitigation, biomass estimation, land cover classification and change detection (i.e., the capacity to observe the evolution of a site over time and to identify permanent and transient structures).

[0006] However, when it comes to digital surface modeling from remote sensing, multi-date imagery is usually disregarded in favor of a few stereo or tri-stereo image products. The latter are more expensive, as images are acquired almost simultaneously to facilitate the use of classic photogrammetry.

[0007] The state of the art in digital surface modelling from satellite images relies on the availability of stereo image products. The reason of this is that the Multi-View Stereo (MVS) logic is predominantly adopted to address the task as a multi-pair problem instead of a real multi-view problem.

[0008] Parallel to this scenario, MVS approaches for 3D reconstruction have lost hegemony for common close-range imagery. Since the release of NeRF [1] in 2020, neural rendering methods have become extremely popular for conventional image collections, as they represent a highly accurate, unsupervised and truly multi-view solution to the 3D modelling problem. Satellite imagery is a niche where neural rendering still has a long way to go, but some pioneering work has already started to bridge the gap between the two worlds. In particular, Shadow NeRF (or S-NeRF) [2] and Satellite NeRF (or Sat-NeRF) [3] explored NeRF for multi-date satellite images, but their performance remained inferior to state-of-the-art MVS.

[0009] BRIEF DESCRIPTION OF THE INVENTION

[0010] The invention aims to improve the performance of neural rendering methods to perform photogrammetry using multi-date remote sensing images.

[0011] To this purpose, the invention proposes a computer-implemented method, comprising: obtaining multi-date images of a ground area of interest captured by a sensor onboard an Earth observation satellite; performing machine-learning of a neural radiance field model using the multi-date images, wherein the neural radiance field model is configured to predict a volume density and an albedo color at a 3D location; and determining a digital surface model of the ground area of interest based on volume densities predicted by the machine-learned neural radiance field model.

[0012] Performing machine-learning of the neural radiance field model based on the multidate images comprises: for each camera ray of a plurality of camera rays each intersecting a surface point of the ground area of interest: o determining a transmittance of a solar ray which intersects the surface point based on volume densities predicted by the neural radiance field model at 3D locations along the solar ray; o determining a radiance of the camera ray by computing the product of an albedo color predicted by the neural radiance field model at the surface point and an irradiance function weighted by the determined transmittance; evaluating a loss function that compares the radiances determined for each camera ray with ground truth radiances derived from the multi-date images; modifying the neural radiance field model based on the evaluated loss function.

[0013] Certain preferred, but non-limiting aspects of the method are as follows: determining a radiance of the camera ray further comprises applying an imagespecific affine transformation to the computed product; the image-specific affine transformation is defined by parameters which are predicted by the neural radiance field model as image-specific embeddings; the irradiance function weighted by the determined transmittance is the determined transmittance and α an ambient color predicted by an auxiliary portion of the neural radiance field model based on a solar direction; the neural radiance field model comprises: o a base portion configured to predict the volume density and output a latent representation, based on the 3D location; o a static content portion configured to predict the albedo color based on the latent representation output by the base portion; o a transient content portion configured to predict one or more coefficients based on the latent representation output by the base portion and an image-dependent embedding; the one or one or more coefficients comprise a transient scalar used to modulate the transmittance of the solar ray; the one or one or more coefficients comprise an uncertainty coefficient used to modulate the loss function; an origin and a direction of a camera ray is determined using a localization function of a RPC, Rational Polynomial Coefficients, camera model; it further comprises adjusting the RPC camera model based on one or more transformation parameters which are predicted by the neural radiance field model as image-specific embeddings; the multi-date images are panchromatic images; performing machine learning further comprises pan-sharpening the panchromatic images; the multi-date images are radar images; the multi-date images are optical images.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other aspects, aims, advantages and features of the invention will better appear upon reading the following detailed description of preferred embodiments thereof, provided as a non-limiting example, and done in reference to the appended drawings, in which:

[0016] - Figure 1 shows steps of a method according to a possible embodiment of the invention;

[0017] - Figure 2 illustrates a network architecture for the NeRF model used in a possible embodiment of the invention;

[0018] - Figure 3 illustrates the shadow model used in the present disclosure in which the transmittance of the last point in a solar ray in the direction of the sun indicates the amount of shadow at a surface point;

[0019] - Figure 4 shows ECEF vs. UTM-based representation of geographic 3D point coordinates.

[0020] DETAILED DESCRIPTION OF THE INVENTION

[0021] A NeRF, or Neural Radiance Field, is a continuous function F that represents the geometry and appearance of a 3D scene. F is encoded using a fully-connected neural network or multilayer perceptron (MLP) that is specific to each scene and does not generalize to others. In its simplest form, a NeRF MLP predicts the observed RGB color c and a non-negative scalar volume density σ at a 3D point of the scene of coordinates x seen from a viewing direction d, i.e.,

[0022] F: (x, d) → (σ, c). (1)

[0023] Note that the volume density c can be derived from a surface model like in [5] or [6].

[0024] Using a collection of input images and their associated camera poses, the MLP encoding is optimized by casting individual rays that trace across the scene and project onto the known pixels of the input images. Each ray r is defined by a point of origin o and a direction vector d. The color c(r) of a ray r(t) = o + td is computed as with Ti, αiand cidefined in the below paragraph.

[0025] The rendered color c(r) results from integrating the colors cipredicted at different points of the ray r. Each ray r is therefore discretized into N 3D points xibetween the near and far bounds of the scene, tnand tƒ. Each point xiin r is obtained as xi= o + tid, where ti∈ [tn, tƒ]. The contribution of each point xiin r to the rendered color depends on the opacity αiand the transmittance Ti: where is the distance between two consecutive points along the ray, i.e., = ti +1- ti,

[0026] Both factors Ti, αitake values in the interval [0,1] and depend only on the volume density σ that defines the geometry. The opacity increases with σ and is the probability that xibelongs to a non-transparent surface. The transmittance is the probability that light reaches xiwithout hitting previous opaque points in the ray r and is used to handle occlusions.

[0027] Using (3), the depth d(r) observed by casting a ray r can be determined in a similar manner to the color as

[0028] The MLP can, for example, be optimized by minimizing the mean squared error (MSE) between the rendered color c(r) and the real color cGT(r) of the image pixel intersected by each ray r: where R is a batch of arbitrarily selected rays.

[0029] NeRF opened the door to a large number of variants that address the limitations of the original method by adding auxiliary networks or heads to the single MLP architecture originally used.

[0030] For instance, NeRF in the Wild (or NeRF-W) [4] was created for collections of hundreds or thousands of multi-date street view images. It added auxiliary heads to predict transient colors and densities in parallel to the original multi-view consistent outputs. Shadow NeRF (or S-NeRF) [2] worked with tens of satellite images modeled locally as pinhole cameras and uses an auxiliary head to render building shadows in the input images as a function of the direction of solar rays. Satellite NeRF (or Sat-NeRF) [3] improved the results of S-NeRF by directly employing the actual RPC (Rational Polynomial Coefficients) camera models associated with the satellite images and by adding an auxiliary head to handle transient objects similarly to NeRF-W.

[0031] The present disclosure is directed to a NeRF variant of improved performance for 3D reconstruction from multiple remote sensing images. It is named Earth Observation NeRf (EO-NeRF).

[0032] With reference to figure 1, the present disclosure relates to a computer-implemented method which comprises as step OBT of obtaining, by a computing system, multi-date images of a ground area of interest captured by a sensor onboard an Earth observation satellite. The multi-date images are images of the same ground area of interest captured at different dates. The multi-date images provide multiple views of the ground area of interest. The multi-date images may be optical images or radar images.

[0033] In the case of radar images, the NeRF MLP needs to predict a non-negative scalar volume density σ at a 3D point of the scene coordinates x as well as the normal n associated to this density. Instead of associating a color to a ray, in the radar case a set of predictions uiis associated to a set of rays rjsuch that such that in, a zero-doppler case, the signal corresponding to uican be approximated by and where αi,jcorrespond to the αi, defined previously, associated to ray rj.

[0034] Still with reference to figure 1, the computer-implemented method further comprises a step ML-NeRF of performing, by the computer system, machine-learning of a neural radiance field model using the multi-date images, wherein the neural radiance field model is configured to predict a volume density σ and an albedo color cαat a 3D point of coordinates x.

[0035] Then, still with reference to figure 1, the computer-implemented method comprises a step DSM of determining, by the computer system, a digital surface model of the ground area of interest based on volume densities predicted by the machine-learned neural radiance field model.

[0036] Machine-learning the neural radiance field model

[0037] Figure 2 shows a network architecture for the NeRF model used in a possible embodiment of the invention. This network comprises a base portion MLP 1 configured to predict the volume density σ and output a latent representation z, based on a 3D location x. The base portion MLP1 may be a multilayer perceptron as in the original NeRF [1] which takes as input a 3D point coordinates x and predicts the volume density σ and a latent representation z.

[0038] The network further comprises a static content portion MLP2 configured to predict an albedo color cαbased on the latent representation z output by the base portion MLP 1. In other words, the latent representation z extracted by the main portion MLP 1 is plugged into an auxiliary head forming the static content portion MLP 2 to predict the albedo color cα.

[0039] In a possible embodiment, the network may further comprise a transient content portion MLP3 configured to predict one or more coefficients based on the latent representation z output by the base portion MLP 1 and an image-dependent embedding tj. The one or more coefficients may comprise a transient scalar τ used as detailed hereafter to modulate the transmittance of solar rays. The one or more coefficients may comprise an uncertainty coefficient β used as detailed hereafter to modulate the loss function employed for training the network.

[0040] The network further comprises an auxiliary portion MLP 4 disconnected from the rest and configured to predict an ambient color a which is constant for all points x and only depends on the solar direction dsun.

[0041] Additional image-dependent embeddings Aj, bjmay be learned in the same manner as tjby using embedding modules that take as input the image index j. As detailed hereafter, these additional image-dependent embeddings Aj, bjmay constitute parameters of an image-specific affine transformation used for mitigating color biases. Such an affine transformation proves useful in that the method may then be implemented using unprocessed, i.e., raw, satellite images.

[0042] The network may use positional encoding for inputs x and dsunas defined in NeRF [1], as well as ReLU non-linearity between MLP layers.

[0043] In a possible embodiment, the network is trained to predict the following function: F: (x, dsun, tj) → (σ, cα, α, β, τ, Aj, bj). (6)

[0044] The inputs of (6) are the same as in Sat-NeRF [3]: x, a 3-valued vector with the spatial coordinates of a point located in the scene volume, dsun, a 3-valued direction vector encoding the direction of solar rays. For each obtained image, dSuncan be extracted from the azimuth and elevation angles (0, φ) that indicate the position of the sun in the satellite image metadata; tj, an embedding vector, learned as a function of the image index j.

[0045] The outputs of (6) are the following: σ : Volume density scalar at location x; cα: Albedo RGB color, related only to the geometry, i.e., the spatial coordinates x; a: Ambient RGB color that defines a global hue according to the solar direction dSun; β : Uncertainty scalar related to the probability that the color of x corresponds to a transient object; τ: Transient scalar, learned as a function of x and tj;

[0046] Aj, bj; RGB vectors encoding a color affine transformation between the albedo and the current image j.

[0047] The outputs of (6) can be used to render the color c(r) of each ray r as follows: where

[0048] According to (7), the output color c(r) results from the product of the albedo cαwith an irradiance function which is weighted by a transmittance quantity s(r), i.e., a shadow-aware scaling scalar which can be interpreted as a solar visibility probability.

[0049] According to (7), this product may be subjected to an image-specific affine transformation defined by Ajand bj.

[0050] The irradiance function weighted by the transmittance quantity s(r) can be understood a weighed sum of a white light source modelling the Sun and of a learned sky color (the ambient color a) using s(r) and (1 — s(r)) as weight respectively. s(r) can be a scalar in the interval [0,1]. Low values of s(r) correspond to points in the shade or involving transient objects, where the ambient color a is allowed to be stronger. High values of s(r) correspond to points of the scene that can be fully explained by the albedo color cα.

[0051] Both S-NeRF [2] and Sat-NeRF [3] predicted shadows as a color property depending on the solar direction. In other words, their model was trained to predict the shadow-aware scaling scalar s used in the weighting of the irradiance function. This leads the model to overfit the input views, taking advantage of that freedom to render other complex changes such as vegetation color or transient objects. As a result, it does not generalize well to solar directions different from those in the input images. S-NeRF and Sat-NeRF attempted to minimize this misbehavior by adding a solar correction term to the loss function that seeks to bring shadow renderings closer to binary masks strictly related to object shadows. At the same time, this choice forces the network to explain differently the other transient phenomena, which may have side effects, so it is necessary to set an appropriate weight for the auxiliary solar correction term.

[0052] In the present disclosure, a neural radiance field model is disclosed which does not predict shadows (i.e., the shadow-aware scaling scalar s is not an output of the network) but instead renders them based on the geometry at each epoch of the machine learning step ML-NeRF.

[0053] As detailed hereafter, performing machine-learning of the neural radiance field model based on the multi-date images comprises, at each of a plurality of iterations (or epochs), for each camera ray r of a plurality of camera rays each intersecting a surface point xsof the ground area of interest: determining a transmittance of a solar ray rsunwhich intersects the surface point xsbased on volume densities predicted by the neural radiance field model at 3D locations along the solar ray; determining a radiance c(r) of the camera ray r by computing, in accordance with equation (7), the product of an albedo color cαpredicted by the neural radiance field model at the surface point xsand an irradiance function weighted by the determined transmittance in accordance with equation (8).

[0054] Then, performing machine-learning of the neural radiance field model comprises, at each iteration, evaluating a loss function that compares the radiances c(r) determined for each camera ray with ground truth radiances derived from the multi-date images and modifying the neural radiance field model based on the evaluated loss function.

[0055] Determining a radiance c(r) of a camera ray r

[0056] The rendering of geometrically consistent shadows is hereafter denoted as sgeo. It is performed for each of a plurality of camera rays. As shown on figure 3, a camera ray r traces across the imaged scene from a position of the sensor CAM at the time of capturing one of the obtained multi-date images and projects onto a pixel of said one of the obtained multi-date images. A camera ray r intersects a surface point xsof the ground area of interest. The surface point xsmay be determined using equation (4). Then, this surface point xsis used to cast a solar ray rsunwhich originates from the surface point xstowards the location of the sun SUN at the time of capturing the multi-date image associated with the camera ray r. Based on volume densities predicted by the neural radiance field model at 3D locations along the solar ray, a transmittance of the solar ray in the imaged scene can be determined, e.g., as the transmittance T(rsun(tN)) given by equation (3) of the last point tNin rsun. This transmittance is equal to the amount of geometric shadow, e.g.,sgeo (r) = T(rsun(tN)), which reveals whether xsis visible in the direction of the sun or not. In this respect, figure 3 schematically depicts the variation of opacity a and transmittance T along a solar ray in a case where the surface point is not occluded and in a case where the surface point is occluded. In the latter case, opacity therefore increases at points along the solar ray where the predicted volume density is important and which correspond to building(s) masking the sun. The determined transmittance sgeo(r) may then be used as s(r) in equation (8) to weight the irradiance function, thereby allowing for determining a radiance c(r) of the camera ray r by computing, in accordance with equation (7), the product of the albedo color cαpredicted by the neural radiance field model at the surface point xsand the irradiance function weighted, in accordance with equation (8), by the determined transmittance snpn(r).

[0057] The value of sgeo(r) renders shadows based on the geometry of the scene and ignores the other transient phenomena in each image. To represent the latter, use can be made in a possible embodiment of the transient portion MLP 3 of the network to predict the transient scalar r(r) and to use the predicted transient scalar r(r) to modulate the transmittance of the solar ray. τ is an arbitrary scalar between 0 and 1 that is not bound by geometry. Modulating the transmittance of the solar ray may comprise computing the product of sgeo(r) with τ(r) so as to define the weight s(r) of the irradiance model: where the dependency of τ on tjis omitted for simplicity and xi∈ r. The use of a product in (9) guarantees that s(r) always contains geometrically consistent shadows in addition to the rest of transient phenomena. τ(r) = 1 can be set at test time for transient object removal.

[0058] Compared to previous work, the strength of (9) is that it decouples shadows from other transient phenomena. Note that casting solar rays is also necessary to compute the solar correction term used in S-NeRF and Sat-NeRF. Here solar rays are used in a more direct way, which does not require modifying the loss function or tuning any hyperparameters to balance the optimization process.

[0059] Evaluating a loss function and modifying the neural radiance field model based on the evaluated loss function

[0060] A loss function similar to NeRF-W [4] and Sat-NeRF [3] may be used in which the uncertainty scalar β reduces the contribution of camera rays emitted from pixels with a high probability of representing transient phenomena: where β'(r) = β (r) + βmin. As in [3], pmin= 0.05 and η = 3 can be used to avoid negative values in the logarithm. The logarithm prevents β from converging to infinity to minimize equation (10) and encourages the network to correctly use β to identify which points are worthy of being regarded as transient or not depending on each image.

[0061] The back-propagation technique may then be used to update the neural radiance field model.

[0062] UTM-based point coordinates

[0063] S-NeRF [2] approximated the RPC camera models of the input images as pinhole camera models. Sat-NeRF [3] demonstrated the benefits in accuracy of directly using the RPC models to sample points in the 3D space. For that purpose, Sat-NeRF uses Earthcentered Earth-fixed (ECEF) coordinates to represent points in the 3D space. In the present disclosure, the same RPC-based sampling as in [3] may be used for casting the rays which impinges onto image pixels. According to such a sampling, an origin and a direction of a camera ray is determined using a localization function of the RPC camera model.

[0064] In a preferred embodiment, UTM coordinates and altitude are used to represent 3D points, rather than the ECEF coordinates, as they locally preserve the properties of a Cartesian system and offer the advantage that the altitude of the scene is aligned with the z-axis.

[0065] In this respect, Figure 4 shows (a) ECEF vs. (b) UTM-based representation of geographic 3D point coordinates. The scene has an altitude range of [hmin, hmax]. Using the same RPC camera, the rays originate at different pixels, localized at hmax, and end at the same pixel, localized at hmin. All coordinates are normalized in the interval [—1,1]. As shown by figure 4, the UTM-based representation ensures a better use of the space occupied by the scene with respect to the 3D volume that contains it.

[0066] Internal bundle adjustment

[0067] Sat-NeRF [3] stressed the importance of bundle adjustment to ensure consistency between the input RPC camera models associated with the satellite images. However, it proposed to perform the bundle adjustment correction before the MLP optimization, using an independent software. In a possible embodiment of the present disclose, it is proposed to refine the camera models simultaneously during the MLP optimization. To this purpose, the machine-learning step may further comprises adjusting the RPC camera models based on one or more transformation parameters which are predicted by the neural radiance field model as image-specific embeddings.

[0068] A simple way to integrate the refinement of RPC camera models into the MLP optimization is based on the assumption that RPC cameras can be locally approximated as affine projection models. Under such assumption it is possible to refine each RPC by composition of the projection function with an offset correction (i.e., a translation on the image plane). Using the UTM-based coordinate system, the altitude of the scene is aligned with the z-axis and the ray origin related to each pixel lies on the upper plane of the 3D volume (see Figure 4), which is coincident with the maximum altitude of the scene.

[0069] Thus, the offset correction can be approximated as a displacement in the x and y axis. If r is a ray intersecting the j-th camera, with origin o and direction d, the bundle adjusted version of r can be approximated as r(t) = (o + qj) + td such that qj = (q1, q2, 0)j. (11)

[0070] Expression (11) replaces the usual r(t) = o + td and is constrained to points inside the boundaries of the cube containing the scene to prevent large displacements. In order to learn the offset coefficients q1and q2of each camera, the NeRF network can use a two- dimensional embedding module in the same fashion as for tj, Ajand bj. In the following Experiments section, the results show that using (11) significantly narrows the gap compared to correcting the camera models separately before optimizing the MLP.

[0071] Internal pan-sharpening

[0072] S-NeRF [2] and Sat-NeRF [3] both work with already pan-sharpened images. This means a pan-sharpening step consisting in fusing the panchromatic image with the different color channels was necessarily performed before the MLP optimization using an independent software.

[0073] In a possible embodiment of the present disclosure, it is proposed to use directly the panchromatic images, i.e., the completely unprocessed, raw, grayscale satellite images. To this purpose, the MLP can be optimized using directly the grayscale intensities u(r) instead of the color c(r) defined previously. In particular, in (5) c(r)and — cGT(r) can be replaced by their grayscale intensities equivalent u(r) and uGT(r~).

[0074] Additionally, it is also proposed to do the pan-sharpening step simultaneously during the MLP optimization. To this purpose, the machine-learning step may further comprise alternating between optimizing on the grayscale panchromatic images as mentioned previously and optimizing on the color channels as done, for example, in (5). For this purpose, the MLP produces a color c(r) and not just an intensity u(r). In both cases, rays casted should be done so at the appropriate scale and u(r) can be derived from c(r) = (blue(r'),green(r'),red(r')) by u(r) = u1. bluefr) + u2. green(r) + u3. red(r) where u1, u2and u3are defined by the spectral characterization of the satellite so as to best approximate the panchromatic spectrum from the blue, green and red spectra.

[0075] On the use of depth priors to initialize the EO-NeRF geometry

[0076] The geometry of EO-NeRF can be initialized using an external elevation model of the geographic area. The external elevation model (DSM, 3D point cloud, mesh, etc.) may have been obtained by any other method, such as a previous lidar scan or a photogrammetric algorithm. The following explanation uses an external DSM for simplicity. The geometry of EO-NeRF can be initialized with such an external model based on 2 steps.

[0077] Step 1. Use the external DSM to initialize an altitude prior value for each image pixel involved in the optimization process.

[0078] For each input view, it is possible to obtain an equivalent initial depth image based on the external DSM. For that purpose, the raster cells of the external DSM can be seen as a cloud of 3D points with coordinates (longitude, latitude, altitude). These 3D points can be projected to the image space of each input satellite view using the projection function of the associated RPC camera model: (pixel_x, pixel_y, altitude) = RPC_projection((longitude, latitude, altitude)). Bilinear interpolation or any other 2D interpolation method can be applied afterwards to ensure that each image pixel is assigned an altitude value.

[0079] Step 2. Use the altitude priors to guide the geometry learning during the optimization process. During the optimization process, it is possible to render the observed depth of the ray emitted from each image pixel. The observed depth d(r) is obtained after Equation (4). The value of d(r) can be converted to an altitude value as explained below in relation to the DSM determination step. First, obtain the corresponding 3D point, (x, y, z) = o + d(r) d, and then convert the point (x, y, z) to geographic coordinates (east, north, altitude) using the scale and offset vectors employed to normalize all coordinates in the range [-1, 1]. The resulting altitude can be encouraged to be similar to the external DSM altitude by adding an auxiliary term in the EO-NeRF loss function:

[0080] L_depth = w_depth * DF (altitude(r), DSM(r)) where altitude(r) is the altitude value predicted by EO-NeRF for a ray r and DSM(r) is the prior altitude value for the image pixel intersected by the same ray r. The prior altitude value DSM(r) is initialized according to Step 1 before the beginning of the optimization process. The function DF in L_depth can be any distance function between scalar values, and it can be weighted by an arbitrary scalar weight w_depth > 0.

[0081] Thanks to gradient backpropagation, the auxiliary term L_depth will encourage the volume density σ of EO-NeRF to replicate the geometry given by the external DSM. Optionally, the auxiliary term L_depth can be dropped from the EO-NeRF loss function after a certain amount of optimization steps is completed.

[0082] Determining, by the computer system, a digital surface model of the ground area of interest

[0083] Once the learning phase of the neural radiance field model is achieved, the digital surface model of the ground area of interest can be determined based on volume densities predicted by the machine-learned neural radiance field model.

[0084] More particularly, such determination can comprise casting a set of rays which traverses the scene, predicting volume densities along each of the rays using the machine- learned neural radiance field model and determining the depth of each of the rays as in equation (4) based on the predicted volume densities.

[0085] An example embodiment of this determination is as follows.

[0086] The geometry learned by EO-NeRF is fully defined by the volume density o predicted by the neural network at each point (x, y, z) of the 3D space. The 3D space is bounded by a cube with a range of [-1, 1] on each axis. Normalized spatial coordinates (x, y, z), defined in the range [-1, 1], correspond to geographic coordinates in the format (east, north, altitude), based on the UTM system. EO-NeRF therefore recovers the geographic coordinates corresponding to each input point (x,y,z) as:

[0087] (east, north, alt) = (x, y, z) * (east_scale, north_scale, alt_scale) + (east_offset, north_offset, alt_offset)

[0088] The scale and offset value for each dimension x are computed from the 3D physical boundaries of the geographic area observed by the satellite images as follows: x_scale = (x_max - x_min) / 2 x_offset = x_min + x_scale

[0089] Using this representation of geographic coordinates, the digital surface model (DSM) of the area can be instantly extracted from the geometry learned by EO-NeRF in 3 steps.

[0090] First, a set of rays R is casted corresponding to NxN points of origin lying on the same plane and observing the 3D cube of the scene from the nadir. For example, this can be done by defining each ray r in R as r = o + t d : o(z) = 1, d = (0, 0, -1), t E [0, 2].

[0091] Then, for each ray r in R :

[0092] - The ray r is discretized into a set of 3D points, using EO-NeRF to predict the volume density a in each point of r, and retrieving the observed depth, d(r) as in Equation (4);

[0093] - The 3D point located at d(r) is computed as (x, y, z) = o + d(r) d

[0094] - Recover the geographic coordinates (east, north, altitude) of (x, y, z) using the scale and offset vectors.

[0095] Finally, the 3D point cloud of (east, north, altitude) coordinates resulting from the previous step is equivalent to the DSM of the geographic area. It can be converted to the DSM (2.5D format) by defining a georeferenced image raster with boundaries [east_min, east_max, north_min, north_max] and assigning to each pixel the (alt) coordinate of the 3D point whose (east, north) coordinates are closer to the pixel (east, north) coordinates.

[0096] Experiments

[0097] Data and evaluation metrics

[0098] The method EO-NeRF proposed by the present disclosure was tested on 7 areas of interest (AOI) covering 256 x 256 m each, using ~10-20 crops from multi-date WorldView- 3 images with a resolution of 30 cm / pixel. The images are taken from the public data of the 2019 IEEE GRSS Data Fusion Contest (DFC2019) [8] and 2016 IARPA Multi-View Stereo 3D Mapping Challenge (IARPA2016) [7]. The specific number of images and other details of each AOI are shown in Table 1 below. In both cases, panchromatic and multispectral products were used to create raw pansharpened images without any color correction. The DFC2019 data additionally provides true color RGB images over the AOIs with reduced dynamic range in 8-bit unsigned integer format. The elevation models obtained by the method of the present disclosure were assessed using lidar DSMs with a resolution of SO- BO cm / pixel. The PSNR (Peak Signal to Noise Ratio) of the image renderings and the altitude mean absolute error (MAE) with respect to the lidar data were used as evaluation metrics. Geometrically corrected RPC camera models were used in all experiments unless raw RPCs is mentioned.

[0099] Table 1 Number of input images used for each area, altitude bounds of the scene and approximate latitude and longitude..

[0100] Implementation details

[0101] A batch size of 1024 rays, Adam optimizer and an initial learning rate of 5e-4were used. The first optimization steps are performed using the standard MSE loss (5). The complexity of the optimization problem is gradually increased: first by plugging β and adopting the full loss (10) and then by adding the internal bundle adjustment offsets (11). Both camera rays r and solar rays rsunare discretized into N = 128 evenly distributed samples. These configuration parameters may be adapted for other areas of interest. DFC2019 areas - Results and discussion

[0102] Three categories of experiments are proposed, corresponding to the rows of Table 2 discussed below. EO-NeRF achieves the lowest MAE on average, as indicated in bold in Table 2 . Due to the difficulty of computing PSNR for unseen views (e.g. change of ambient color or transients), it is computed using the input views and thus might not improve over Sat-NeRF. The latter has more freedom to overfit the input views since shadows are not forced to be consistent with geometry.

[0103] Category 1. The state-of-the-art Sat-NeRF model was tested using N = 64 points per ray (row 1), as in [3]. Using UTM-based point coordinates, similar or better altitude MAE with respect to the results originally reported in [3] were obtained, while using half of units in the MLP layers (h = 256 instead of h = 512). Increasing the number of points per ray to N = 128 (row 2) does not necessarily improve the performance. It is noted that that Sat-NeRF does not adapt to non-radiometrically normalized inputs, as the MAE increases significantly using raw pansharpened data. Visual inspection of DSMs shows that Sat-NeRF is more prone to surface irregularities than EO-NeRF. The geometrically consistent shadow model of EO-NeRF is a natural solution to prevent undesired holes or blobs in the geometry that would result in unrealistic shadows.

[0104] Category 2. EO-NeRF was tested (row 3) using the same externally bundle adjusted RPC cameras as in the Sat-NeRF experiments. The altitude MAE improves dramatically by an average of more than half a meter. The improvement is especially noticeable for raw pansharpened images. Using the unrefined camera models from the image metadata significantly reduces performance in both PSNR and MAE (row 4), unless the EO-NeRF internal bundle adjustment is active (row 5). Externally corrected RPCs provide the most consistent geometry by a small margin, which is attributed to the fact that the proposed offset correction is highly dependent on the affine approximation of the RPC functions.

[0105] Category 3. The open-source stereo pipeline S2P was used to reconstruct 10 manually selected image pairs. All pairs are reconstructed using bundle adjusted RPCs and they are merged into a single surface model by taking the median altitude along the z- axis. For disparity estimation, the classic MGM (More Global Matching) algorithm (row 6) and the Pyramid Stereo Matching (PSM) network (row 7) were used. MGM is clearly outperformed by EO-NeRF provides better MAE in certain areas using pansharpened inputs, but visual inspection reveals missing structures in the DSMs that only EO-NeRF manages to capture. For instance, certain tall buildings are lost because they fall outside the maximum disparity range used to train the PSM network.

[0106] Table 2

[0107] IARPA2016 areas - Results and discussion The same categories of experiments were repeated. Only raw pansharpened images are available in this case. The numerical results in the different rows of Table 3 are discussed below.

[0108] Category 1. Sat-NeRF is tested using UTM-based point coordinates and N = 128 points per ray (row 1). Consistent with the DFC2019 results, the model does not fit non- radiometrically normalized inputs and the altitude MAE is above 2 m in all areas.

[0109] Category 2. EO-NeRF is tested using externally bundle adjusted and internally bundle adjusted RPC camera models (row 2 and 4 respectively). When the input RPCs are consistent, because of the external or the internal bundle adjustment, EO-NeRF outperforms Sat-NeRF by about one meter of altitude in average. This advantage is lost if the unrefined camera models are used instead (row 3).

[0110] Category 3. As for Category 3 above, the stereo pipeline S2P is used to reconstruct 10 manually selected image pairs, with MGM (row 5) and PSM for dense matching (row 6). Again, EO-NeRF achieves better overall altitude MAE with respect to MGM and PSM.

[0111] Conclusion

[0112] The present disclosure presents EO-NeRF, a variant of NeRF for Earth observation adapted to multi-date satellite images. The disclosed method achieves state-of-the-art novel view synthesis and digital surface modeling from this kind of data. Using input views with resolution of 30 cm / pixel, EO-NeRF elevation models reveal scene geometry with a level of detail comparable to intrusive aerial acquisitions. EO-NeRF improves on previous concurrent NeRF variants mainly thanks to a geometrically consistent rendering of shadows that does not modify the loss function and the addition of network parameters to handle raw satellite images.

[0113] The geometrically consistent shadows provide hints that permit to refine the geometry, which in turn refines the shadows. This approach also ensures that the model generalizes for novel view synthesis using solar directions completely different from those of the input images.

[0114] In addition, a variety of strategies can be implemented to use unprocessed satellite images. These include the use of UTM coordinates to handle georeferenced data in a more appropriate way and the addition of network parameters to address inaccuracies in the camera models and color biases in the images during the optimization process.

[0115] The fine-scale details recovered by EO-NeRF are lost in modern multi-view stereo pipelines for satellite 3D reconstruction, which are not adapted for multi-date inputs. Classic matching algorithms are usually subject to strong regularization, whereas deep neural networks may have generalization problems.

[0116] The invention is not limited to the method disclosed above, but also relates to a system comprising a processor configured to perform said method, to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out said method and to a non-transitory computer-readable medium storing a program including instructions that, when executed by a processor of a computing device, cause the computing device to carry out said method.

[0117] References

[0118] [1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Computer Vision - ECCV 2020, pages 405-421, 2020.

[0119] [2] Dawa Derksen and Dario Izzo. Shadow neural radiance fields for multi-view satellite photogrammetry. In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1152-1161, 2021. [3] Roger Mari, Gabriele Facciolo, and Thibaud Ehret. Sat-NeRF: Learning multi-view satellite photogrammetry with transient objects and shadow modeling using rpc cameras. In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1310-1320, 2022.

[0120] [4] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7206-7215, 2021.

[0121] [5] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. "Volume rendering of neural implicit surfaces." Advances in Neural Information Processing Systems 34 (2021): 4805-4815.

[0122] [6] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. "NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction." Advances in Neural Information Processing Systems 34 (2021): 27171- 27183.

[0123] [7] Marc Bosch, Zachary Kurtz, Shea Hagstrom, and Myron Brown. A multiple view stereo benchmark for satellite imagery. In 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pages 1-9, 2016.

[0124] [8] Marc Bosch, Kevin Foster, Gordon Christie, Sean Wang, Gregory D Hager, and Myron Brown. Semantic stereo for incidental satellite images. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1524- 1532, 2019.

Claims

CLAIMS1. A computer-implemented method, comprising: obtaining (OBT) multi-date images of a ground area of interest captured by a sensor onboard an Earth observation satellite; performing (ML-NeRF) machine-learning of a neural radiance field model using the multi-date images, wherein the neural radiance field model is configured to predict a volume density ( σ ) and an albedo color (cα) at a 3D location (x); and determining (DSM) a digital surface model of the ground area of interest based on volume densities predicted by the machine-learned neural radiance field model; wherein performing machine-learning of the neural radiance field model based on the multi-date images comprises: for each camera ray (r) of a plurality of camera rays each intersecting a surface point (xs) of the ground area of interest: o determining a transmittance of a solar ray (rsun) which intersects the surface point (xs) based on volume densities predicted by the neural radiance field model at 3D locations along the solar ray; o determining a radiance of the camera ray by computing the product of an albedo color (cα) predicted by the neural radiance field model at the surface point and an irradiance function weighted by the determined transmittance; evaluating a loss function that compares the radiances determined for each camera ray with ground truth radiances derived from the multi-date images; modifying the neural radiance field model based on the evaluated loss function.

2. The method of claim 1, wherein determining a radiance of the camera ray further comprises applying an image-specific affine transformation to the computed product.

3. The method of claim 2, wherein the image-specific affine transformation is defined by parameters ( Aj, bj) which are predicted by the neural radiance field model as imagespecific embeddings.

4. The method of any one of claims 1 to 3, wherein the irradiance function weighted by the determined transmittance is = s(r) + (1 — s(r))α, with s(r) the determined transmittance and a an ambient color predicted by an auxiliary portion (MLP 4) of the neural radiance field model based on a solar direction (dsun).

5. The method of any one of claims 1 to 4, wherein the neural radiance field model comprises: a base portion (MLP 1) configured to predict the volume density (σ) and output a latent representation (z), based on the 3D location (x); a static content portion (MLP 2) configured to predict the albedo color (cα) based on the latent representation (z) output by the base portion; a transient content portion (MLP 3) configured to predict one or more coefficients (β , τ) based on the latent representation (z) output by the base portion and an image-dependent embedding (tj).

6. The method of claim 5, wherein the one or one or more coefficients comprise a transient scalar (τ) used to modulate the transmittance of the solar ray.

7. The method of claim 6, wherein the one or one or more coefficients comprise an uncertainty coefficient (β ) used to modulate the loss function.

8. The method of any one of claims 1 to 7 , wherein an origin and a direction of a camera ray is determined using a localization function of a RPC, Rational Polynomial Coefficients, camera model.

9. The method of claim 8, further comprising adjusting the RPC camera model based on one or more transformation parameters which are predicted by the neural radiance field model as image-specific embeddings.

10. The method of any one of claims 1 to 9, wherein the multi-date images are panchromatic images.

11. The method of claim 10, wherein performing machine learning further comprises pan-sharpening the panchromatic images.

12. The method of claim 1, wherein the multi-date images are radar images.

13. The method of claim 1, wherein the multi-date images are optical images.

14. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 13.

15. A system, comprising a processor configured to perform the method of any one of claims 1 to 13.

Citation Information

Cited By

  • Aerospace three-dimensional scene dynamic change detection method and device, medium and product

    CN122200415A