Self-supervised spatio-temporal field reconstruction method based on multi-mode spatial integral coding and spectral band optimization, storage medium and equipment

Abstract

The application discloses a self-supervision space-time field reconstruction method based on multi-mode space integral coding and spectrum optimization, a storage medium and equipment, and belongs to the technical field of remote sensing three-dimensional reconstruction. In order to solve the problem that the three-dimensional reconstruction effect of a time-varying scene based on a hyper-generalized stereoscopic image pair is affected by the difference in spatial and spectral resolution, light rays are generated through each image of the hyper-generalized stereoscopic image pair and a space-time field imaging geometric model of the hyper-generalized stereoscopic image pair, basic self-supervision information is given to the light rays based on image pixels, light ray samples are formed, the pixels with different spatial resolutions are bound with actual constraint space ranges through a space integral strategy of the light rays, a multi-channel joint self-supervision strategy is optimized and designed, support conditions are provided for jointly establishing geometric constraints based on light ray samples with different wave band numbers and numerical accuracies, a space-time field function is supervised, and finally, space-time field reconstruction is realized based on the space-time field function.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing 3D reconstruction technology, specifically relating to a self-supervised spatiotemporal field reconstruction method, storage medium, and device. Background Technology

[0002] Remote sensing 3D reconstruction technology is a major approach to acquiring large-scale 3D scene information. Compared to traditional remote sensing 3D reconstruction, time-varying scene 3D reconstruction is not a 3D reconstruction for a specific point in time, but rather a reconstruction of the scene's different geometric shapes at different times and the geometric changes between different times. It is an important means of acquiring time-varying scene 3D information.

[0003] Super-generalized stereo image pairs: A combination of two or more optical remote sensing images acquired in any manner and from different viewing angles, covering the same scene, is defined as a "super-generalized stereo image pair". The composition of a super-generalized stereo image pair is free; it can be a "standard stereo image pair" or a "generalized stereo image pair", or it can be a general frontal or oblique remote sensing image. The number of images is not fixed and can be incrementally accumulated as new images are captured, forming an input increment. They can be acquired by different payload platforms at different times, and the multiple images have non-fixed differences in viewing angle and resolution. The intersection angle, datum-to-height ratio, and overlap between pairs of images do not necessarily meet the conditions for forming a "standard stereo image pair" or a "generalized stereo image pair". Once there are differences in viewing angles among multiple optical remote sensing images, they can form geometric constraints on the three-dimensional morphology of the scene. Therefore, theoretically, three-dimensional scene reconstruction can be achieved based on super-generalized stereo image pairs. Compared to standard or generalized stereo image pairs, the main advantage of "ultra-generalized stereo image pairs" lies in the high frequency and multi-view acquisition of image data. They possess the necessary data conditions to realize the 3D reconstruction of time-varying scenes from "static" to "temporal" perspectives, and are more in line with the needs of real-world 3D.

[0004] However, time-varying scene 3D reconstruction based on "ultra-generalized stereo image pairs" faces severe challenges from many aspects, such as the complexity of scene spatiotemporal representation, the imaging differences of multi-source images, the imaging differences of time-varying scenes, and the complexity of image increment methods. Furthermore, time-varying scene 3D reconstruction based on ultra-generalized stereo image pairs faces even more complex problems formed by the coupling of multiple challenges, which is a technical bottleneck that is currently difficult to overcome.

[0005] The imaging differences among multi-source images are mainly due to the differences in imaging geometry, spatial resolution, spectral resolution, and radiometric resolution between spaceborne and airborne images from different sources, which greatly affects the reliability of 3D reconstruction based on ultra-generalized stereo image pairs.

[0006] First, the images have different imaging geometric models:

[0007] For imaging geometry models of spaceborne images, the Rational Function Model (RFM) has been widely used as a replacement for the rigorous imaging geometry model in most application scenarios. It describes the mapping relationship between the row and column coordinates of image pixels and the three-dimensional coordinates of latitude, longitude, and altitude using a set of Rational Polynomial Coefficients (RPCs). It has many advantages, including high computational efficiency, confidentiality of satellite and sensor geometric parameters, and no need for iterative coordinate inversion. Currently, sensor calibration products released by high-resolution satellites both domestically and internationally all include RPC-related files. However, the generation methods and accuracy of RPCs vary among different satellites, leading to uncertain coordinate mapping deviations in the imaging geometry model. This makes it difficult to directly establish accurate joint geometric constraints based on multiple images from different satellites, resulting in low accuracy of scene 3D reconstruction or even large overall morphological deviations. For the imaging geometry model of airborne images, the imaging geometry model of airborne images is generally considered to satisfy the perspective projection model, also known as the pinhole model. On the one hand, the manufacturing processes of different lenses are different, and the accuracy of the imaging geometry model has inherent differences. On the other hand, the parameters of the imaging geometry model of airborne images are usually unknown, and the approximate imaging geometry model estimated by different types of methods has different degrees of error.

[0008] It is evident that for ultra-generalized stereo image pairs, regardless of whether the images are from spaceborne or airborne sources, there are certain differences in the imaging geometric models of the sensors, making it difficult to guarantee the consistency of geometric constraints in three-dimensional space. On one hand, the parameter representations of the imaging geometric models are inconsistent; for example, the parameter representations and the number of parameters in the RFM model and the pinhole model are different. On the other hand, the accuracy of the imaging geometric models of different images is inconsistent, leading to spatial mapping deviations. Therefore, to effectively unite ultra-generalized stereo image pairs to establish geometric constraints in the spatiotemporal field and fit the scene's spatiotemporal field function, it is necessary to reconstruct the image's imaging geometric model in the spatiotemporal field coordinate system, standardize its parameter representation, and form a spatiotemporal field imaging geometric model. Simultaneously, the model parameters should be further optimized based on spatial geometric constraint relationships to improve spatial mapping accuracy.

[0009] Second, the images have differences in spatial resolution:

[0010] In the early definition of generalized stereo image pairs in the field of remote sensing, differences in spatial resolution were a key concern. Research primarily focused on spaceborne images, employing a technical framework based on traditional stereo matching methods. Common approaches included using sampling or super-resolution strategies to achieve uniform resolution across multiple images, or implementing spatial resolution-oriented weighting strategies before stereo matching. A typical example is the work of Spanish scholar Manuel... Aguilar collected multiple sets of GeoEye-1 and WorldView-2 satellite data and conducted research on the 3D coordinate calculation of generalized stereo image pairs. Considering image characteristics such as radiometry and illumination, it comprehensively analyzed the relationship between 3D reconstruction accuracy and various factors such as ground control points, intersection angles between image pairs, datum-to-height ratio, and resolution. Professor Gu Yanfeng of Harbin Institute of Technology proposed adding a scale difference factor to the traditional calculation model to reduce the impact of resolution differences on the accuracy of 3D coordinate calculation. However, most related studies have avoided the problem of dense stereo matching, focusing only on the accuracy of calculation for a few manually selected matching points, and thus failing to achieve 3D reconstruction of the complete scene.

[0011] The actual spatial range covered by pixels in images with different spatial resolutions varies. When there are differences in image spatial resolution, light rays from low-resolution images actually cover a larger spatial range, while light rays from high-resolution images actually cover a smaller spatial range. The NeRF series of methods models a one-to-one correspondence between light samples and image pixels. However, most NeRF methods, focusing more on natural scene images, do not pay attention to distinguishing the actual spatial scale corresponding to pixels, treating all pixels with a single scale. This leads to inaccuracies in the actual spatial range corresponding to light rays acquired from images with different spatial resolutions, resulting in errors in the spatial information of the generated light samples. MipNeRF improves upon the aliasing problem caused by image spatial resolution by mapping pixels to spatial frustums instead of single spatial locations. However, its processing method is limited to near-field pinhole models and is not suitable for remote sensing images. For ultra-generalized stereo image pairs, because the actual spatial scales corresponding to pixels in images with different spatial resolutions are different, the single-scale processing of existing light sample generation methods will cause varying degrees of geometric constraint errors; while the spatial frustum-based processing method is difficult to uniformly process images with different observation distances and imaging methods in the remote sensing field. Therefore, there is an urgent need for a ray sample generation strategy that adapts to spatial resolution.

[0012] Third, the images exhibit differences in spectral and radiometric resolution:

[0013] Research in remote sensing addressing spectral resolution differences primarily focuses on image super-resolution and fusion. According to research, there is currently no research on joint 3D reconstruction based on visible light, multispectral, and hyperspectral images. Research on 3D reconstruction based on hyperspectral images currently employs two main approaches: the first is the combined LiDAR approach. (Harbin Institute of Technology)

[23] Professor Gu Yanfeng's team fused hyperspectral images with LiDAR point clouds to generate 3D hyperspectral point clouds, achieving 3D scene reconstruction. The second method directly utilizes multiple hyperspectral images. Ali Zia et al. used VisualSFM software to create different 3D models of objects at various wavelengths and merge them, achieving structural complementarity. Ali Can Karaca et al. from Kocaeli University in Turkey used two parallel hyperspectral cameras and proposed the SegmentSpectra algorithm, achieving reconstruction accuracy similar to LiDAR point clouds. However, the aforementioned 3D reconstruction methods based on multiple spectral band images do not address situations where spectral resolution differs. In reality, the images constituting a super-generalized stereo pair may be panchromatic, RGB, multispectral, hyperspectral, etc., and their imaging band ranges and numbers may differ. This difference causes feature differences between images, leading to unsatisfactory results from traditional stereo matching. Furthermore, since NeRF series methods obtain self-supervised information by analyzing the pixel values ​​of the image corresponding to each ray, the difference in the number of spectral bands results in dimensional differences in the self-supervised information of rays generated from images with different band numbers, making it difficult to construct joint geometric constraints. Although common remote sensing images exist in single-band, 3-band, 4-band, 8-band, and even hundreds of band configurations, the number of bands is not the key factor determining geometric constraint capability. The observation perspective is more important for geometric constraints. Involving too many bands does not necessarily improve geometric constraint capability and may even waste computational resources. Therefore, adaptive self-supervised information dimensionality processing is a prerequisite for establishing effective geometric constraints based on ray self-supervised learning. As shown in Table 1, different bands have different application scenarios. Therefore, for 3D reconstruction of specific scenarios, appropriate band selection can be made to limit the impact of dimensionality differences.

[0014] Furthermore, the radiometric resolution of remote sensing images determines the precision bit width of each pixel value; 10-bit to 12-bit remote sensing images are common. Neural implicit representation methods based on NeRF construct a loss function by accumulating the rendering error of each ray. Differences in radiometric resolution can cause variations in the precision of the self-supervised information of rays, leading to a loss bias in the iterative process for pixel values ​​with a certain bit width. This is detrimental to the joint geometric constraints between rays and even more so to the accurate fitting of the spatiotemporal field function. Therefore, it is necessary to normalize the bit width of the input image to avoid this loss bias.

[0015] Table 1. Characteristics of common wavebands in the field of remote sensing technology.

[0016]

[0017] In summary, the differences in spectral and radiometric resolution of images result in inconsistent dimensionality and precision of pixel values ​​at the same spatial location across different image types. This makes it difficult to backpropagate errors through a common neural network structure and easily leads to biases in joint self-supervised learning. Therefore, it is essential to research self-supervised learning strategies that adapt to differences in spectral and radiometric resolution, or to introduce other auxiliary geometric constraints. Summary of the Invention

[0018] This invention aims to address the problem that the difference in spatial spectral resolution affects the 3D reconstruction effect of time-varying scenes based on ultra-generalized stereo image pairs.

[0019] A self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band optimization is proposed. The spatiotemporal field is characterized using the time-varying radiation function (TVSRF) and the time-varying symbol distance function (TVSDF), and spatiotemporal field reconstruction is achieved through optimized TVSRF and TVSDF. The optimization process of TVSRF and TVSDF includes:

[0020] A light ray emitted by the camera and passing through image pixels is constructed based on the spatiotemporal field imaging geometric model; an integral space V of the spatial sampling points is constructed based on the spatial sampling point i on the light ray. i ;

[0021] Position encoding is performed on the coordinates p = (x, y, z) in space to obtain the encoding result pos_enc(p); the encoding result is integrated in the integration space V to obtain the multi-mode spatial integral encoding result MSI(p);

[0022] Convert the pixel values ​​I(i,j) in the image to m-bit precision to obtain I. * (i,j), then transform the result I * Round (i,j) to the nearest integer and convert it to an m-bit binary representation; then use the resulting I... * (i,j) divided by 2 m We obtain I'(i,j);

[0023] For a set of bands out ={B,G,R,IR,Pan}, and perform volume rendering along this ray based on the output of the time-varying radiometric function (TVSRF), including the following steps:

[0024] Given a pixel pix and the ray Ray = {r(d) = o + dv, d ≥ 0} passing through that pixel, and the depth range [d] of the reconstructed target scene. n , t f When ], first in [d n d f Sampling is performed along the inner edge of the ray, resulting in a set of sampling points P = {pi |p i =r(d i ),i=1,2…,N};where v is the unit direction vector of the light ray, d is the depth of the light ray starting from the camera origin o, and N represents the number of sampling points;

[0025] Then, the volume rendering result corresponding to the pixels through which the light passes is obtained according to the following formula:

[0026]

[0027] w(p i )=T(p i )a(p i ),

[0028] δ(p i )=d i+1 -d i

[0029] σ(p i )=κ·σ'(-TVSDF(MSI(p i ),t i )),

[0030] Wherein, δ(p) i ) represents the depth interval between adjacent sampling points, a(p i ) is along the length of the ray segment r(d) i+1 )-r(d i The opacity of T(p) i () represents the light ray that travels from the origin o to the spatial point p. i Cumulative transparency at time, w(p) i ) is at point p in space i The corresponding volume rendering integral weight at the location; σ(p) i ) for t i At any point p in space i The volume density at a given location; k, λ are learnable parameters, while σ'(x) is the cumulative distribution function based on the Laplace distribution, with a mean of 0 and a scale parameter of λ; TVSRF employs a neural network model, which is based on the input MSI(p i ) and t i get These are the radiation values ​​for the corresponding bands, TVSRF(MSI(p) i ),t i ) for t i At any point p in space iThe output of the time-varying radiation function; the time-varying symbol distance function (TVSDF) adopts a neural network model, which is based on the input MSI(p) i ) and t i The sign distance value is obtained. The sign distance value refers to the minimum distance between the coordinates of a spatial point p and the coordinates of a point on the surface of the terrain. TVSDF(MSI(p) i ),t i ) for t i At any point p in space i The output of the time-varying symbol distance function;

[0031] Assume the input image I corresponding to the pixel i The included band set is Band in The channel optimization mechanism will also construct a 5-dimensional vector R corresponding to that pixel. in =(r B ,r G ,r R ,r IR ,r Pan ), the value of one dimension r k Let I'(i,j) be the value corresponding to the corresponding band, k∈{B,G,R,IR,Pan}; r k Satisfy rule: r k =0, if

[0032] By R in R out Constructing a self-supervised spectral loss at the pixel level N = ||R in ||0, where ||·||1 and ||·||0 correspond to the 1-norm and 0-norm, respectively;

[0033] The optimization of the time-varying radiation function (TVSRF) and the time-varying symbol distance function (TVSDF) is achieved based on spectral loss.

[0034] Furthermore, the integration space V cube V cone Let I represent the integral spaces of the corresponding cuboid and cone shapes, respectively; i Ψ represents the image corresponding to spatial sampling point i. sat Ψ air These represent satellite and airborne image sets, respectively.

[0035] Furthermore, the position encoding method for the coordinates p = (x, y, z) in the integration space is as follows;

[0036] pos_enc(p)=[a1cos(2πω1p),a1sin(2πω1p),…,a n cos(2πω n p),a n sin(2πω n p)](2)

[0037] Wherein, ω1...ω n For frequency coefficients, a1...a n For different frequencies of coding weights, pos_enc(p) represents the result of position coding.

[0038] Furthermore, the pixel values ​​I(i,j) in the image are converted to m-bit precision to obtain I. * The formula for (i,j) is as follows:

[0039]

[0040] Where I(i,j) represents the i-th row and j-th column of the image, and n represents the original quantization bit depth of the image.

[0041] Furthermore, position encoding is performed on the coordinates p = (x, y, z) in space to obtain the encoding result pos_enc(p); the encoding result is integrated in the integration space V to obtain the multi-mode spatial integral encoding result MSI(p):

[0042]

[0043] Where V represents the integration space corresponding to p.

[0044] Furthermore, the aforementioned In σ'(x), x represents the input to the cumulative distribution function.

[0045] A computer storage medium storing at least one instruction, which is loaded and executed by a processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection.

[0046] A self-supervised spatiotemporal field reconstruction device based on multi-mode spatial integral coding and spectral band selection is disclosed. The device includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection.

[0047] Beneficial effects:

[0048] This invention generates rays from each image of a super-generalized stereo image pair and its spatiotemporal field imaging geometric model. Based on image pixels, it assigns basic self-supervised information to the rays, forming ray samples. Through a spatial integration strategy for the rays, pixels with different spatial resolutions are bound to their actual constraint space ranges. Simultaneously, it optimizes the design of a multi-channel joint self-supervised strategy, providing support for establishing geometric constraints based on ray samples with different band numbers and numerical precisions. This leads to the development of a supervisory spatiotemporal field function, ultimately achieving spatiotemporal field reconstruction. This invention effectively addresses the impact of differences in spatial-spectral-radial resolution on the 3D reconstruction results of time-varying scenes based on super-generalized stereo image pairs, thus improving the 3D reconstruction effect. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating the resolution differences between multiple image sources.

[0050] Figure 2 This is a flowchart for generating self-supervised ray samples in the spatiotemporal field.

[0051] Figure 3 This is a flowchart of the self-supervised ray sample generation process based on multi-mode spatial integral coding and channel optimization. Detailed Implementation

[0052] To address the issue of spatial-spectral-radial resolution differences in multi-source images, this invention primarily aims to solve the following specific problems:

[0053] ① The actual three-dimensional spatial range corresponding to light samples in images with different spatial resolutions is inconsistent, such as... Figure 1 As shown in (a), this causes a spatial positional deviation in the joint geometric constraints.

[0054] ② The dimensions and precision of image values ​​differ between images with different spectral and radiometric resolutions, such as... Figure 1 As shown in (b), the inconsistent form and significant differences in accuracy of the self-supervised information of the light samples make it difficult to effectively combine self-supervised learning.

[0055] For ultra-generalized stereo image pairs with differences in spatial, spectral, and radiometric resolution, this invention proposes a self-supervised ray sample generation method for supervised spatiotemporal field function fitting. For example... Figure 2As shown, this invention generates rays from each image of a super-generalized stereo image pair and its spatiotemporal field imaging geometric model. Based on image pixels, it assigns basic self-supervised information to the rays, forming ray samples. It delves into the spatial integration strategy of rays, binding pixels with different spatial resolutions to their actual constraint space range. An optimized multi-channel joint self-supervised strategy is designed to provide supporting conditions for jointly establishing geometric constraints based on ray samples with different band numbers and numerical precisions, providing the necessary ray sample set for fitting the spatiotemporal field function. Thus, a method for supervising the spatiotemporal field function is developed, ultimately achieving spatiotemporal field reconstruction. The invention will be further described below with reference to specific implementation methods. Specific implementation method one:

[0057] This implementation presents a self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection. It generates ray samples based on the pixel coordinates of the image and the spatiotemporal field imaging geometric model, and assigns self-supervised information to the ray samples based on pixel values. A multi-mode spatial integral coding strategy is proposed, which optimizes spatial information processing by combining the imaging characteristics of spaceborne and airborne images, reducing geometric constraint deviations caused by differences in spatial resolution. A self-supervised learning strategy based on channel selection is also proposed, which fully utilizes ray samples generated from images with different spectral and radiometric resolutions to jointly establish geometric constraint rules.

[0058] like Figure 3 As shown in this embodiment, a self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection mainly includes a multi-mode spatial integral coding strategy and a self-supervised learning strategy based on spectral band selection.

[0059] The multi-mode spatial integral coding strategy is as follows:

[0060] First, based on the ultra-generalized stereo image pairs and their spatiotemporal field imaging geometric model, a corresponding ray can be generated for any pixel in each image. Then, multi-mode spatial integral coding is performed. The purpose of spatial integration is to more accurately encode the three-dimensional information of spatial points. By constructing an integral space that conforms to the characteristics of image imaging, the traditional single-scale position coding is extended to scale-adaptive spatial integral coding, thereby expanding the neural network's ability to perceive spatial scale and reducing the geometric constraint deviation caused by differences in spatial resolution.

[0061] like Figure 3 The upper part shows the following steps:

[0062] S1. Construct the light rays emitted by the camera and passing through the image pixels based on the spatiotemporal field imaging geometric model.

[0063] The aim is to mimic the camera's emission process during imaging. Satellite imaging models often rely on RFM imaging models, while pinhole camera models approximate RFM imaging models. Therefore, in some embodiments, the pinhole model can be used as a standardized representation of imaging geometric model parameters, approximating the satellite image imaging geometric model as the pinhole model, in order to maintain formal consistency with the airborne imaging geometric model. Specifically, the spatiotemporal field imaging geometric model P is a projection matrix for perspective imaging, which is a 4x4 matrix. Given a coordinate point p = (x, y, z) in three-dimensional space, P can map p to a pixel coordinate pix = (u′, v′) in image I.

[0064] P[x,y,z,1] T =[ku′,kv′,k,1] T

[0065] Given a pixel coordinate (u,v), through matrix P -1 It can then be mapped to three-dimensional space:

[0066] P -1 [u′,v′,1,1]=[x′,y′,z′,1]

[0067] Therefore, in some embodiments, the above formula is used to construct a spatiotemporal field imaging geometric model, where image I = {I(u,v)|u∈1…m,v∈1…n}, indicating that image I is an m-row n-column matrix.

[0068] S2, based on the input image I i The imaging payload platform and spatial resolution, and the multi-mode adaptive construction of the integral space V of the spatial sampling points. i .

[0069] In step S1, the light rays have been obtained, but relying on a single light ray form cannot reflect the difference in spatial resolution between spaceborne and airborne images in the subsequent rendering process. Therefore, different spatial integration construction modes are adopted for the two, and the light rays are expanded into "cylinders" with different shapes by constructing an integration space, thereby enabling the light ray samples generated from images with different spatial resolutions to adapt to the spatial scale.

[0070] In terms of the shape of the integration space, for satellite images, the satellite observation point is far away from the observed ground surface and the field of view is narrow, so a cuboid shape of integration space is adopted; while for airborne images, the observation point is much closer to the observed ground object and the field of view is also larger, so a cone shape of integration space is adopted.

[0071] Integral Space:

[0072]

[0073] Where Vi represents the integration space corresponding to spatial sampling point i, V cube V cone Let I represent the integral spaces of the corresponding cuboid and cone shapes, respectively; i Ψ represents the image corresponding to spatial sampling point i. sat Ψ air These represent satellite and airborne image sets, respectively.

[0074] Regarding the size of the integration space, it is determined based on the input image type, the corresponding spatial resolution of the image, and the interval between adjacent sampling points. For high spatial resolution images, the corresponding integration space V should be smaller, while for low spatial resolution images, the corresponding integration space V should be larger, in order to achieve uniformity in the actual three-dimensional spatial range of the imaging.

[0075] Then, position encoding is performed on the input coordinates p = (x, y, z). The purpose of position encoding is to improve the neural network's ability to distinguish between high and low frequency signals. The encoding process is shown in equation (2).

[0076] pos_enc(p)=[a1cos(2πω1p),a1sin(2πω1p),…,a n cos(2πω n p),a n sin(2πω n p)](2)

[0077] Wherein, ω1...ω n For frequency coefficients, a1...a n For different frequencies of coding weights, pos_enc(p) represents the result of position coding.

[0078] Finally, the encoding result is integrated in the integration space V, that is, the positional encoding result is integrated in the space of the corresponding mode according to the load plateau and spatial resolution of the input image, thus obtaining the Multimodal Spatial Integration (MSI) encoding result MSI(p):

[0079]

[0080] Where V represents the integration space corresponding to p.

[0081] In MSI processing, the spatial resolution during rendering determines the actual spatial size corresponding to one rendered pixel. Multimodal spatial integral coding can adjust the encoding results to different degrees according to the actual spatial range covered by the spatial sampling points, thereby giving the spatiotemporal field function the ability to perceive spatial scale.

[0082] The self-supervised learning strategy based on channel selection:

[0083] A channel-optimized self-supervised learning strategy determines the number of channels and numerical precision of the rendered output image. To achieve self-supervised learning based on images with different spectral and radiometric resolutions, it is necessary to unify the dimension and precision of pixel values.

[0084] First, to address the issue of radiometric resolution differences, the high-quantization precision map can be uniformly converted to m-bit numerical precision using the data scaling method, as shown in equation (4):

[0085]

[0086] Where I(i,j) represents the pixel value in the i-th row and j-th column of the image, and n represents the original quantization bit depth of the image.

[0087] For the spectral values ​​of each band, I is obtained by converting them according to the above formula. * (i,j), then transform the result I * (i,j) is rounded down and converted to an m-bit binary representation (the specific value of m can be freely set according to the specific input image) to achieve radiometric resolution normalization. Then, the I obtained from formula (4) is... * (i,j) divided by 2 m We obtain I'(i,j) to normalize the range of band values, thus avoiding gradient explosion during training.

[0088] Secondly, regarding the issue of spectral resolution differences, as mentioned earlier, the observation perspective of an image is crucial for constructing spatial geometric constraints. However, the number of image bands is not a decisive factor in geometric constraint capability. Including too many bands in the supervision process does not necessarily improve geometric constraint capability and may even waste computational resources. The channel optimization mechanism proposed in this invention can appropriately select bands for 3D reconstruction of specific scenes, limiting the impact of dimensional differences. Based on the spectral range characteristics of typical Chinese high-resolution satellite sensors listed in Table 1, single-band panchromatic (Pan) and multispectral images covering red (R), green (G), blue (B), and near-infrared (IR) are the most common and have the widest application scenarios. Airborne images also cover these bands. Therefore, it is assumed that all images will be selected for self-supervised learning using the following process on these five channels. Figure 3 As shown, after the radiometric resolution of the pixels is normalized, the image channels are first optimized by selecting the inherent bands of the image based on the five preset channels. Then, each image undergoes self-supervised learning based on pixel consistency loss only on its own inherent channels corresponding to the preset channels.

[0089] Specifically, firstly, a band set is defined.out ={B,G,R,IR,Pan}, and then set the dimension of the TVSRF output radiation value to 5 dimensions, and stipulate that the meaning of each dimension corresponds one-to-one with the meaning of each element in the band set, that is... Given a pixel and a ray passing through it, volume rendering along the ray based on the TVSRF output yields a 5-dimensional vector output corresponding to the input pixel. The process of volume rendering along the ray based on the TVSRF output includes:

[0090] Given a pixel pix and the ray Ray = {r(d) = o + dv, d ≥ 0} passing through that pixel, and the depth range [d] of the reconstructed target scene. n , t f When ], first in [d n d f Sampling is performed along the inner edge of the ray, resulting in a set of sampling points P = {p i |p i =r(d i ),i=1,2…,N}.

[0091] Where v is the unit direction vector of the light ray, d is the depth of the light ray starting from the camera origin o, and N represents the number of sampling points.

[0092] Then, the volume rendering result corresponding to the pixels through which the light passes is obtained according to the following formula:

[0093]

[0094] w(p i )=T(p i )a(p i )

[0095]

[0096] δ(p i )=d i+1 -d i

[0097] Wherein, δ(p) i ) represents the depth interval between adjacent sampling points, a(p i ) is along the length of the ray segment r(d) i+1 )-r(d i The opacity of T(p) i () represents the light ray that travels from the origin o to the spatial point p. i Cumulative transparency at time, w(p) i ) is at point p in space iThe corresponding volume rendering integral weights at the location; TVSRF represents the time-varying radiation function, which is determined by the input MSI(p i ) and t i Radiation intensity can be obtained using a neural network model, such as TVSRF(MSI(p)). i ),t i ) for t i At any point p in space i The output of the time-varying radiation function at point P, i.e., the radiation intensity; σ'(p i ) for t i At any point p in space i The volume density at that location.

[0098] σ(p i ) for t i At any point p in space i The volume density at a given location is given by the time-varying symbolic distance function t. i At any point p in space i The output value at TVSDF(MSI(p) i ),t i The following formula can be used to transform the expression:

[0099] σ(p i )=κ·σ'(-TVSDF(MSI(p i ),t i ))

[0100]

[0101] Where κ and λ are learnable parameters, and σ'(x) is actually the cumulative distribution function based on the Laplace distribution, with a mean of 0 and a scale parameter of λ; x in σ'(x) represents the input of the cumulative distribution function; TVSDF is the time-varying symbol distance function, which is based on the input MSI(p i ) and t i The sign distance value is obtained. The sign distance value refers to the minimum distance between the coordinates of a point p in space and the coordinates of a point on the surface of the terrain. This can be achieved through a neural network model, such as TVSDF(MSI(p)). i ),t i ) for t i At any point p in space i The output of the time-varying symbol distance function, i.e., the symbol distance value.

[0102] Assume the input image I corresponding to the pixel i The included band set is Band in The channel optimization mechanism will also construct a 5-dimensional vector R corresponding to that pixel. in =(r B ,rG ,r R ,r IR ,r Pan ), the value of one dimension r k Let I'(i,j) be the value corresponding to the corresponding band, k∈{B,G,R,IR,Pan}; r k The following rules must be met:

[0103]

[0104] For example, if a certain image Band in ={B,G,R,}∪Band other and Then R in =(r B ,r G ,r R ,0,0); Band other This indicates bands other than B, G, and R.

[0105] Ultimately, by R in R out Self-supervised spectral loss can be constructed at the pixel level:

[0106]

[0107] Here, ||·||1 and ||·||0 correspond to the 1-norm and 0-norm, respectively.

[0108] The role of the loss function is to indirectly optimize the spatiotemporal field function by fitting the spectral values. R in With R out These correspond to the input real image and the output rendered image of the spatiotemporal field, respectively. In this invention, the spatiotemporal field is mainly represented by two functions encoded by a neural network: TVSDF and TVSRF. TVSDF represents the geometry of the spatiotemporal field at a certain moment through the signed distance field value, while TVSRF represents the appearance of the spatiotemporal field at a certain moment through the radiation intensity value. By performing differentiable rendering of the three-dimensional geometry and appearance estimated by the two functions in a specific space, this invention obtains the image R at a specific moment and a specific viewpoint. out And through R in and R out The resulting loss function can then be used to optimize the estimation of the spatiotemporal field functions TVSDF and TVSRF. Finally, the optimized TVSDF and TVSRF are used to reconstruct the spatiotemporal field. Specific Implementation Method Two:

[0110] This embodiment is a computer storage medium that stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection.

[0111] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in this invention; the instructions can be used to program computer systems or other electronic devices. Computer storage media may include readable media on which instructions are stored, and may include, but are not limited to, magnetic storage media, optical storage media; magneto-optical storage media include read-only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0113] This embodiment is a self-supervised spatiotemporal field reconstruction device based on multi-mode spatial integral coding and spectral band selection. The device includes a processor and a memory. It should be understood that this includes any device including a processor and a memory described in this invention. The device may also include other units and modules that perform display, interaction, processing, control and other functions through signals or instructions.

[0114] The memory stores at least one instruction, which is loaded and executed by the processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection.

[0115] Those skilled in the art will understand that at least one stored instruction constitutes a computer program product corresponding to a method or system. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application, and can also be used with corresponding devices. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0120] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0121] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection, characterized in that, Based on time-varying radiation function and time-varying symbolic distance function The spatiotemporal field is characterized using an optimized time-varying radiation function. and time-varying symbolic distance function Achieving spatiotemporal field reconstruction; time-varying radiation function and time-varying symbolic distance function The optimization process includes: A light ray emitted by the camera and passing through image pixels is constructed based on the spatiotemporal field imaging geometric model; an integral space of the spatial sampling points is constructed based on the spatial sampling point i on the light ray. ; coordinates in space Position encoding is performed to obtain the encoding result. In the integration space The coding result is integrated to obtain the multi-mode spatial integral coding result. ; pixel values ​​in the image Switch to Bit precision obtained Then convert the result Round down to the nearest integer and convert to an m-bit binary representation; then use the resulting integer... remove get ; For a set of bands According to the time-varying radiation function The output is rendered along this ray, including the following steps: Given a certain pixel and the light passing through that pixel and the depth range of the reconstructed target scene. At that time, firstly Sampling is performed along the inner edge of the ray, resulting in a set of sampling points. ;in, It is the unit direction vector of light. From the camera origin Starting along the depth of the light ray, N represents the number of sampling points; Then, the volume rendering result corresponding to the pixels through which the light passes is obtained according to the following formula: ， ， ， in, Indicates the depth interval between adjacent sampling points. For the length of the ray segment Opacity on For the length of the ray segment The opacity on the surface, the calculation method and same; For light rays to travel from the origin Departure and arrival at space point Cumulative transparency over time For a point in space The corresponding volume rendering integral weight at that location; for At all times in space Volume density at that location; , These are learnable parameters, and It is a cumulative distribution function based on the Laplace distribution, with a mean of 0 and a scale parameter of . ; A neural network model is used, which is based on the input and get , These are the radiation values ​​for the corresponding wavebands. for At all times in space The output of the time-varying radiation function at the given time; Time-varying symbolic distance function A neural network model is used, which is based on the input and The signed distance value is obtained; the signed distance value refers to the coordinates of a point in space. The minimum distance from the coordinates of a point on the surface of the terrain feature. for At all times in space The output of the time-varying symbol distance function; Assuming the input image corresponding to the pixels The included band set is The channel optimization mechanism will also construct a 5-dimensional vector corresponding to that pixel. The value of one dimension For the corresponding band value, ; Meet the rules: ; Depend on , Construct a self-supervised spectral loss at the pixel level ,in, , These correspond to the 1-norm and the 0-norm, respectively. Time-varying radiation function is achieved based on spectral loss. and time-varying symbolic distance function Optimization.

2. The self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection according to claim 1, characterized in that, The integration space , , Let these represent the integral spaces of the corresponding cuboid and cone shapes, respectively; This represents the image corresponding to spatial sampling point i. , These represent satellite and airborne image sets, respectively.

3. The self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection according to claim 2, characterized in that, For coordinates in the integration space The location encoding method is as follows; （2） in, For frequency coefficients, For encoding weights of different frequencies, This indicates the result of the position encoding.

4. A self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection according to any one of claims 1 to 3, characterized in that, pixel values ​​in the image Switch to Bit precision obtained The formula is as follows: in, Indicates the first in the image line, number Columns, Represents the original quantization bit depth of the image.

5. The self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection according to claim 4, characterized in that, coordinates in space Position encoding is performed to obtain the encoding result. In the integration space The coding result is integrated to obtain the multi-mode spatial integral coding result. : Where V represents the integration space corresponding to p.

6. The self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection according to claim 5, characterized in that, The , In This represents the input to the cumulative distribution function.

7. A computer storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection as described in any one of claims 1 to 6.

8. A self-supervised spatiotemporal field reconstruction device based on multi-mode spatial integral coding and spectral band selection, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the self-supervised spatiotemporal field reconstruction method based on multi-mode spatial integral coding and spectral band selection as described in any one of claims 1 to 6.

Images