Deep space 4d dynamic scene reconstruction method and system based on large model

Abstract

The application relates to the technical field of scene reconstruction, and discloses a deep space 4D dynamic scene reconstruction method and system based on a large model, wherein the construction and reasoning process of a reconstruction model comprises the following steps: obtaining image tokens by performing word segmentation on images, calculating the Plucker ray embedding of each pixel of the image through camera external parameters and camera internal parameters, and converting a timestamp into time coding; dividing the image tokens and the Plucker ray embedding into blocks and combining the time coding to form input tokens; obtaining a query token according to the Plucker ray embedding of a target view angle and the time coding of a target timestamp; inputting the input tokens and the query token into a Transform model to output a target image token corresponding to the target view angle and the target timestamp, and generating a target view angle image at the target timestamp. The application shows good generalization capability for complex dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of scene reconstruction technology, specifically to a method and system for deep space 4D dynamic scene reconstruction based on a large model. Background Technology

[0002] Reconstructing dynamic objects and scenes from videos is a fundamental yet challenging task in computer vision. Its goal is to build a consistent four-dimensional (4D) representation across different viewpoints and time frames under sparse visual observations. This task is crucial for applications such as 4D asset generation, virtual reality, and robotics. In particular, in the field of deep space exploration, the reconstruction of dynamic scenes on the lunar surface needs to address issues such as low light, high radiation, communication delays, and unique dynamic characteristics (such as dust suspension and object movement in low gravity environments). Existing static scene reconstruction technologies are difficult to adapt to these challenges, and dynamic scene reconstruction faces even more severe challenges.

[0003] Existing 4D reconstruction methods can be broadly classified into two categories:

[0004] The first category comprises scene-by-scene optimization methods, such as Neural Radiant Fields (D-NeRF) for dynamic scene representation, deformable neural radiant fields (Nerfies), HyperNeRF for handling topologically changing scenes, and deep learning-based dynamic scene modeling (Neural Scene Flow Fields). These methods model scene-specific dynamics through neural fields. Recent methods, such as 4D Gaussian Splatting, GaussianFlow-based dynamic 3D reconstruction, and deformation-based schemes (see Tretschk et al. 2021), have further improved quality. These methods typically offer high accuracy, but each scene requires separate optimization, leading to high training costs and poor generalization.

[0005] The second category comprises generalizable 4D reconstruction methods that utilize pre-trained models to improve inference efficiency and transferability. Examples include Geo4D (see Jiang et al. 2025), Monst3r (see Zhang et al. 2024a), L4GM (see Ren et al. 2024), and transformer or diffusion frameworks (see Sajjadi et al. 2022; Kulhanek et al. 2022; Rombach, Esser, and Ommer 2021; Liang et al. 2024b; Yinet et al. 2023). While these methods offer better scalability, they are often limited to object-level scenes and rely on hand-designed priors, increasing complexity and limiting adaptability.

[0006] Based on the above analysis, this invention aims to construct a 4D feedforward reconstruction framework that combines high accuracy, strong generalization, and efficient end-to-end inference capabilities to meet the needs of lunar dynamic scene reconstruction in deep space exploration. A natural approach is to extend large models that have demonstrated good generalization capabilities in static scene reconstruction, such as LVSM (see Jin et al. 2024a), to 4D tasks. However, this faces two major challenges: (1) how to effectively model special lunar dynamics (such as dust movement and robotic arm operation trajectories under low gravity) without relying on explicit motion priors; and (2) how to construct a large-scale lunar dynamic dataset (including deep space exploration measured data and high-fidelity simulation data) for robust training and evaluation. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method and system for deep space 4D dynamic scene reconstruction based on a large model.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] Firstly, this invention provides a method for reconstructing a deep-space 4D dynamic scene based on a large model. The construction and reasoning process of the reconstruction model includes:

[0010] Get Images from various perspectives, along with corresponding camera extrinsic parameters, camera intrinsic parameters, and timestamps; the images are from optical cameras carried by deep space exploration spacecraft, covering different types of lunar terrain, such as plains, impact craters, and mountains, and are labeled with the camera's pose parameters in the lunar coordinate system;

[0011] For each viewpoint image, image tokens are obtained by segmenting the image, and Plück ray embeddings for each pixel of the image are calculated using camera extrinsic and intrinsic parameters. Timestamps are converted into time codes based on Transformer position coding. The set of image tokens and Plück ray embeddings is divided into blocks and combined with the time codes to form the input token.

[0012] The query token is obtained by splicing and linearly projecting the set of Plück ray embeddings from the target viewpoint and the time code of the target timestamp; the set of Plück ray embeddings from the target viewpoint is obtained through the camera extrinsic and intrinsic parameters of the target viewpoint.

[0013] Input tokens and query tokens are fed into the Transformer model, which outputs a target image token corresponding to the target viewpoint and the target timestamp. The Transformer model is a decoder-only architecture or an encoder-decoder architecture, which achieves dynamic scene reconstruction by implicitly learning spatiotemporal patterns.

[0014] The target image token is decoded to generate a target view image at the target timestamp.

[0015] In one embodiment, the step of calculating the Plücker ray embedding of each pixel of the image using camera extrinsic and intrinsic parameters, and converting the timestamp into time code based on Transformer position coding, specifically includes:

[0016] For the image at the i-th perspective Using the corresponding camera extrinsic parameters and camera internal reference Calculate the Plücker ray embedding for each pixel to obtain the image. Plück ray embedding set , , Total number of images; for timestamp After normalization, Transformer position encoding is used to convert the timestamp into a time code. :

[0017] ;

[0018] Indicates a linear projection layer. This indicates the number of frequency bands used in Transformer position coding.

[0019] In one embodiment, the step of dividing the image token and the set of Plück ray embeddings into blocks and combining them with time encoding to form the input token specifically includes:

[0020] Image from the i-th perspective The set of image tokens divided into image blocks is denoted as . , for The corresponding j-th image patch, These represent the height and width of the image, respectively; the width and height of the image patch are both [missing information]. ;

[0021] The corresponding Plück ray embedding set The set of embedded blocks into which the parts are divided is denoted as , for The j-th embedded block;

[0022] The input token corresponding to the j-th image patch for:

[0023] ;

[0024] in, As a potential spatial dimension, Represents the linear projection layer of the input token. for timestamp Time encoding, Indicates splicing.

[0025] In one embodiment, the step of concatenating and linearly projecting the set of Plück ray embeddings from the target viewpoint and the time code of the target timestamp to obtain the query token specifically includes:

[0026] Plücker ray embedding set from the target perspective With target timestamp Time-encoded data is spliced ​​and linearly projected:

[0027] ;

[0028] in, Indicates the target perspective. An embedding block of a Plück ray embedding set A linear projection layer representing the target's viewpoint. This indicates that a time encoding operation is being performed. From the perspective of the target A query token.

[0029] In one embodiment, the input tokens and query tokens are rearranged before being input into the Transformer model:

[0030] The sequence of input tokens is rearranged into a one-dimensional sequence. ,in Given the length of the input token sequence, For the first One input token, These are the height and width of the image, respectively. The width and height of the image block corresponding to the input token are both [missing information]. The sequence of query tokens is also arranged as a one-dimensional sequence. ,in, To query the length of the token sequence, For the first A query token.

[0031] In one embodiment, the step of inputting the input token and query token into the Transformer model and outputting a target image token corresponding to the target viewpoint and the target timestamp specifically includes:

[0032] Using the Transformer model Given a conditional input token, generate a target image token from the target's perspective:

[0033]

[0034] For the first One target image token, This indicates conditional input.

[0035] In one embodiment, decoding the target image token to generate a target viewpoint image at the target timestamp specifically includes:

[0036] Each target image token corresponds to an image patch in the target viewpoint image, via a linear header. and the Sigmoid function This yields the RGB vector of the target viewpoint:

[0037] ;

[0038] This represents the j-th RGB vector generated from the target's viewpoint; For the first One target image token;

[0039] The generated RGB vector is converted to a shape of The image blocks are stitched together in order to obtain the final target view image. .

[0040] In one embodiment, the Transformer model is a decoder-only architecture or an encoder-decoder architecture, specifically including:

[0041] Decoder-only architecture: The input token sequence and query token sequence are concatenated and used as input to the Transformer model. The final output target image token sequence is decoded only for the query token sequence, while the Transformer model outputs an update token sequence based on the input token sequence. Discarded:

[0042] ;

[0043] in, The Transformer model employs a decoder-only architecture and consists of multiple stacked layers of fully self-attention layers;

[0044] Encoder-decoder architecture: The encoder first encodes the input token into a latent token, and the decoder then predicts the output based on the latent token and the query token, specifically including:

[0045] Introduction The k-th learnable latent token is used to extract and aggregate information from the input token; the k-th learnable latent token is... , ;

[0046] The encoder of the Transformer model Composed of multiple self-attention layers, it concatenates the input token sequence with the sequence of learnable latent tokens to generate an updated latent token sequence. The encoder outputs an updated token sequence based on the input token sequence. Discarded;

[0047] ;

[0048] decoder It consists of multiple layers of self-attention Transformer modules, and the input is the updated sequence of potential tokens. The query token sequence is used as the output, which is the target image token sequence corresponding to the position of the query token sequence. The modified potential token sequence output by the decoder Discarded:

[0049] .

[0050] In one embodiment, the loss function used for training the reconstruction model is... for:

[0051] ;

[0052] in, Image from the target's perspective. This is the true image of the target's viewpoint at time t. This is the term for calculating the mean square error. For the calculation of perceived loss, The weights of the terms are calculated for the perceived loss.

[0053] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0054] Compared with the prior art, the beneficial technical effects of the present invention are:

[0055] This invention proposes a generalizable 4D reconstruction framework that integrates temporal encoding into the Transformer model. It can learn dynamic spatiotemporal patterns from data without manual 4D priors, and establishes a simple and effective fully data-driven 4D dynamic reconstruction paradigm.

[0056] This invention constructs a large-scale, diverse dataset and designs a unified training scheme with progressive temporal sampling to achieve efficient joint learning of static and dynamic scenes. This method simultaneously achieves higher reconstruction quality, generalization ability, and inference speed on multiple dynamic view synthesis benchmarks. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the reconstruction model in an embodiment of the present invention.

[0058] Figure 2 This is a qualitative comparison chart of different models in this embodiment of the invention on the DyCheck dataset.

[0059] Figure 3 This is a qualitative comparison chart of different models in the embodiments of the present invention on the NVIDIA dynamic scene dataset.

[0060] Figure 4 This is a qualitative comparison chart of different variants of the reconstruction model in this embodiment of the invention on the RealEstate10K dataset.

[0061] Figure 5 This is a visual comparison chart of different time encoding methods on the DyCheck dataset in an embodiment of the present invention.

[0062] Figure 6 This is a schematic diagram illustrating zero-sample generalization on the DyCheck dataset for different numbers of input images in an embodiment of the present invention.

[0063] Figure 7 This is a schematic diagram illustrating the rendering performance of the reconstruction model in this embodiment of the invention under different input image quantities.

[0064] Figure 8 This is an extended qualitative comparison graph of different models in the embodiments of the present invention on the DyCheck dataset.

[0065] Figure 9 This is a visualization result on the terrain camera dataset of the Chang'e-4 lander in an embodiment of the present invention.

[0066] Figure 10 This is a schematic diagram of the comprehensive training curve in an embodiment of the present invention.

[0067] Figure 11 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0068] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0069] like Figure 11 As shown, the deep space 4D dynamic scene reconstruction method based on a large model in this invention includes the following steps in the construction and reasoning process of the reconstruction model:

[0070] S1, obtain Images from each perspective, along with the corresponding camera extrinsic parameters, camera intrinsic parameters, and timestamps;

[0071] S2, for each viewpoint image, image tokens are obtained by segmenting the image, and Plück ray embeddings of each pixel of the image are calculated using camera extrinsic and intrinsic parameters. The timestamps are converted into time codes based on Transformer position coding. The set of image tokens and Plück ray embeddings is divided into blocks and combined with the time codes to form the input token.

[0072] S3, the query token is obtained by splicing and linearly projecting the set of Plück ray embeddings of the target viewpoint and the time code of the target timestamp; wherein the set of Plück ray embeddings of the target viewpoint is obtained through the camera extrinsic and intrinsic parameters of the target viewpoint.

[0073] S4, input the input token and query token into the Transformer model, and output the target image token corresponding to the target viewpoint and the target timestamp; the Transformer model is a decoder-only architecture or an encoder-decoder architecture, and realizes dynamic scene reconstruction by implicitly learning the spatiotemporal pattern;

[0074] S5, decode the target image token to generate a target view image at the target timestamp.

[0075] This invention proposes an end-to-end 4D scene reconstruction model based on the Transformer model, capable of reconstruction from sparse multi-view observations. Given... A sparse viewpoint image and its corresponding camera external parameters Camera internal parameters With timestamp , The objective of this invention is to provide new perspectives in any new way. With time Synthesize the target viewpoint image. These are the camera extrinsic and intrinsic parameters from the target's perspective, respectively. The input embedding scheme, Transformer model architecture, and training methods will be described in detail below.

[0076] 1. Input embedding scheme:

[0077] like Figure 1 As shown, the reconstruction model (Large View 4D Synthesis Model, LVSM4D) of this invention directly renders the target image using an end-to-end Transformer model. The reconstruction model first segments the input image. For each viewpoint, the Plücker ray embedding of each pixel is calculated using camera extrinsic and intrinsic parameters, denoted as... To encode time information, this invention uses timestamps. Normalized to the interval [0,1], and time-encoded using the standard Transformer position encoding method: .

[0078] in, This indicates the number of frequency bands used in the Transformer position encoding. These sine and cosine functions are mapped into learnable embeddings through a linear projection layer, thereby effectively capturing temporal variations at multiple scales.

[0079] Referring to the image tokenization strategy of the ViT model, this invention divides the set of images and Plücker ray embeddings into non-overlapping patches. (Image) The corresponding image blocks and embedding blocks are denoted as follows: , ; , ; , ,in, That's the size of the block. Image. The corresponding image patch and embedding patch are linearly mapped to an input token:

[0080] ;

[0081] in, As a potential spatial dimension, This indicates splicing. The process ensures that each input token simultaneously contains complete information about the image, geometry, and time.

[0082] In order to achieve the target timestamp The present invention constructs a query representation using a synthesized target view image, specifically by using a set of Plück ray embeddings. (Camera extrinsics from the target's perspective) With camera internal parameters (Obtained), and combined with the time encoding of the target timestamp to obtain:

[0083] ;

[0084] in, Indicating the first from the target's perspective The embedding block of the Plück ray embedding set.

[0085] The input token indicates that the sequence is rearranged into a 1D sequence. ,in The input token sequence length is specified; the query token sequence is also arranged as follows. ,in .

[0086] 2. Transformer model architecture:

[0087] This invention employs a Transformer model. Given a conditional input token, generate a target image token:

[0088] ;

[0089] Each target image token This corresponds to a block in the target view image. The RGB vector prediction is obtained by applying a linear head layer with a sigmoid function activation:

[0090] ;

[0091] The generated RGB vector is converted to a shape of Two-dimensional image blocks are generated and stitched together in block order to obtain the final target view image. .

[0092] Figure 2 The first column contains four different input images, the last column contains real images from a specific viewpoint, and the other columns contain images synthesized from the input images using the HyperNeRF model, Nerfies model, T-NeRF model (see Gao et al. 2022), Mosca model (see Lei et al. 2025), 4D-GS model (see Wu et al. 2024), LVSM3D model (see Jin et al. 2024a), the LVSM4D model of this invention (decoder-only architecture), and the LVSM4D model of this invention (encoder-decoder architecture).

[0093] This invention proposes two architectural variants, including as follows: Figure 1 The decoder-only architecture shown in (a) and as in Figure 1 The encoder-decoder architecture is shown in (b) of the diagram.

[0094] Decoder-only architecture: The input token sequence and query token sequence are concatenated as input to the Transformer model. The final output target image token sequence is decoded only for the query token sequence, while the Transformer model outputs an update token sequence based on the input token sequence. It is discarded. The process is formalized as follows:

[0095] ;

[0096] in, The Transformer model employs a decoder-only architecture and consists of multiple stacked layers of fully self-attention layers.

[0097] Encoder-decoder architecture: The encoder first encodes the input token into a latent token representation, and the decoder then predicts the output based on the latent token representation and the query token. This invention introduces... The k-th learnable latent token is used to extract and aggregate information from the input token; the k-th learnable latent token is... , .

[0098] The encoder of the Transformer model It consists of multiple self-attention layers. It concatenates the input token sequence with the sequence of learnable latent tokens to generate an updated latent token sequence. This serves as an intermediate representation of the scene, while the encoder outputs an updated token sequence based on the input token sequence. It was discarded.

[0099] decoder It consists of multiple layers of self-attention Transformer modules. Specifically, the input to the decoder is the updated sequence of latent tokens. The query token sequence is used. After multiple self-attention operations, the output sequence dimension remains unchanged, and the output is the target image token sequence corresponding to the position of the query token sequence. The modified potential token sequence output by the decoder Discarded. This structure can be represented as:

[0100] ;

[0101] .

[0102] 3. Training methods:

[0103] To effectively train the reconstruction model (LVSM4D), this invention designs a dedicated training strategy and loss function for the spatiotemporal characteristics of 4D scene reconstruction.

[0104] In terms of temporal sampling, this invention employs a progressive strategy: in the early stages of training, frames with shorter time intervals are used to help the reconstruction model learn short-term dynamics; subsequently, frames with larger time intervals are gradually introduced to capture long-term motion. In terms of spatial sampling, this invention randomly selects viewpoints within a given time range to ensure the diversity of spatial coverage, enabling the model to effectively utilize both temporal and spatial cues from sparse multi-view inputs.

[0105] Synthetic target view image Supervision by combination of photometric loss and perceptual loss:

[0106] ;

[0107] in, Weights are lost based on perceived similarity. This strategy enables the reconstruction model (LVSM4D) to learn dynamic scene changes from sparse multi-view inputs.

[0108] 4. Experiment:

[0109] This invention conducted comprehensive experiments to evaluate the effectiveness of the reconstruction model (LVSM4D) in dynamic scene reconstruction. The evaluation included qualitative comparisons, quantitative assessments, ablation experiments, and efficiency and generalization analyses.

[0110] (1) Datasets: Training directly on limited 4D data often leads to overfitting and poor generalization ability. To alleviate this problem, this invention adopts a data augmentation strategy that integrates heterogeneous data sources. This invention uses a variety of 4D dynamic scene datasets, including the PointOdyssey dataset (see Zheng et al. 2023), the DynamicStereo dataset (see Karaev et al. 2023), the Spring dataset (see Mehl et al. 2023), static 3D scene datasets (such as the Re10K dataset, see Jin et al. 2024b), and video data (such as the CamVid-30K dataset, see Zhao et al. 2025).

[0111] Two key lunar surface measurement datasets were also included: First, the Chang'e-4 topographic camera dataset (CSTR:14804.11.03.88.00006, DOI:10.12176 / 03.88.00006), which contains panoramic image sequences of points A and D on the far side of the moon, as well as images of the lunar rover's front and side views. It covers multi-view data with different projection methods, including azimuth and cylindrical projections, recording the rover's position at the exploration point and surrounding terrain features, used for assessing the accuracy of lunar far-side scene reconstruction. Second, the Chang'e-4 panoramic camera Class 2B scientific dataset (CSTR:14804.11.03.04.004, DOI:10.12176 / 03.04.004), compiled by the National Astronomical Observatories of China, contains lunar surface image data around the landing area acquired by the Chang'e-4 rover during the first 15 lunar days. Geometric positioning information has been added, and the data uses PDS 4. The format (containing paired data and tag files) provides detailed lunar surface topography and precise location correlation data for training and validating the model's geometric positioning accuracy. Dataset statistics are shown in Table 1. This diverse training set enables the model to learn strong 3D spatial representations while capturing unique 4D temporal variations on the lunar surface (such as lighting dynamics during lunar day and night cycles and the rover's trajectory).

[0112] Table 1 shows the statistics of the dataset used for training.

[0113]

[0114] (2) Experimental setup:

[0115] This invention evaluates the reconstruction model (LVSM4D) on three benchmarks:

[0116] The DyCheck dataset (see Gao et al. 2021) contains 12 dynamic scenes with non-rigid deformation, camera motion, and lighting variations, captured using mobile devices for real-world scene evaluation. This invention selects eight temporally adjacent input images from the training set. Baseline results are reproduced using its official implementation.

[0117] The NVIDIA Dynamic Scene Dataset (see Yoon et al. 2020) contains dynamic scenes with diverse motion patterns. Following the DynNeRF (Kratimenos, Lei, and Daniilidis 2024) setup, input frames are sampled in a round-robin fashion at different timestamps. Evaluation is performed across time on the first camera pose.

[0118] The RealEstate10K dataset (see Zhou et al. 2018) is a large-scale static scene reconstruction dataset containing 80K video clips, using the same training / test split as pixelSplat (see Charatan et al. 2024).

[0119] This invention employs three standard evaluation metrics: PSNR (pixel-level reconstruction accuracy), SSIM (structural similarity assessment), and LPIPS (perceptual quality assessment).

[0120] This invention compares with several state-of-the-art methods on different datasets. For dynamic scenes (DyCheck and NVIDIA), the comparison methods include HyperNeRF (Park et al. 2021b), Nerfies (Park et al. 2021a), T-NeRF (Gao et al. 2022), Mosca (Lei et al. 2025), 4D-GS (Wu et al. 2024), DynNeRF (Kratimenos, Lei, and Daniilidis 2024), NSFF (Li et al. 2021), RoDynNeRF (Ma et al. 2023), CasualFVS (Leroy, Cabon, and Revaud 2024), and LVSM (Jin et al. 2024a). For static scenes (RealEstate10K), this invention is compared with pixelNeRF (Yu et al.2021), pixelSplat (Charatan et al. 2024), GS-LRM (Zhang et al. 2024b), and LVSM (Jin et al. 2024a).

[0121] The LVSM4D model of this invention is trained under four input images. The block size is set to... Potential dimensions The model was trained on eight NVIDIA A100 GPUs with a batch size of 16 per GPU, a total of 30,000 steps, a learning rate of 1e-4, and the AdamW optimizer. Mixed precision training was employed to improve efficiency.

[0122] (3) Qualitative comparison:

[0123] Figure 2 and Figure 3Visual comparisons across different datasets are presented. The LVSM4D model of this invention generates sharper geometry, more coherent motion, and better lighting consistency, demonstrating strong robustness under diverse motion patterns and varying scene complexities. Furthermore, Figure 4 This further demonstrates that the LVSM4D model maintains high-quality view compositing performance in a RealEstate10K static scene.

[0124] Figure 3 For a qualitative comparison of different models on the NVIDIA Dynamic Scene Dataset, from left to right, the images are synthesized by the D-Nerf model, Mononerf model, MoSca model, and LVSM4D model (decoder architecture only), and a real image.

[0125] Figure 4 Qualitative comparison of different models on the RealEstate10K dataset. From left to right: input image, images synthesized by the LVSM model (decoder architecture only) and the LVSM4D model (decoder architecture only), and a real image.

[0126] (4) Quantitative assessment:

[0127] This invention quantitatively evaluates LVSM4D on three benchmark datasets to verify its performance in dynamic and static scenes.

[0128] Table 2 shows a quantitative comparison of different models on the DyCheck dataset.

[0129]

[0130] PSNR stands for Pixel Reconstruction Precision, SSIM for Structural Similarity Assessment, and LPIPS for Perceptual Quality Assessment. Upward arrows indicate that a higher score is better, while downward arrows indicate that a lower score is better.

[0131] Table 3 shows a quantitative comparison of different models on the NVIDIA dynamic scene dataset.

[0132]

[0133] Table 4 shows a quantitative comparison of different models on the Re10K dataset.

[0134]

[0135] Table 2 compares the LVSM4D model with state-of-the-art models on the DyCheck dataset, which contains challenging dynamic scenes. The LVSM4D model (decoder architecture variant only) outperforms the previous best Mosca model by 4.30 dB in PSNR, improves SSIM by 0.107, and reduces LPIPS by 0.141, demonstrating superior reconstruction quality and perceptual fidelity.

[0136] Based on Table 3, it can be observed that on the NVIDIA Dynamic Scene Dataset (Yoon et al. 2020), there are significant differences between the input and target viewpoint images in both time and space. The LVSM4D model achieves performance comparable to the scene-by-scene optimization method, demonstrating its robustness under large temporal and spatial gaps.

[0137] This invention tests the LVSM4D model on the RealEstate10K dataset. Based on the results in Table 4, the unified 4D framework of this invention maintains strong static 3D reconstruction capabilities while generalizing to dynamic scenes.

[0138] (5) Ablation test

[0139] This invention validates different temporal encoding methods of the LVSM4D model through ablation experiments on the DyCheck dataset (Table 5). The visualization results are shown below. Figure 5 . Figure 5 This is a visual comparison of different temporal encoding methods on the DyCheck dataset. Each column from left to right represents the input image, Fourier encoding, learnable embedding, Multilayer Perceptron (MLP) encoding, positional encoding, and the ground truth image. The positional encoding scheme of this invention generates the most accurate reconstruction with better temporal consistency.

[0140] All time values ​​are amplified by a factor of 1000 and discretized into position indices before being input to the time encoding module. This invention evaluates four time encoding strategies: (1) Fourier encoding: capturing time patterns using a sine function of log-scaled frequencies, with no trainable parameters; (2) Learnable embeddings: introducing a trainable embedding table for discrete time indices; (3) MLP encoding: transforming time values ​​using a multilayer perceptron; (4) Position encoding: based on standard Transformer position encoding (Vaswani et al. 2017; Dosovitskiy et al. 2020) with an added linear projection layer. The absence of time encoding results in a significant performance degradation, demonstrating its crucial importance for dynamic modeling.

[0141] Table 5 shows the time-encoded ablation results on the DyCheck dataset.

[0142]

[0143] (6) Experimental analysis:

[0144] Table 6 compares the efficiency of the reconstruction model (LVSM4D) of this invention with existing models (T-NeRF, Nerfies, HyperNeRF, 4D-GS, Mosca), where FPS (frames per second) was measured under the condition of 8 input images. The method of this invention does not require scene-by-scene training and significantly improves inference speed while maintaining competitive memory usage.

[0145] Table 6. Efficiency Comparison.

[0146]

[0147] Figure 6 The performance of the reconstruction model (LVSM4D) of this invention is demonstrated under different numbers of input images. The model performance steadily improves with the increase of the number of viewpoints; competitive results can be achieved even with only a single view input, indicating its strong generalization ability.

[0148] This invention proposes a generalizable and efficient Transformer-based dynamic 4D scene reconstruction framework, particularly suitable for lunar surface scenes in deep space exploration. By enhancing a lightweight temporal coding module on top of a static 3D model, without introducing complex spatiotemporal structures, data-driven spatiotemporal modeling of the lunar surface is achieved without relying on explicit 4D priors (such as Gaussian representations or dynamic NeRF). The reconstruction model of this invention includes two variants: a model using a decoder-only architecture focuses on high-quality generation and is suitable for detailed lunar surface topographic analysis from the ground; a model using an encoder-decoder architecture achieves faster inference through latent feature aggregation and is suitable for real-time scene monitoring from the spacecraft. To support large-scale training, this invention compiles a multimodal dataset containing static lunar surface scenes (such as pre-exploration images of the lunar landing area), multi-view dynamic sequences of the lunar surface (such as videos captured during the rover's movement), and simulated lunar surface data (such as dynamic scenes generated based on a lunar digital twin system), enabling the model to learn the special dynamics of the lunar surface while maintaining spatial accuracy. Extensive experiments based on benchmarks such as the DyCheck dataset, RealEstate10K dataset, and NVIDIA Dynamic Scene dataset demonstrate that the reconstruction model (LVSM4D) of this invention outperforms both optimization-based methods and feedforward baselines in terms of reconstruction quality, inference speed, and generalization ability.

[0149] Given the scarcity of high-quality 4D datasets, this invention compiles a large-scale, diverse hybrid dataset, including static scenes from Re10K (Jin et al. 2024b), multi-view dynamic sequences (PointOdyssey (Zheng et al. 2023), DynamicStereo (Karaev et al. 2023), Spring (Mehl et al. 2023)), and monocular videos from CamVid-30K (Zhao et al. 2025). To efficiently utilize this hybrid data, this invention employs a unified training strategy, progressively increasing the time interval between the input and target viewpoint images, enabling the model to progressively learn more challenging dynamic patterns. Extensive benchmark results demonstrate that the LVSM4D model significantly outperforms existing scene-by-scene optimization methods and large pre-trained models; on the DyCheck benchmark, the decoder-only architecture variant of this invention achieves state-of-the-art results, with PSNR exceeding existing methods by 1.5-3.5 dB.

[0150] Figure 7 This demonstrates the trend of rendering performance as the number of input images changes. Here, "rendering" refers to the decoding process. Both variants of this invention maintain reasonable frame rates despite increased input complexity, with the encoder-decoder architecture exhibiting higher efficiency due to its use of a fixed-size latent representation.

[0151] Figure 8 Further qualitative comparisons on the DyCheck dataset are provided, demonstrating the superior performance of the reconstruction model (LVSM4D) of this invention in diverse dynamic scenes containing complex motion patterns and varying lighting.

[0152] The encoder extracts and aggregates information from the input tokens into a latent representation, while the decoder generates the target output. Both use a self-attention mechanism and do not introduce additional structural inductive bias.

[0153] The reconstruction model of this invention uses a standard Transformer model (multi-head self-attention, layer normalization, and feedforward network). The decoder-only architecture variant contains 24 Transformer layers, with a latent space dimension of [missing information]. The model features 8 attention heads; the encoder-decoder architecture variant consists of a 12-layer encoder and a 12-layer decoder, with 2048 learnable latent tokens introduced into the intermediate latent space. All models employ QK-Normalization to improve training stability.

[0154] The time encoding module uses sinusoidal position encoding. Each frequency band (corresponding to an embedding dimension of 128, i.e.) The temporal embedding dimension is set to 128, and then the dimension is hidden using a linear projection matching model. This invention normalizes the timestamp to [0,1] and applies this encoding consistently to all blocks of each input image, following the standard Transformer positional encoding formula.

[0155] Figure 9 shows the visualization results on the Chang'e-4 lander terrain camera dataset, demonstrating the reconstruction capability of the LVSM4D model (decoder architecture only) of this invention for real lunar scenes. From left to right: input image (measured image of the Chang'e-4 landing area, including core features such as lunar rock distribution, crater edges, rover body and surrounding environment, providing basic observation information for reconstruction), new perspective image generated by the LVSM4D model (decoder architecture only) of this invention (new perspective reconstruction results corresponding to key lunar exploration areas, such as the 1km range around the landing point, the permanent shadow area of ​​the lunar south pole, etc.), and real image (real lunar scene under the new perspective of the target, including precise geometric positioning and terrain details, serving as a direct reference for reconstruction accuracy). It can be seen that the new perspective image generated by the LVSM4D model (decoder architecture only) can accurately restore lunar terrain features (such as rock texture details and visual effects of crater depth) and the spatial position of the rover. It highly matches the real image in terms of detail fidelity and overall scene consistency, verifying the model's adaptability to real deep-space lunar scenes.

[0156] Figure 10 The comprehensive training curves are presented, demonstrating the convergence behavior of the reconstruction model of this invention under the progressive training strategy. The overall training loss, normalized loss, and original perceptual loss all steadily decrease, while the PSNR continuously improves, reflecting the convergence behavior and stability of the reconstruction model during the optimization process and verifying the effectiveness of the progressive training strategy and its smooth transition between stages. Figure 10 (a) in the figure is a graph showing the change of training pixel-level reconstruction accuracy (PSNR) over time. Figure 10 Figure (b) shows the evolution of the norm-aware loss throughout the training process. Except for a few spikes during stage transitions, the overall process is stable, indicating that the optimization process is healthy and there is no gradient explosion or vanishing. Figure 10 (c) in the figure represents perceived loss. Figure 10 In the diagram, (d) represents the overall training loss.

[0157] The overall training loss steadily decreased from approximately 0.152 to approximately 0.045, indicating effective learning in all three stages; the normalized perceptual loss stabilized at around 40-41, and the original perceptual loss converged from 0.279 to approximately 0.086; the PSNR improved from approximately 19.2 dB to 26.8 dB, validating the effectiveness of the progressive strategy. Key features include:

[0158] Phase 1 (0-500 steps): The model learns to reconstruct the basic space, and the loss decreases rapidly;

[0159] Phase 2 (500-35000 steps): Steady improvement through the introduction of dynamic modeling over time.

[0160] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0161] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0162] In one embodiment, a computer system is provided, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0165] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for reconstructing a deep-space 4D dynamic scene based on a large model, characterized in that, The construction and reasoning process of the reconstruction model used includes: Get Images from each perspective, along with the corresponding camera extrinsic parameters, camera intrinsic parameters, and timestamps; For each viewpoint image, image tokens are obtained by segmenting the image, and Plücker ray embeddings for each pixel of the image are calculated using camera extrinsic and intrinsic parameters. Timestamps are then converted to time codes based on Transformer positional encoding. The set of image tokens and Plücker ray embeddings is divided into blocks and combined with the time codes to form the input token, specifically including: the image from the i-th viewpoint. The set of image tokens divided into image blocks is denoted as . , for The corresponding j-th image patch, These represent the height and width of the image, respectively; the width and height of the image patch are both [missing information]. ; The corresponding Plück ray embedding set The set of embedded blocks into which the parts are divided is denoted as , for The j-th embedded block; The input token corresponding to the j-th image patch for: , As a potential spatial dimension, Represents the linear projection layer of the input token. for timestamp Time encoding, Indicates splicing; The query token is obtained by splicing and linearly projecting the set of Plück ray embeddings from the target viewpoint and the time code of the target timestamp; the set of Plück ray embeddings from the target viewpoint is obtained through the camera extrinsic and intrinsic parameters of the target viewpoint. Input tokens and query tokens are fed into the Transformer model, which outputs a target image token corresponding to the target viewpoint and the target timestamp. The Transformer model is a decoder-only architecture or an encoder-decoder architecture, which achieves dynamic scene reconstruction by implicitly learning spatiotemporal patterns. The target image token is decoded to generate a target view image at the target timestamp.

2. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 1, characterized in that, The process of calculating the Plücker ray embedding of each pixel in the image using camera extrinsic and intrinsic parameters, and converting the timestamp into a time code based on Transformer position encoding, specifically includes: For the image at the i-th perspective Using the corresponding camera extrinsic parameters and camera internal reference Calculate the Plücker ray embedding for each pixel to obtain the image. Plück ray embedding set , , Total number of images; for timestamp After normalization, Transformer position encoding is used to convert the timestamp into a time code. : ； Indicates a linear projection layer. This indicates the number of frequency bands used in Transformer position coding.

3. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 1, characterized in that, The process of concatenating and linearly projecting the set of Plück ray embeddings from the target viewpoint and the time code of the target timestamp to obtain the query token specifically includes: Plücker ray embedding set from the target perspective With target timestamp Time-encoded data is spliced ​​and linearly projected: ； in, Indicates the target perspective. An embedding block of a Plück ray embedding set A linear projection layer representing the target's viewpoint. This indicates that a time encoding operation is being performed. From the perspective of the target A query token.

4. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 1, characterized in that, Before inputting the sequence of input tokens and query tokens into the Transformer model, the input tokens and query tokens are rearranged: The sequence of input tokens is rearranged into a one-dimensional sequence. ,in Given the length of the input token sequence, For the first One input token, These are the height and width of the image, respectively. The width and height of the image block corresponding to the input token are both [missing information]. The sequence of query tokens is also arranged as a one-dimensional sequence. ,in, To query the length of the token sequence, For the first A query token.

5. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 4, characterized in that, The step of inputting the input token and query token into the Transformer model and outputting a target image token corresponding to the target viewpoint and the target timestamp specifically includes: Using the Transformer model Given a conditional input token, generate a target image token from the target's perspective: ； For the first One target image token, This indicates conditional input.

6. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 5, characterized in that, Decoding the target image token to generate a target view image at the target timestamp specifically includes: Each target image token corresponds to an image patch in the target viewpoint image, via a linear header. and the Sigmoid function This yields the RGB vector of the target viewpoint: ； This represents the j-th RGB vector generated from the target's viewpoint; For the first One target image token; The generated RGB vector is converted to a shape of The image blocks are stitched together in order to obtain the final target view image. .

7. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 5, characterized in that, The Transformer model is either a decoder-only architecture or an encoder-decoder architecture, specifically including: Decoder-only architecture: The input token sequence and query token sequence are concatenated and used as input to the Transformer model. The final output target image token sequence is decoded only for the query token sequence, while the Transformer model outputs an update token sequence based on the input token sequence. Discarded: ； in, The Transformer model employs a decoder-only architecture and consists of multiple stacked layers of fully self-attention layers; Encoder-decoder architecture: The encoder first encodes the input token into a latent token, and the decoder then predicts the output based on the latent token and the query token, specifically including: Introduction The k-th learnable latent token is used to extract and aggregate information from the input token; the k-th learnable latent token is... , ; The encoder of the Transformer model Composed of multiple self-attention layers, it concatenates the input token sequence with the sequence of learnable latent tokens to generate an updated latent token sequence. The encoder outputs an updated token sequence based on the input token sequence. Discarded; ； decoder It consists of multiple layers of self-attention Transformer modules, and the input is the updated sequence of potential tokens. The query token sequence is used as the output, which is the target image token sequence corresponding to the position of the query token sequence. The modified potential token sequence output by the decoder Discarded: 。 8. The method for deep space 4D dynamic scene reconstruction based on a large model according to claim 1, characterized in that, The loss function used in training the reconstruction model for: ； in, Image from the target's perspective. This is the true image of the target's viewpoint at time t. This is the term for calculating the mean square error. For the calculation of perceived loss, The weights of the terms are calculated for the perceived loss.

9. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images