Dynamic nerve radiation field real-time three-dimensional reconstruction method and device based on window attention and gradient balance

Abstract

The invention discloses a dynamic nerve radiation field real-time three-dimensional reconstruction method and device based on window attention and gradient equilibrium, and the method comprises the steps: carrying out the stratified sampling of rays, and obtaining sampling points; projecting the sampling points to six groups of two-dimensional feature planes; querying corresponding feature vectors from the six groups of feature planes through bilinear interpolation, and generating spatio-temporal features through Hadamard product fusion; inputting the fused spatial-temporal characteristics into a decoding network of a Swin Transform, and outputting the color and the volume density of a corresponding sampling point; and carrying out volume rendering integration along the ray of the camera to generate a composite image, and optimizing and updating the model according to a multi-target loss function between the composite image and a real image. The invention aims to solve the problems of large memory consumption, poor rendering quality, long training time and poor geometric consistency in the prior art, and realizes real-time three-dimensional reconstruction of a dynamic scene by using six-plane grid representation, Swin Transform feature generation and a gradient equilibrium mechanism.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, specifically to a method and apparatus for real-time 3D reconstruction of dynamic neural radiation fields based on window attention and gradient balance. Background Technology

[0002] In the industrial production sector, the steel smelting process involves high temperatures, and 3D reconstruction technology is a core technology for acquiring spatial structure and appearance information of scenes, widely used in virtual reality, autonomous driving, and robot perception. Traditional 3D reconstruction methods such as stereo vision and structured light imaging have been implemented, but they have significant limitations: stereo vision is sensitive to changes in lighting and texture loss, and requires precise camera calibration; structured light relies on dedicated projection equipment and cannot handle dynamic scenes with many moving objects.

[0003] In recent years, neural radiation field technology has achieved new perspectives in the synthesis of high-quality static scenes by implicitly modeling the color and density of scenes. However, existing dynamic neural radiation field models (such as V4D, TiNeuVox, and DyNeRF) still have the following key problems:

[0004] High memory consumption: Dynamic scenes need to process spatial and temporal information simultaneously. Existing models (such as V4D) use high-dimensional grids to store spatiotemporal features, resulting in a large number of parameters and high memory consumption.

[0005] Poor rendering quality: Some models (such as TiNeuVox) use simplified feature representations to reduce memory usage, resulting in loss of details and blurred textures in the composite view;

[0006] Long training time: The spatiotemporal correlation of dynamic scenes increases the complexity of the model. Existing methods (such as DyNeRF) take up to 1344 hours to train, which is extremely inefficient.

[0007] Poor geometric consistency: The depth information of moving objects in dynamic scenes is prone to "floating object" artifacts, and existing regularization methods cannot effectively solve the problem of depth discontinuity in the time dimension. Summary of the Invention

[0008] The main objective of this application is to provide a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance, comprising the following steps:

[0009] Spatial-temporal sampling points are obtained by performing layered sampling of the rays emitted by the camera;

[0010] The spatial-temporal sampling points are projected onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT. The corresponding feature vectors are queried from the six feature planes using bilinear interpolation, and spatiotemporal features are generated by fusing them using Hadamard product. The fused spatiotemporal features are input into a Swin Transformer-based decoding network, which outputs the color and volume density of the corresponding sampling points. A synthetic image from the target viewpoint is generated by volume rendering integration along the camera's rays, and the model is optimized and updated based on the multi-objective loss function between the synthetic image and the real image.

[0011] In one embodiment, the step of obtaining spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera includes:

[0012] The stratified sampling includes coarse sampling and fine sampling. The coarse sampling uniformly selects several points along the ray to estimate the density distribution. The fine sampling further selects several refined points in the high-density region based on the results of the coarse sampling, and calculates the sampling weight based on the density variance in the time dimension, giving priority to refined sampling in the high-density spatiotemporal region.

[0013] Based on the coarse and fine sampling, sampling points corresponding to space and time are obtained.

[0014] In one embodiment, the step of projecting the space-time sampling points onto six sets of two-dimensional feature planes, the six sets of feature planes including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT, includes:

[0015] The six sets of two-dimensional feature planes are constructed using low-rank tensor decomposition, representing the 4D spatiotemporal radiation field as six sets of learnable 2D feature planes. Each set of planes stores M-dimensional features, where M is a positive integer, and the resolution of each set of feature planes is set independently.

[0016] When projecting a 4D spatiotemporal point (x, y, z, t), the XY plane, XZ plane, and YZ plane are used to extract spatial features, while the XT plane, YT plane, and ZT plane are used to extract spatiotemporal correlation features.

[0017] In one embodiment, the step of querying the corresponding feature vectors from the six sets of feature planes through bilinear interpolation and generating spatiotemporal features through Hadamard product fusion includes:

[0018] At each sampling point, it is projected onto six sets of two-dimensional feature planes to obtain the corresponding two-dimensional coordinate pairs;

[0019] In each set of two-dimensional feature planes, the corresponding M-dimensional feature vector is queried using bilinear interpolation based on the coordinates of the projection; the above six sets of feature vectors are then fused using Hadamard product in a preset three-pair combination method to generate three intermediate fused features.

[0020] The three intermediate fused features are concatenated along the channel dimension to obtain a joint feature vector of length 3M. The final fused feature vector is then used as the sampling point.

[0021] In one embodiment, the step of inputting the fused spatiotemporal features into a Swing Transformer-based decoding network and outputting the color and volume density of the corresponding sampling points includes:

[0022] The spatiotemporal features are processed using the Swing Transformer module, which includes input encoding, Transformer blocks, and output decoding.

[0023] The input encoding maps coordinates to a high-dimensional space through a position encoding function. The Transformer block is a multi-layer, multi-head self-attention structure. The output decoding outputs RGB color and volume density.

[0024] In one embodiment, the step of generating a synthetic image from the target viewpoint by performing volume rendering integration along the ray of the camera, and optimizing and updating the model based on a multi-objective loss function between the synthetic image and the real image, includes:

[0025] Integrate RGB color and volume density along the camera ray to generate a reconstructed synthetic image;

[0026] A multi-objective loss function is constructed between the synthetic image and the real image, and optimized by combining a gradient balancing mechanism. The multi-objective loss function includes photometric loss L_color, total variational loss L_TV, temporal smoothing loss L_smooth, and depth consistency loss L_depth. The gradient balancing mechanism adjusts the gradient magnitude of samples at different locations by using a distance-related gradient scaling factor to optimize the backpropagation path.

[0027] In one embodiment, the multi-objective loss function L = L_color + L_TV + L_smooth + L_depth;

[0028] Among them, photometric loss Minimize the error between the rendered color and the true color;

[0029] Total variational loss When applied to a spatial plane, it promotes spatial smoothness;

[0030] Time smoothing loss Reduce time jitter;

[0031] Deep consistency loss Constrained geometric consistency.

[0032] In one embodiment, the depth consistency loss is optimized in conjunction with a gradient balancing mechanism, which introduces a distance-dependent gradient scaling factor to attenuate the gradient in the near field region by the square of the distance and the gradient in the far field region by a slow attenuation, so as to balance the gradient magnitude of sampling points at different depths during backpropagation.

[0033] A real-time 3D reconstruction device for dynamic neural radiation fields based on window attention and gradient balance, the device comprising:

[0034] The sampling module obtains spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera;

[0035] The projection module projects the spatial-temporal sampling points onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, and YZ, and spatiotemporal planes XT, YT, and ZT. The extraction module queries the corresponding feature vectors from the six feature planes using bilinear interpolation and generates spatiotemporal features through Hadamard product fusion. The decoding module inputs the fused spatiotemporal features into a Swin Transformer-based decoding network and outputs the color and volume density of the corresponding sampling points. The output module generates a synthetic image from the target viewpoint by performing volume rendering integration along the camera's rays and optimizes and updates the model based on a multi-objective loss function between the synthetic image and the real image.

[0036] Therefore, this application has the following beneficial effects:

[0037] This application provides a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balancing, comprising the following steps: obtaining spatial-temporal sampling points by hierarchically sampling rays emitted from a camera; projecting the spatial-temporal sampling points onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT; querying the corresponding feature vectors from the six sets of feature planes through bilinear interpolation, and generating spatiotemporal features through Hadamard product fusion; inputting the fused spatiotemporal features into a decoding network based on a Swing Transformer, outputting the color and volume density of the corresponding sampling points; generating a synthetic image from the target viewpoint by performing volume rendering integration along the rays of the camera, and optimizing and updating the model based on the multi-objective loss function between the synthetic image and the real image. This application aims to solve the problems of high memory consumption, poor rendering quality, long training time, and poor geometric consistency in existing technologies by using a six-plane mesh representation, Swing Transformer feature generation, and gradient balancing mechanism to achieve efficient 3D reconstruction of dynamic scenes. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the overall working process of a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance.

[0040] Figure 2 This is a system flowchart of a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance;

[0041] Figure 3 This is a schematic diagram comparing the six-plane mesh representation of the dynamic neural radiation field real-time 3D reconstruction method based on window attention and gradient balance with the traditional representation method;

[0042] Figure 4 This is a schematic diagram of the results of the real-time 3D reconstruction method of dynamic neural radiation field based on window attention and gradient balance on the D-NeRF dataset.

[0043] Figure 5 This is a schematic diagram of the results of a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance on the Plenoptic Video dataset. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0046] To address the shortcomings of existing technologies, this application provides a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balancing. This method compresses spatiotemporal features through six-plane low-rank decomposition, captures global dynamic correlations using a Swing Transformer, and designs a gradient balancing mechanism to optimize geometric consistency, ultimately achieving fast and high-precision 3D reconstruction of dynamic scenes. Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the overall operation of a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance, used to demonstrate the entire process from ray sampling to volume rendering. The core objective of 3D imaging is to extract the 3D spatial structure from 2D observation information, i.e., to reconstruct the depth information of objects or scenes (distances from each point on the object to the observation device), and then combine this with the lateral (X / Y axis) position to construct a complete 3D coordinate system (X / Y / Z axes). The specific technical solution of this method is represented by the following concepts:

[0047] The dynamic neural radiance fields (NeRF) represent the expansion of the input from the 5-dimensional (x, y, z, θ, ϕ) of the static NeRF to the 6-dimensional (x, y, z, t, θ, ϕ); the introduction of a canonical space and a learnable deformable field Ψt maps any point in time back to the canonical configuration, achieving cross-temporal geometric alignment.

[0048] Ray sampling: 3D spatial points are sampled from camera rays. A layered strategy of coarse and fine sampling is combined with importance sampling based on temporal variance. First, 64 coarse sampling points are sampled to estimate the density distribution, and then 128 fine sampling points are sampled for high-density areas to improve sampling efficiency and reconstruction accuracy.

[0049] Six-plane mesh representation: 4D spatiotemporal points are projected onto six sets of feature planes (XY, XZ, YZ, XT, YT, ZT). A 32-dimensional feature vector is extracted for each set of planes using bilinear interpolation. Spatial and temporal features are fused using Hadamard product, and then upscaled to 64 dimensions via position encoding, achieving efficient association of spatiotemporal information. This step reduces the storage complexity of the 4D mesh from O(N) through low-rank decomposition. 4 The computational complexity is reduced to O(N²), and the model memory is only 28MB.

[0050] Transformer-driven feature generation: The Swing Transformer is used instead of the traditional Multilayer Perceptron (MLP) to process fused features. Specifically, this includes:

[0051] Input encoding: The coordinates are mapped to a high-dimensional space through position encoding γ(p), which enhances the feature discrimination.

[0052] Transformer block: 4-layer, 8-head self-attention structure that captures motion patterns in dynamic scenes by calculating global correlations between features;

[0053] Output decoding: The linear layer outputs RGB color and volume density, avoiding the limitations of local features in traditional MLP.

[0054] Window attention mechanism: This mechanism aims to retain the powerful modeling capabilities of the Swin Transformer while reducing computational complexity, making it particularly suitable for high-resolution image or 3D spatial modeling tasks (such as neural radiation fields). It is used in the Swin Transformer decoding network to efficiently process spatiotemporal features retrieved from six plane queries. This processing handles the fused spatiotemporal feature vector, while simultaneously calculating attention for sampling points on each ray in a windowed manner, avoiding global overhead and achieving high-density sampling. The weights of the window attention enhance the feature response of moving regions (such as points with high temporal variance) and suppress static backgrounds.

[0055] Volume rendering: Integrate color and density along the camera ray to calculate the desired rendered color of the pixels and output the reconstructed composite image.

[0056] Loss calculation: Specifically, it is achieved through a multi-objective loss function, including fusion photometric loss (minimizing color error), total variational loss (spatial smoothing), temporal smoothing loss (reducing motion jitter), and depth consistency loss (constraining geometric accuracy) to ensure reconstruction quality.

[0057] Gradient balancing mechanism: The gradient magnitude in different regions is adjusted by a distance-dependent gradient scaling factor. The near-field gradient decays by the square of the distance to avoid premature convergence, while the far-field gradient decays slowly to avoid noise amplification, thus optimizing backpropagation stability.

[0058] This application provides a method for real-time three-dimensional reconstruction of dynamic neural radiation fields based on window attention and gradient balance, including steps S10-S40, as described above. Figure 2 , Figure 2 This is a system flowchart of a real-time 3D reconstruction method for dynamic neural radiation fields based on window attention and gradient balance.

[0059] Step S10: Obtain spatial-temporal sampling points by performing layered sampling on the rays emitted by the camera;

[0060] Step S20: Project the spatial-temporal sampling points onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT; Step S30: Query the corresponding feature vectors from the six sets of feature planes using bilinear interpolation, and generate spatiotemporal features by fusing them using Hadamard product; Step S40: Input the fused spatiotemporal features into a decoding network based on Swin Transformer, and output the color and volume density of the corresponding sampling points; Step S50: Generate a synthetic image from the target viewpoint by performing volume rendering integration along the ray of the camera, and optimize and update the model according to the multi-objective loss function between the synthetic image and the real image.

[0061] Specifically, in this embodiment, high-quality 3D scene reconstruction and dynamic rendering are achieved through a complete chain of hierarchical sampling, multi-plane projection, feature fusion, Swing Transformer decoding, and volume rendering. The six-plane projection mechanism effectively integrates spatial and temporal features, the hierarchical structure of the Swing Transformer balances local details and global consistency, and volume rendering integrals ensure realistic image synthesis. A multi-objective loss function guides the model to optimize multiple dimensions such as color accuracy, geometric details, and temporal coherence, ultimately outputting realistic synthesized images and supporting iterative model updates, making it suitable for cutting-edge application scenarios such as dynamic scene modeling and virtual reality rendering.

[0062] Step S10: Hierarchical sampling to obtain spatial-temporal points

[0063] This step simulates the light projection process using a camera, performing layered sampling along each ray in the 3D scene. The sampling strategy combines spatial depth and temporal dimensions, dividing the continuous space into multiple discrete layers, each corresponding to the coordinates of a different depth or time point. Through layered sampling, a four-dimensional set of sampling points containing spatial location (x, y, z) and time t information can be obtained, providing basic data for subsequent feature extraction.

[0064] Step S20: Projection of six sets of two-dimensional feature planes

[0065] Spatial-temporal sampling points are mapped to six sets of orthogonal two-dimensional feature planes: spatial planes XY, XZ, and YZ capture the geometric structure information of the scene, while spatiotemporal planes XT, YT, and ZT introduce temporal dimension features. Each plane reduces the dimensionality of three-dimensional points to two dimensions through orthogonal projection, forming a multi-view feature representation. For example, the XY plane retains the xy coordinates while ignoring the z-axis, while the XT plane associates the x-coordinate with time t. This multi-plane projection can comprehensively capture the spatial structure and dynamic evolution features of the scene.

[0066] Step S30: Bilinear Interpolation and Feature Fusion

[0067] Bilinear interpolation is performed on the projected 2D plane, and the feature vector corresponding to each sampling point is retrieved from the pre-computed feature map. Bilinear interpolation achieves sub-pixel-level accurate feature extraction by weighted averaging of features from surrounding grid points. Subsequently, the six sets of feature vectors are fused using Hadamard product (element-wise multiplication) to generate spatiotemporal features. This fusion method emphasizes the synergistic effect between features from different planes, preserving spatial structural information while integrating temporal dynamic features to form a semantically rich spatiotemporal feature representation.

[0068] Step S40: Swing Transformer Decoding Network

[0069] The fused spatiotemporal feature input is based on a Swin Transformer decoding network. The Swin Transformer employs a hierarchical window attention mechanism, capturing detailed features through self-attention within local windows and achieving global context modeling through cross-window connections. The decoder progressively upsamples the feature map, ultimately outputting the color (RGB) and volume density (σ) for each sample point. Volume density represents the degree of light attenuation at the sample point, while color reflects the appearance attributes of the scene; together, they constitute the fundamental parameters for volumetric rendering.

[0070] Step S50: Volume rendering and model optimization

[0071] Volume rendering integrals are performed along the camera ray to measure color and density, accumulating the color contribution of sampling points from near to far. The final pixel value is then synthesized using volume density weighting to generate a synthetic image from the target viewpoint. In the optimization phase, a multi-objective loss function is employed to compare the differences between the synthetic image and the real image. This loss function typically includes multi-reconstruction loss, perceptual loss (such as VGG feature matching), and adversarial loss (GAN framework). Backpropagation updates the network parameters, gradually improving rendering quality and scene representation capabilities.

[0072] Furthermore, in this embodiment, the step of obtaining spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera includes:

[0073] The stratified sampling includes coarse sampling and fine sampling. The coarse sampling uniformly selects several points along the ray to estimate the density distribution. The fine sampling further selects several refined points in the high-density region based on the results of the coarse sampling, and calculates the sampling weight based on the density variance in the time dimension, giving priority to refined sampling in the high-density spatiotemporal region.

[0074] Based on the coarse and fine sampling, sampling points corresponding to space and time are obtained.

[0075] Specifically, in this embodiment, the step of obtaining spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera is a key preliminary step in the dynamic neural radiation field reconstruction process. Its purpose is to efficiently and accurately capture the geometric and appearance information that changes over time in a dynamic scene, while avoiding redundant calculations on invalid or low-information areas. This step specifically includes two stages: coarse sampling and fine sampling. These two stages work together to form an adaptive, importance-driven sampling strategy.

[0076] First, in the coarse sampling stage, the system uniformly selects a number of sampling points (typically 64) along the ray direction originating from the camera's optical center and passing through the target pixel, between a pre-defined near clipping plane and a far clipping plane. These points only contain spatial coordinates (x, y, z) and the corresponding time t (determined by the current frame or interpolation time), forming a preliminary spatiotemporal sampling set. The main goal of this stage is not directly used for final rendering, but rather to perform forward inference on these points using a neural radiation field model to estimate their volume density (σ) distribution, thereby obtaining an initial density probability distribution along the ray. This distribution reflects which segments in the scene are more likely to contain visible surfaces or specular details, providing prior guidance for subsequent fine sampling.

[0077] The system then enters the fine-sampling stage. This stage abandons the uniform strategy and instead performs importance sampling based on the density distribution obtained from coarse sampling. Specifically, the ray is divided into several intervals, and a piecewise probability density function is constructed based on the density values ​​of the coarse sampling points. On this basis, several additional refinement points (typically 128) are selected, allowing high-density regions (such as near object surfaces) to be sampled more densely, while low-density regions (such as empty backgrounds) are sampled more sparsely. To further enhance sensitivity to dynamic content, this embodiment introduces the density variance in the time dimension as a sampling weight adjustment factor: for the density estimates of the same spatial location at adjacent time steps, if the variance is large, it indicates that the region has significant motion or deformation, belonging to a "high-dynamic region." In this case, the system dynamically increases the sampling weight of this spatiotemporal region, prioritizing the allocation of more sampling points during the fine-sampling process. This mechanism ensures that, with limited computational resources, the model focuses its attention on spatiotemporal regions with drastic geometric or appearance changes, such as flowing liquids, swaying clothing, or the edges of rapidly moving objects.

[0078] Finally, the coarse and fine sampling points are merged to form a complete, non-uniformly distributed sequence of space-time sampling points along a ray. Each point carries its three-dimensional spatial coordinates (x, y, z) and corresponding time t. These sampling points serve as input for subsequent six-plane feature queries and Transformer feature decoding, laying the data foundation for achieving high-quality, high-efficiency dynamic 3D reconstruction. This hierarchical sampling strategy significantly reduces the number of invalid queries (experiments show a reduction of approximately 40%) while maintaining rendering accuracy, effectively improving overall training and inference efficiency.

[0079] Further, in this embodiment, the step of projecting the space-time sampling points onto six sets of two-dimensional feature planes, the six sets of feature planes including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT, includes:

[0080] The six sets of two-dimensional feature planes are constructed using low-rank tensor decomposition, representing the 4D spatiotemporal radiation field as six sets of learnable 2D feature planes. Each set of planes stores M-dimensional features, where M is a positive integer, and the resolution of each set of feature planes is set independently.

[0081] When projecting a 4D spatiotemporal point (x, y, z, t), the XY plane, XZ plane, and YZ plane are used to extract spatial features, while the XT plane, YT plane, and ZT plane are used to extract spatiotemporal correlation features.

[0082] Specifically, in this embodiment, the core step of low-rank feature representation of dynamic neural radiation fields is to transform high-dimensional spatiotemporal information into efficiently queryable 2D features through the process of "low-rank tensor decomposition to construct a planar carrier → 4D point dimension mapping and normalized projection". This solves the problem of parameter explosion in traditional 4D meshes and achieves accurate separation and extraction of spatial structure and spatiotemporal correlation information. The specific steps are as follows:

[0083] First, six sets of learnable 2D feature planes are constructed based on low-rank tensor decomposition. For dynamic scenes with 4D spatiotemporal radiation fields (dimensions x, y, z, t), traditional high-dimensional grid storage leads to parameter redundancy (complexity O(N)). 4 To address the problem of (where N is the single-dimensional resolution), a low-rank tensor decomposition technique is employed. This technique decomposes the high-dimensional feature tensor of the 4D spatiotemporal radiation field into six independent 2D feature planes (XY, XZ, YZ, XT, YT, ZT), reducing storage complexity to O(N²) and significantly decreasing memory usage. Each plane stores an M-dimensional feature vector (M is a positive integer, practically set to 32 to balance feature representation and computational efficiency). The vector parameters are initialized to a normal distribution and subsequently optimized through model training iterations to adapt to scene features. Simultaneously, the resolution of each plane is independently set: the resolution of the spatially relevant XY, XZ, and YZ planes is set to 512×512 to capture fine-grained spatial information such as object surface texture and spatial contours; the resolution of the spatiotemporally relevant XT, YT, and ZT planes is set to 128×128. This balances memory consumption and training efficiency while ensuring sampling density in the temporal dimension, ultimately keeping the overall memory usage of the model composed of the six planes within 28MB.

[0084] Secondly, dimensional culling and coordinate projection are performed on the 4D spatiotemporal sampling points (x, y, z, t). The projection follows the principle of retaining corresponding dimensions and culling irrelevant dimensions. For the dimensional combinations of the six planes, the coordinate components of the 4D points that match the plane dimensions are extracted and then mapped to the resolution range of the corresponding planes, preparing for subsequent bilinear interpolation to query feature vectors. The specific projection logic is as follows: For the XY plane, the spatial depth dimension z and time dimension t of the 4D points are removed, and only the horizontal-vertical spatial coordinates (x, y) are retained. The original (x, y) coordinates are normalized to the 512×512 pixel index range (0~511) of the XY plane to extract the static structural features of the scene in the XY two-dimensional plane (such as object surface texture and horizontal contours); For the XZ plane, the vertical spatial dimension y and time dimension t are removed, and the horizontal-depth coordinates (x, z) are retained and normalized to the 512×512 range to focus on extracting the depth level and spatial layout features of the scene in the XZ plane; For the YZ plane, the horizontal spatial dimension x and time dimension t are removed, and the vertical-depth coordinates (y, z) are retained and normalized to the 512×512 range to capture the vertical structure and height distribution features of the scene in the YZ plane.

[0085] For the spatiotemporal correlation plane, the projection process prioritizes preserving the correlation information between the spatial and temporal dimensions. The XT plane discards the vertical spatial dimension y and the depth dimension z, retaining the horizontal spatial coordinate x and the temporal coordinate t, normalizing (x, t) to a range of 128×128. This is used to extract the motion characteristics of the object in the X direction over time (such as horizontal displacement and shape expansion / contraction). The YT plane discards the horizontal spatial dimension x and the depth dimension z, retaining the vertical spatial coordinate y and the temporal coordinate t, normalizing to a range of 128×128, capturing the dynamic changes of the object in the Y direction (such as vertical rise and fall, and vertical deformation). The ZT plane discards the horizontal spatial dimension x and the vertical spatial dimension y, retaining the depth coordinate z and the temporal coordinate t, normalizing to a range of 128×128, focusing on extracting the spatiotemporal correlation of the object in the depth direction (such as near and far movement, and changes in occlusion). The six sets of 2D coordinates obtained after projection are directly used as input indices for subsequent bilinear interpolation to query feature vectors, achieving accurate mapping of 4D spatiotemporal information to the 2D feature plane.

[0086] Further, in this embodiment, the step of querying the corresponding feature vectors from the six sets of feature planes through bilinear interpolation and generating spatiotemporal features through Hadamard product fusion includes:

[0087] At each sampling point, it is projected onto six sets of two-dimensional feature planes to obtain the corresponding two-dimensional coordinate pairs;

[0088] In each set of two-dimensional feature planes, the corresponding M-dimensional feature vector is queried using bilinear interpolation based on the coordinates of the projection; the above six sets of feature vectors are then fused using Hadamard product in a preset three-pair combination method to generate three intermediate fused features.

[0089] The three intermediate fused features are concatenated along the channel dimension to obtain a joint feature vector of length 3M. The final fused feature vector is then used as the sampling point.

[0090] Specifically, in this embodiment, the step of querying the corresponding feature vectors from six sets of feature planes through bilinear interpolation and generating spatiotemporal features through Hadamard product fusion is the core step in achieving efficient, low-rank dynamic neural radiation field representation. This process fully utilizes the hexagonal plane structure to compactly encode 4D spatiotemporal information and reconstructs a complete spatiotemporal context representation through a feature fusion mechanism, specifically including the following sub-steps:

[0091] First, for each spatiotemporal sampling point p=(x, y, z, t) obtained through hierarchical sampling, the system projects it onto six predefined two-dimensional learnable feature planes. These six planes cover all pairwise coordinate combinations and are divided into two categories: three spatial planes (XY, XZ, YZ) for capturing static geometry, and three spatiotemporal planes (XT, YT, ZT) for modeling temporal evolution and motion information. The projection operation extracts the two-dimensional combinations of corresponding coordinates. For example, the projected coordinates of a point (x, y, z, t) on the XY plane are (x, y), and on the XT plane are (x, t), and so on, resulting in six pairs of two-dimensional coordinates.

[0092] Next, in each set of two-dimensional feature planes, the system employs bilinear interpolation to query the corresponding M-dimensional feature vector (M=32 in typical settings) based on the projected coordinates. Since the feature planes are learnable parameters stored in a discrete grid (e.g., resolution 256×256 or 512×128), and the coordinates of the sampling points are usually continuous real numbers, a weighted average is performed between the four nearest neighbor grid points using bilinear interpolation to obtain a smooth, differentiable feature response. This operation ensures the feature query process is robust to small coordinate changes and supports end-to-end gradient backpropagation. Hadamard products (i.e., element-wise multiplication) are then fused according to three pre-defined complementary combinations. Specifically, the spatial plane XY is fused with the spatiotemporal plane ZT to obtain f1; XZ is fused with YT to obtain f2; and YZ is fused with XT to obtain f3. This cross-pairing design ensures that information from any two coordinate dimensions is coupled during the fusion process, thus fully preserving the low-rank structure of the four-dimensional spatiotemporal tensor while avoiding information loss.

[0093] Finally, the three intermediate fusion features f1, f2, and f3 are concatenated along the channel dimension (i.e., the feature dimension) to form a joint feature vector f = [f1:f2:f3] ∈ R3M with a length of 3M. This vector integrates the coupled information of spatial geometry and temporal dynamics, serving as the final spatiotemporal context representation of the sampling point (x, y, z, t), and is input into the subsequent Swin Transformer decoding network for predicting color and volume density. The entire process significantly reduces the number of model parameters while maintaining high-fidelity spatiotemporal modeling capabilities, providing a solid foundation for fast and high-quality dynamic 3D reconstruction. Figure 3 This is a schematic diagram comparing the six-plane mesh representation of the dynamic neural radiation field real-time 3D reconstruction method based on window attention and gradient balance with the traditional representation method.

[0094] Furthermore, in this embodiment, the step of inputting the fused spatiotemporal features into a SwinTransformer-based decoding network and outputting the color and volume density of the corresponding sampling points includes:

[0095] The spatiotemporal features are processed using the Swing Transformer module, which includes input encoding, Transformer blocks, and output decoding.

[0096] The input encoding maps coordinates to a high-dimensional space through a position encoding function. The Transformer block is a multi-layer, multi-head self-attention structure. The output decoding outputs RGB color and volume density.

[0097] Specifically, in this embodiment, the step of inputting the fused spatiotemporal features into a Swin Transformer-based decoding network and outputting the color and volume density of the corresponding sampling points is a key decoding step for achieving high-quality dynamic scene reconstruction. This step abandons the local modeling method relying on multilayer perceptrons in traditional NeRF and instead adopts the Swin Transformer architecture, which has global modeling capabilities and efficient computational characteristics, to more effectively capture long-range spatiotemporal dependencies in dynamic scenes. The specific process includes three stages: input encoding, Transformer block processing, and output decoding.

[0098] First, in the input encoding stage, the system enhances the fused spatiotemporal feature vector (3M=96 dimensions) by introducing positional encoding to explicitly inject the geometric prior information of the spatial coordinates (x, y, z) and observation directions (θ, ϕ) of the sampling points. Positional encoding employs a high-frequency sine and cosine function form:

[0099]

[0100] The spatial coordinates are encoded using 10 layers (L=10), and the observation direction uses 4 layers (L=4), both mapped to a high-dimensional frequency domain. This encoding, concatenated with the fused features, forms an enhanced input rich in geometric and viewpoint information, which is then fed into the subsequent Transformer module.

[0101] Secondly, in the Transformer block processing stage, a multi-layer Swin Transformer structure is used for deep modeling of the input features. Each layer includes a window-based multi-head self-attention (W-MSA) submodule, layer normalization, and a feedforward MLP submodule. In this embodiment, a four-layer stacked structure is set, with each layer containing eight attention heads. The core advantage of the Swin Transformer lies in its local window partitioning and cross-window shifting mechanism: self-attention is calculated within each local window, significantly reducing computational complexity; at the same time, cross-regional information interaction is achieved by alternately shifting window boundaries, balancing efficiency and global receptive field. This mechanism enables the model to adaptively focus on regions of intense motion (such as a waving arm or flowing liquid) while suppressing redundant computation of static backgrounds, significantly improving the reconstruction accuracy and training efficiency of dynamic details, and reducing training time.

[0102] Finally, in the output decoding stage, the high-dimensional features (typically 256 dimensions) processed by the Transformer block are fed into two independent linear projection layers: one for predicting RGB color values ​​c=(r, g, b)∈[0, 1], and the other for outputting a non-negative volume density σ≥0 through the Softplus activation function. This decoding process is fully differentiable and supports end-to-end training. In summary, this Swing Transformer decoding network not only inherits the powerful sequence modeling capabilities of the Transformer but also adapts to the characteristics of sparse, unstructured sampling points in neural radiation fields through a window mechanism. Compared to traditional MLPs, its parallel computing architecture improves training speed by nearly 2 times, while significantly improving PSNR (up to 31.79) and perceptual quality (LPIPS as low as 0.023) on datasets such as D-NeRF and Plenoptic Video. It effectively solves problems such as detail blurring and motion artifacts in dynamic scenes, achieving the goal of real-time, high-fidelity 3D dynamic reconstruction.

[0103] Further, in this embodiment, the step of generating a synthetic image from the target viewpoint by performing volume rendering integration along the ray of the camera, and optimizing and updating the model based on the multi-objective loss function between the synthetic image and the real image, includes:

[0104] Integrating the RGB color and volume density along the camera ray generates a reconstructed synthetic image;

[0105] A multi-objective loss function is constructed between the synthetic image and the real image, and optimized by combining a gradient balancing mechanism. The multi-objective loss function includes photometric loss L_color, total variational loss L_TV, temporal smoothing loss L_smooth, and depth consistency loss L_depth. The gradient balancing mechanism adjusts the gradient magnitude of samples at different locations by using a distance-related gradient scaling factor to optimize the backpropagation path.

[0106] Specifically, in this embodiment, the steps of generating a synthetic image from the target viewpoint by performing volume rendering integration along the camera's rays, and optimizing and updating the model based on the multi-objective loss function between the synthetic image and the real image, are the final output and feedback control links of the entire dynamic neural radiation field reconstruction process. The aim is to achieve high-fidelity, geometrically consistent, and training-stable dynamic new viewpoint synthesis.

[0107] First, in the volume rendering integration stage, the system performs color synthesis for each ray originating from the camera's optical center and passing through image pixels. Specifically, for each sampling point on the ray whose RGB color and volume density σ have been predicted, its transmittance and opacity along the ray are calculated. Finally, the pixel color corresponding to that ray is obtained by integration using the classical volume rendering formula. This process fuses the color and density information of discrete sampling points into a continuous, realistic 2D synthesized image, which serves as the current reconstruction result of the model.

[0108] Subsequently, in the loss function construction and optimization stage, a multi-objective joint loss function was designed to comprehensively constrain the model's performance in terms of appearance, spatial smoothness, temporal coherence, and geometric consistency. Specifically, the photometric loss L_color uses mean squared error (MSE) to measure the pixel-level difference between the synthesized image C and the real image C*; the total variational loss L_TV operates on the spatial planes (XY / XZ / YZ) in the six planes, suppressing grid-like artifacts and improving spatial smoothness through L2 norm regularization of the feature plane gradients; the temporal smoothing loss L_smooth constrains the consistency of features in adjacent frames through second-order temporal differences, effectively reducing dynamic jitter; and the depth consistency loss L_depth utilizes real or pseudo-depth supervision to constrain the alignment of the predicted depth with the geometric structure, mitigating the floating object problem.

[0109] Crucially, this embodiment introduces a gradient balancing mechanism to optimize the backpropagation process. Since ray sampling is more dense in the near-field region, its gradients tend to dominate training, leading to underoptimization in the far-field or dynamic regions. To address this, a distance-dependent gradient scaling factor, in the form of a smooth transition function, is designed to achieve triple adjustment: in the near-camera region, the gradient decays by the square of the distance to prevent premature convergence; in the main scene region, the original gradient strength is maintained; and in the far-field region, the gradient decays slowly to avoid amplifying noise. This mechanism dynamically adjusts the contribution of sampling points at different depths to parameter updates, resulting in a more balanced gradient distribution and significantly improving the stability and consistency of geometric reconstruction.

[0110] Finally, based on this multi-objective loss and gradient balancing strategy, the system uses the Adam optimizer (initial learning rate 5e-4, exponential decay) to perform end-to-end updates of model components such as the six-plane parameters and Transformer weights. Experiments show that this approach not only reduces LPIPS to 0.023 and effectively eliminates artifacts, but also reduces the number of training iterations by about 50%, achieving a balance between high quality and high efficiency.

[0111] Furthermore, in this embodiment, the multi-objective loss function ;

[0112] Among them, photometric loss Minimize the error between the rendered color and the true color;

[0113] Total variational loss When applied to a spatial plane, it promotes spatial smoothness;

[0114] Time smoothing loss Reduce time jitter;

[0115] Deep consistency loss Constrained geometric consistency.

[0116] Specifically, in this embodiment, the multi-objective loss function guides the model to learn the true appearance and spatial structure of dynamic scenes through the collaborative optimization of four dimensions: photometric matching, spatial smoothing, temporal stability, and geometric constraints. This avoids color distortion, artifact retention, or motion jitter problems caused by a single loss function. The following provides a detailed explanation of the calculation logic, parameter definitions, optimization objectives, and technical value of each loss term:

[0117] I. Photometric Loss (L_color): Ray-by-ray color error constraint

[0118] Photometric loss is a core constraint on the accuracy of model appearance reconstruction. By minimizing the ray-by-ray error between rendered colors and real colors, it ensures the visual consistency of the composite view. The calculation logic and function are as follows:

[0119] Iterate through all camera rays (denoted as r, covering all pixels of the image), calculate the square of the L2 norm for the rendered color C(r) of each ray (obtained by volume rendering integration) and the real color Ĉ(r) (extracted from the labeled images in the dataset), and then sum the errors over all rays. The formula is:

[0120]

[0121] Where Ĉ(r) is the RGB value of the real pixel corresponding to ray r (normalized to the [0, 1] interval), and C(r) is the predicted RGB value rendered by the model along ray r. This forces the model to learn the mapping relationship between rays and colors, making the rendering result close to the real scene at the pixel level, avoiding problems such as global color cast and local texture loss (e.g., blurred surface patterns). L_color is directly related to visual perception quality; its optimization effect determines the realism of the synthesized view, providing an appearance benchmark for subsequent spatial, temporal, and geometric optimizations.

[0122] II. Total Variational Loss (L_TV): Spatial Feature Plane Smoothing Constraint

[0123] The total variational loss (TV loss) applies smoothing regularization to the spatial feature planes (XY, XZ, YZ planes), suppressing abrupt changes in feature pixels and reducing mesh-like artifacts or spatial discontinuities in the reconstruction results. Its calculation logic and function are as follows:

[0124]

[0125] Iterate through all pixels in the spatial feature planes (denoted as c, covering each feature point in the XY, XZ, and YZ planes), calculate the gradient ∇P_c between pixel P_c and its neighboring pixels (usually taking the first-order difference in the horizontal and vertical directions, i.e., ∇P_c = (P_{c+1,x} - P_c, P_{c+1,y} - P_c)), sum the squares of the gradients over the L2 norm, and then multiply by the weight coefficient λ_TV; λ_TV is used to balance spatial smoothing and detail preservation. If the weight is too large, it will lead to over-smoothing of features (loss of texture), and the criterion will not be able to suppress artifacts.

[0126] By constraining the continuous change of pixel values ​​in the spatial feature plane, and avoiding abrupt changes between adjacent pixels (such as sudden changes in feature values ​​on the surface of the same object), blocky blurring or mesh textures in the rendered image are eliminated. This addresses the spatial discontinuity problem that may be caused by the discretization of feature planes in six-plane decomposition, ensuring that the spatial structure of the reconstructed scene (such as object edges and surface textures) is smooth and natural, consistent with the spatial continuity of the real scene.

[0127] III. Temporal Smoothing Loss (L_smooth): Inter-frame Motion Stability Constraint

[0128] Temporal smoothing loss reduces inter-frame jitter in dynamic scenes (such as position jumps of moving objects) by constraining feature changes in the time dimension through second-order difference, ensuring the continuity of motion. The calculation logic and function are as follows:

[0129] Iterate through all continuous time steps (denoted as t, covering the entire time series of the scene's dynamic process), calculate the second-order difference (simulating acceleration constraints) for the feature pixels P_t, P_{t+1}, and P_{t+2} at the same spatial location at times t, t+1, and t+2. Sum the squared L2 norms of the second-order differences and multiply by the weighting coefficient λ_smooth, as shown in the formula:

[0130]

[0131] Where λ_smooth is the time smoothing weight, and the second difference is... Essentially, it's the difference between adjacent frames, capturing abrupt changes in motion. A larger value indicates a more discontinuous motion at time t+1 relative to times t and t+2. This forces the model to learn the characteristic changes of continuous time steps, ensuring that the inter-frame position and shape changes of moving objects (such as flowing coffee or moving animals) conform to physical inertia, avoiding unnatural motion such as jitter. It overcomes the shortcomings of traditional NeRF in modeling the temporal dimension by designing constraints for the core requirement of dynamic scenes (motion continuity), ensuring that the synthesized multi-frame views form a smooth video sequence, rather than isolated static images.

[0132] IV. Depth Consistency Loss (L_depth): Geometric Accuracy Constraint

[0133] Depth consistency loss constrains the error between predicted depth and true depth, ensuring the geometric integrity of the reconstructed scene and reducing floating object artifacts (such as depth discontinuities of moving objects). Its calculation logic and function are as follows:

[0134] Iterate through all camera rays r, and for each ray, calculate the square of the L2 norm of the predicted depth d(r) (obtained from the depth integral of the volume rendering) and the true depth d_gt(r) (extracted from the depth map labeled in the dataset), sum them, and multiply by the weight coefficient λ_depth. The formula is as follows:

[0135]

[0136] Here, λ_depth is the depth weight (set to 1 in the experiment), which is the largest weight among the four loss terms. Geometric consistency is the core of dynamic reconstruction and must be prioritized. d(r) is the average depth of the effective density region on ray r, and d_gt(r) is the true depth value corresponding to ray r. The model is forced to learn the mapping relationship between rays and depth, ensuring that the predicted depth field is consistent with the spatial structure of the real scene, avoiding depth floating of moving objects (e.g., the depth of an object is too far forward in one frame and too far backward in the next) or geometric misalignment (e.g., the depth of an object overlaps with the background). This addresses the key pain point of poor geometric consistency in existing dynamic NeRF by binding appearance optimization and geometric optimization through depth constraints, ensuring that the synthesized view is not only visually realistic but also that its spatial structure conforms to the three-dimensional laws of the physical world.

[0137] V. Collaborative Calculation and Optimization Process for Multi-Objective Losses

[0138] The total loss is calculated by weighted summation of the four loss items. The collaborative optimization process is as follows:

[0139] In each iteration of the model, C(r) (color) and d(r) (depth) of all rays are first obtained through volume rendering, and then L_color (color error) and L_depth (depth error) are calculated respectively; at the same time, P_c of the spatial feature plane and P_t of the time series are extracted, and L_TV (spatial smoothing) and L_smooth (temporal smoothing) are calculated.

[0140] The settings of weights λ_TV, λ_smooth, and λ_depth reflect the optimization priorities of geometry priority, spatiotemporal assistance, and appearance basis. Depth has the largest weight (to ensure geometric correctness), spatiotemporal smoothing has a smaller weight (to avoid excessive smoothing and loss of details), and photometric loss has no additional weight (it directly participates in the summation with the original error to ensure appearance baseline). Figure 4 This paper presents visual comparison results of novel perspective synthetic images from comparative experiments conducted on the standup, hook, and bouncingballs scenes of the D-NeRF dataset. Figure 4 It can be seen that the method of this application is significantly superior to the other methods in terms of reconstruction results. Compared with other advanced methods, the method of this application also performs better in reconstructing the texture and shadow details of the helmet surface. Figure 5 This paper presents some novel results on the Plenoptic Video dataset. By comparing two small regions containing detailed texture information, the method in this application performs well in reconstructing moving objects (such as flowing coffee and a moving dog) and capturing better texture details.

[0141] Furthermore, in this embodiment, the depth consistency loss is optimized by combining a gradient balancing mechanism. The gradient balancing mechanism introduces a distance-dependent gradient scaling factor, which decays the gradient in the near field region by the square of the distance and the gradient in the far field region decays slowly, so as to balance the gradient magnitude of sampling points at different depths during backpropagation.

[0142] Specifically, in this embodiment, a gradient balancing mechanism is introduced to optimize the depth consistency loss and address the geometric discontinuity problem that occurs during dynamic scene reconstruction. The core of this mechanism lies in adjusting the gradient magnitude of samples at different locations by introducing a distance-dependent gradient scaling factor, thereby optimizing the backpropagation path and making model training more stable and efficient.

[0143] Depth consistency loss is primarily used to constrain the difference between the predicted depth map and the true depth map, thereby enhancing the geometric consistency of the reconstruction results. It is typically defined as the squared error of the difference between the predicted depth and the true ground depth. To further improve the stability of model training and ensure reasonable gradient contributions across different depth regions, this embodiment proposes a distance-based gradient scaling strategy. Specifically:

[0144] Near-field region: Due to the high density of sampling points near the camera, if left uncontrolled, the gradients at these points may dominate the entire training process, leading to underoptimization in the far field or dynamic regions. Therefore, in the near-field region, the gradient decays with the square of the distance; that is, as the distance increases, the gradient scaling factor decreases rapidly to avoid premature convergence.

[0145] Mid-field region: In this region, the gradient scaling factor remains relatively constant, allowing the original gradient strength to play a role and maintaining the model's ability to learn the main scene structure.

[0146] Far-field region: To avoid amplifying gradient noise caused by sparse sampling, the gradient in the far-field region decays slowly. This gradual adjustment helps improve the reconstruction quality of far-field details while reducing unnecessary gradient fluctuations.

[0147] Other embodiments or specific implementations of the real-time three-dimensional reconstruction device for dynamic neural radiation fields based on window attention and gradient balance in this application can be found in the above-described method embodiments, and will not be repeated here.

[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0149] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] It should be particularly noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for real-time 3D reconstruction of dynamic neural radiation fields based on window attention and gradient balance, characterized in that, Includes the following steps: Spatial-temporal sampling points are obtained by performing layered sampling of the rays emitted by the camera; The space-time sampling points are projected onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT; the corresponding feature vectors are queried from the six sets of feature planes by bilinear interpolation, and spatiotemporal features are generated by Hadamard product fusion. The fused spatiotemporal features are input into a Swin Transformer-based decoding network, which outputs the color and volume density of the corresponding sampling points. A synthetic image from the target viewpoint is generated by performing volume rendering integration along the ray of the camera, and the model is optimized and updated based on the multi-objective loss function between the synthetic image and the real image.

2. The method according to claim 1, characterized in that, The step of obtaining spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera includes: The stratified sampling includes coarse sampling and fine sampling. The coarse sampling uniformly selects several points along the ray to estimate the density distribution. The fine sampling further selects several refined points in the high-density region based on the results of the coarse sampling, and calculates the sampling weight based on the density variance in the time dimension, giving priority to refined sampling in the high-density spatiotemporal region. Based on the coarse and fine sampling, sampling points corresponding to space and time are obtained.

3. The method according to claim 1, characterized in that, The step of projecting the space-time sampling points onto six sets of two-dimensional feature planes, the six sets of feature planes including spatial planes XY, XZ, YZ and spatiotemporal planes XT, YT, ZT, includes: The six sets of two-dimensional feature planes are constructed using low-rank tensor decomposition, representing the 4D spatiotemporal radiation field as six sets of learnable 2D feature planes. Each set of planes stores M-dimensional features, where M is a positive integer, and the resolution of each set of feature planes is set independently. When projecting a 4D spatiotemporal point (x, y, z, t), the XY plane, XZ plane, and YZ plane are used to extract spatial features, while the XT plane, YT plane, and ZT plane are used to extract spatiotemporal correlation features.

4. The method according to claim 1, characterized in that, The step of querying the corresponding feature vectors from the six sets of feature planes through bilinear interpolation and generating spatiotemporal features through Hadamard product fusion includes: At each sampling point, it is projected onto six sets of two-dimensional feature planes to obtain the corresponding two-dimensional coordinate pairs; In each set of two-dimensional feature planes, the corresponding M-dimensional feature vector is queried using bilinear interpolation based on the coordinates of the projection; the above six sets of feature vectors are then fused using Hadamard product in a preset three-pair combination method to generate three intermediate fused features. The three intermediate fused features are concatenated along the channel dimension to obtain a joint feature vector of length 3M. The final fused feature vector is then used as the sampling point.

5. The method according to claim 1, characterized in that, The step of inputting the fused spatiotemporal features into a Swing Transformer-based decoding network and outputting the color and volume density of the corresponding sampling points includes: The spatiotemporal features are processed using the Swing Transformer module, which includes input encoding, Transformer blocks, and output decoding. The input encoding maps coordinates to a high-dimensional space through a position encoding function. The Transformer block is a multi-layer, multi-head self-attention structure. The output decoding outputs RGB color and volume density.

6. The method according to claim 1, characterized in that, The step of generating a synthetic image from the target viewpoint by performing volume rendering integration along the ray of the camera, and optimizing and updating the model based on the multi-objective loss function between the synthetic image and the real image, includes: Integrate RGB color and volume density along the camera ray to generate a reconstructed synthetic image; A multi-objective loss function is constructed between the synthetic image and the real image, and optimized by combining a gradient balancing mechanism. The multi-objective loss function includes photometric loss L_color, total variational loss L_TV, temporal smoothing loss L_smooth, and depth consistency loss L_depth. The gradient balancing mechanism adjusts the gradient magnitude of samples at different locations by using a distance-related gradient scaling factor to optimize the backpropagation path.

7. The method according to claim 6, characterized in that, The multi-objective loss function ; Among them, photometric loss Minimize the error between the rendered color and the true color; Total variational loss When applied to a spatial plane, it promotes spatial smoothness; Time smoothing loss Reduce time jitter; Deep consistency loss Constrained geometric consistency.

8. The method according to claim 6, characterized in that, The depth consistency loss is optimized by combining a gradient balancing mechanism, which introduces a distance-dependent gradient scaling factor. The gradient in the near field region decays by the square of the distance, while the gradient in the far field region decays slowly, so as to balance the gradient magnitude of sampling points at different depths during backpropagation.

9. A real-time three-dimensional reconstruction device for dynamic neural radiation fields based on window attention and gradient balance, characterized in that, The device includes: a sampling module, which obtains spatial-temporal sampling points by performing layered sampling of the rays emitted by the camera; The projection module projects the spatial-temporal sampling points onto six sets of two-dimensional feature planes, including spatial planes XY, XZ, and YZ, and spatiotemporal planes XT, YT, and ZT. The extraction module queries the corresponding feature vectors from the six feature planes using bilinear interpolation and generates spatiotemporal features through Hadamard product fusion. The decoding module inputs the fused spatiotemporal features into a Swin Transformer-based decoding network and outputs the color and volume density of the corresponding sampling points. The output module generates a synthetic image from the target viewpoint by performing volume rendering integration along the camera's rays and optimizes and updates the model based on a multi-objective loss function between the synthetic image and the real image.

Images