A dynamic scene volume video reconstruction method based on feature group gating fusion

Abstract

The application provides a dynamic scene volume video reconstruction method based on feature group gating fusion. The method comprises the following steps: sampling a group of three-dimensional Gaussian primitives in a 3D scene space, dividing all local features of all Gaussian primitives into several feature groups according to a space-time relationship; inputting local features of a current sampling position into a gating network, weighting and fusing the local features in each feature group according to fusion weights of the feature groups to obtain fused space-time features; predicting an update amount of position, rotation and scale of each Gaussian primitive at each time instant by a morphing decoder to obtain a morphed Gaussian primitive; and rendering a group of target Gaussian primitives into a target perspective image through a differentiable rasterization, and taking the target perspective image as a synthesized dynamic scene image. The method can simultaneously improve the problems of insufficient space-time feature expression and insufficient local sampling density in a complex dynamic scene, and suppress redundant Gaussians, thereby significantly improving the local details and overall quality of the reconstruction.

Description

Technical Field

[0001] This invention relates to the field of dynamic scene video reconstruction technology, and in particular to a dynamic scene volumetric video reconstruction method based on feature group gating fusion. Background Technology

[0002] 3D reconstruction and free-viewpoint rendering of dynamic scenes are research hotspots in computer vision and computer graphics. The goal is to recover continuously changing 3D scenes from multi-view or single-view video sequences and support high-quality rendering of scenes from any viewpoint at any time. This video format, capable of simultaneously recording 3D geometry, surface appearance, and their evolution over time, is called volumetric video. Compared to traditional 2D video, volumetric video allows viewers to freely switch viewing angles within the reconstructed 3D space, making it a core data format for immersive applications such as virtual reality (VR), augmented reality (AR), free-viewpoint video, remote immersive conferencing, film and television special effects, and multi-view replay of sporting events.

[0003] In recent years, with the rapid development of implicit and explicit 3D representation techniques such as Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS), significant breakthroughs have been achieved in the quality and efficiency of new perspective synthesis of static scenes. Building on this foundation, researchers have further introduced the temporal dimension, proposing dynamic scene reconstruction methods represented by 4D Gaussian Splatting (4DGS). These methods typically employ a compact spatiotemporal feature structure (such as a HexPlane six-plane representation) combined with a deformation field network to model the motion and deformation of 3D Gaussian primitives over time, thereby achieving efficient reconstruction and real-time rendering of dynamic scenes. Because explicit Gaussian representation combines editability and rendering efficiency, 4DGS-like methods have gradually become the mainstream technology for volumetric video reconstruction.

[0004] However, real-world dynamic scenes often involve a large amount of rapid motion, non-rigid deformation, frequent occlusion, and complex hand-object interactions. In these complex dynamic regions, existing methods still have significant shortcomings in restoring details of rapidly moving edges, fine-grained textures, and complex interactive areas. The reconstruction results are prone to problems such as blurred edges, smoothed textures, and missing local structures, making it difficult to meet the high-fidelity reconstruction requirements of immersive applications. Therefore, how to further enhance the ability of 4DGS to express local details in complex dynamic regions while maintaining its high efficiency advantages has become a key problem urgently needing to be solved in this field.

[0005] Although dynamic scene reconstruction methods, represented by 4DGS, have advantages in efficiency and overall quality, they still have the following prominent problems in practical applications for complex dynamic scenes:

[0006] (1) Fixed spatiotemporal feature fusion methods result in insufficient representation of complex motion regions. Existing methods, when using structures such as HexPlane for spatiotemporal feature fusion, typically employ direct splicing or fixed weighting for the six spatial-temporal feature planes. Since the spatiotemporal relationships depicted by different planes are not consistent—some planes focus more on local spatial geometry, while others focus more on dynamic information that changes over time—if the same fusion method is used in all spatial locations and temporal states, it is difficult to reflect the relative importance of different plane features in different regions. Especially in areas of rapid motion, non-rigid deformation, or dense texture, a fixed fusion method leads to insufficient feature representation, thereby affecting the reconstruction quality.

[0007] (2) The spatial distribution of Gaussian primitives does not match the error distribution, resulting in insufficient local representation density. Existing methods mainly rely on gradient-based adaptive densification strategies to increase Gaussian primitives. However, this strategy cannot directly perceive the reconstruction error distribution in the image space. It often repeatedly adds Gaussian primitives in regions that have already been well-fitted, while lacking sufficient explicit representation in high-error regions that are truly underrepresented (such as motion edges, complex texture regions, and hand-object contact surfaces). This results in limited detail recovery capabilities in these key regions.

[0008] (3) Unreasonable opacity distribution and redundant Gaussians lead to blurred edges and temporal jitter. In dynamic Gaussian reconstruction, if there are a large number of Gaussian primitives with intermediate opacity in a local area, color overlap is likely to occur, reducing edge sharpness. At the same time, if the opacity of the same Gaussian primitive changes too much in adjacent time steps, it will also cause discontinuity in the rendering results of adjacent time steps, resulting in temporal jitter. Existing methods lack effective constraints on opacity distribution and temporal changes.

[0009] Based on the different scene representation and dynamic modeling methods, the existing dynamic scene reconstruction methods can be roughly divided into the following three categories: (1) methods based on implicit neural radiation fields; (2) methods based on explicit dynamic Gaussian; and (3) methods based on spatiotemporal feature decomposition.

[0010] One existing dynamic scene reconstruction method based on four-dimensional Gaussian splatting (4DGS) is a representative scheme for explicit dynamic Gaussian reconstruction. This method introduces a temporal dimension on top of three-dimensional Gaussian splatting, employs a HexPlane six-plane structure to compactly encode the spatiotemporal field, and uses a multilayer perceptron (MLP) deformation decoder to predict the position, rotation, and scale changes of each Gaussian element over time, thereby achieving the synthesis of new perspectives and moments in the dynamic scene. Its main algorithm flow is as follows:

[0011] Normalized Gaussian initialization: First, a set of three-dimensional Gaussian primitives are initialized in normalized space. Each Gaussian has attributes such as position, covariance (parameterized by rotation and scale), opacity, and spherical harmonic color.

[0012] Spatiotemporal feature query: For a given sampling point (x, y, z, t), the corresponding local features are obtained by interpolation from the six space-time planes of HexPlane, and then fused into a unified spatiotemporal feature representation by a fixed method (splicing or weighted summation).

[0013] Deformation prediction and Gaussian transform: The fused spatiotemporal features are input into the MLP deformation decoder to predict position offset, rotation update and scale update respectively, which are then applied to the normalized Gaussian space to obtain the deformed Gaussian at the current time.

[0014] Differentiable rasterization rendering: The deformed Gaussian elements are projected onto the target viewpoint, and the image is synthesized through a differentiable rasterization splatter process. End-to-end optimization is performed by utilizing the color loss between the rendered image and the real image.

[0015] The drawbacks of the existing dynamic scene reconstruction methods based on four-dimensional Gaussian splashing mentioned above include:

[0016] 1. Fixed spatiotemporal feature fusion method. This method uses a uniform fusion method for the six spatiotemporal feature planes, which cannot adaptively adjust the contribution of each plane feature according to spatial location, temporal state and local dynamic change. In areas of rapid motion, non-rigid deformation and dense texture, insufficient feature expression is likely to occur, resulting in blurring and loss of detail in the reconstruction results in complex dynamic areas.

[0017] 2. The Gaussian rasterization strategy is disconnected from the error distribution. Method 1 mainly relies on a gradient-based densification strategy to increase Gaussian raster, which cannot directly perceive the reconstruction error distribution in the image space. It is difficult to selectively supplement the newly added Gaussian raster into the truly underrepresented high-error regions, resulting in insufficient local representation density in motion edges, complex texture regions, and interactive regions.

[0018] 3. Lack of effective constraints on opacity. This method does not impose constraints on the distribution and temporal variation of Gaussian opacity, which can easily leave a large amount of semi-transparent redundant Gaussian opacity in local areas, causing edge blurring; at the same time, the opacity change between adjacent time steps lacks smoothness, which can easily cause temporal jitter in the rendering results.

[0019] One existing dynamic reconstruction method based on spatiotemporal feature plane decomposition includes representative structures such as HexPlane and Grid4D. This type of method explicitly decomposes the four-dimensional spatiotemporal field into several two-dimensional feature planes to reduce the enormous storage and computational overhead of directly modeling four-dimensional voxels. Its core steps are as follows:

[0020] Spatiotemporal decomposition representation: The spatiotemporal field of the scene is decomposed into six two-dimensional feature planes, three of which are spatial planes (xy, xz, yz) used to characterize static geometric structures; and three are space-time planes (xt, yt, zt) used to characterize dynamic information that changes over time.

[0021] Feature interpolation and fusion: For any spatiotemporal sampling point, bilinear interpolation is performed on each plane to obtain local features. Then, the features of the six planes are fused into a unified spatiotemporal feature by element-wise multiplication or concatenation.

[0022] Decoding and rendering: The fused features are input into the decoding network to predict density, color or Gaussian deformation, and finally the image is synthesized through volume rendering or Gaussian splashing.

[0023] The disadvantages of the existing dynamic reconstruction methods based on spatiotemporal feature plane decomposition include:

[0024] 1. The planar feature fusion method lacks adaptability. When fusing six planar features, this method uses a fixed element-wise multiplication or concatenation method, which cannot distinguish the relative importance of spatial and temporal features based on the local dynamic characteristics of the current sampling position. Using the same fusion strategy in static and dynamic regions makes it difficult to balance geometric stability and dynamic expressiveness.

[0025] 2. Limited representation of complex local details. This method lacks a targeted enhancement mechanism for high-error regions, resulting in limited reconstruction quality in fast-moving edges, fine-grained textures, and complex interactive regions. Local details are easily smoothed, making it difficult to meet the requirements of high-fidelity volumetric videos. Summary of the Invention

[0026] The embodiments of the present invention provide a dynamic scene volumetric video reconstruction method based on feature group gating fusion to achieve effective reconstruction of dynamic scene images.

[0027] To achieve the above objectives, the present invention adopts the following technical solution.

[0028] A dynamic scene volumetric video reconstruction method based on feature group gating fusion includes:

[0029] A set of three-dimensional Gaussian elements is sampled in the 3D scene space. Based on the three-dimensional position and timestamp of each Gaussian element, multiple local features are interpolated from the six space-time planes of HexPlane. All local features of all Gaussian elements are divided into several feature groups according to their spatiotemporal relationship.

[0030] By using the local features of the current sampling location as input through a gating network, the fusion weights of each feature group are adaptively calculated. Based on the fusion weights of each feature group, the local features in each feature group are weighted and fused to obtain the fused spatiotemporal features.

[0031] The fused spatiotemporal features are input into the deformation decoder, which predicts the update amounts of the position, rotation, and scale of each Gaussian element at each time step. The update amounts of the position, rotation, and scale of each Gaussian element at each time step are applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space.

[0032] A set of deformed target Gaussian primitives in a 3D scene space is selected, and the set of target Gaussian primitives is rendered into a target view image through differentiable rasterization. The target view image is then used as a synthesized dynamic scene image.

[0033] Preferably, the method further includes:

[0034] The target view image is compared with the real image to obtain a pixel-level error map between the target view image and the real image. Based on the pixel-level error map between the target view image and the real image, a high error region mask in the target view image is obtained through threshold filtering and morphological processing. The high error region mask is projected back onto the 3D scene space along the camera ray of the corresponding pixel. New Gaussian primitives are added to the high error region in the 3D scene space with a low opacity initialization method.

[0035] The color reconstruction loss is the main component, which is combined with the depth consistency loss, spatial smoothing term, and opacity constraint term to form a joint optimization objective function. This joint optimization objective function is then used to optimize the attributes of all Gaussian units, HexPlane features, gating network, and deformation decoder parameters.

[0036] Preferably, the step of sampling a set of three-dimensional Gaussian primitives in the 3D scene space, interpolating multiple local features from the six space-time planes of HexPlane based on the three-dimensional position and timestamp of each Gaussian primitive, and dividing all local features of all Gaussian primitives into several feature groups according to spatiotemporal relationships, including:

[0037] In a standardized 3D scene space, a set of three-dimensional Gaussian primitives are sampled to obtain the position, shape, color, and transparency information of each Gaussian primitive. For each Gaussian primitive, its three-dimensional position (x, y, z) and timestamp t are used to interpolate the local features of multiple planes from the six space-time planes (xy, xz, yz, xt, yt, zt) of HexPlane. All local features of all Gaussian primitives are divided into several feature groups according to their spatiotemporal relationships.

[0038] Preferably, the step of using a gating network with local features at the current sampling location as input, adaptively calculating the fusion weights of each feature group, and weighting and fusing the local features in each feature group according to the fusion weights to obtain fused spatiotemporal features includes:

[0039] For a given Gaussian element's 3D position (x, y, z) and timestamp t, HexPlane samples the corresponding local features from six space-time planes, denoted as:

[0040]

[0041] in, Indicates the first Local features of each feature plane at the current sampling position;

[0042] A corresponding gating coefficient is introduced for each set of local features, which is defined as follows:

[0043] (1)

[0044] in, This indicates a lightweight gating network. For activation function, Indicates the first Weights of local features in a group;

[0045] We perform weighted fusion of local features in each feature group to obtain fused spatiotemporal features:

[0046] (2)

[0047] in, This represents the fused spatiotemporal features, which specifically include: local spatial geometry and appearance structure information provided by the spatial planes (xy, xz, yz), and dynamic deformation information that changes over time provided by the spatiotemporal planes (xt, yt, zt).

[0048] Preferably, the step of inputting the fused spatiotemporal features into the deformation decoder, the deformation decoder predicting the update amounts of the position, rotation, and scale of each Gaussian element at each time step, and applying the update amounts of the position, rotation, and scale of each Gaussian element at each time step to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space includes:

[0049] The spatiotemporal features are fused and input into the deformation decoder. The deformation decoder predicts the update amount of each Gaussian element at each time step. The position offset Δx, Δy, Δz, rotation update Δr, and scale update Δs of each Gaussian element are predicted by the distribution of the position prediction head, rotation prediction head, and scale prediction head in the deformation decoder.

[0050] The position, rotation, and scale updates of each Gaussian element at each time step are applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space.

[0051] Preferably, the step of comparing the target view image with the real image to obtain a pixel-level error map between the target view image and the real image, using the pixel-level error map between the target view image and the real image as a basis, obtaining a high-error region mask in the target view image through threshold filtering and morphological processing, projecting the high-error region mask back onto the 3D scene space along the camera ray of the corresponding pixel, and supplementing the high-error region in the 3D scene space with new Gaussian elements in a low-opacity initialization manner, includes;

[0052] In each training iteration, the target view image is compared with the real image to obtain a pixel-level error map between the target view image and the real image:

[0053] (3)

[0054] Where p represents the pixel position in the image plane, I(p) is the color value of the real image at pixel p, Î(p) is the pixel value rendered by the current model, and E(p) represents the pixel-level error map between the target view image and the real image at the corresponding position.

[0055] Based on the pixel-level error map E(p), thresholding and morphological processing are performed on the target viewpoint image to retain regions with errors higher than a set threshold, thus obtaining a high-error region mask in the target viewpoint image:

[0056] (4)

[0057] in, As the error screening threshold, It is a binary mask function;

[0058] When the error at a certain pixel position exceeds a threshold At this point, the location is considered a high-error region. After determining the high-error region, a new Gaussian element is generated in 3D space along the camera ray of the corresponding pixel. Let the camera center of the current viewpoint be o, and the direction of the unit ray corresponding to the pixel be d_p. Then, the camera ray originating from this pixel is represented as:

[0059] (5)

[0060] in, Let λ* represent the depth parameter along the ray direction. For pixels in the high-error region, candidate positions are selected in 3D space along the corresponding ray, and new Gaussian primitives are initialized at these positions to map the high-error region in image space to the corresponding position in 3D space. Let the sampling depth selected along the ray be λ*, then the center position of the newly added Gaussian primitive is represented as:

[0061] (6)

[0062] in, Let the three-dimensional center position of the newly generated Gaussian element be represented by the following: Let the scale and opacity of the newly added Gaussian element be respectively... and Then its initialization form is written as:

[0063] , (7)

[0064] in, and These represent the preset initial scale and initial opacity parameters, respectively.

[0065] Preferably, the method mainly uses color reconstruction loss, combined with depth consistency loss, spatial smoothing term, and opacity constraint term to form a joint optimization objective function. The joint optimization objective function is used to optimize the attributes of all Gaussian units, HexPlane features, gated network, and deformation decoder parameters, including:

[0066] In the dynamic Gaussian reconstruction process, an opacity constraint is introduced, let the first... The opacity of each Gaussian element is The corresponding color is The color of the target pixel is then represented as:

[0067] (8)

[0068] in, Indicates the final rendered color. Indicates the first The effective opacity of each Gaussian element in the current viewing direction. Indicates the transmission term;

[0069] Imposing a binarization tendency constraint on the opacity distribution of Gaussian elements, let the th element... The opacity of each Gaussian element is If ∈ [0,1], then the constraint term is defined as:

[0070] (9)

[0071] From equation (9), it can be seen that when When the constraint term is close to 0 or 1, the constraint term is smaller; when When the range is in the middle, the constraint terms are larger;

[0072] Introduce a smoothing constraint in the time dimension, let the... A high-ranking official in Time and The opacity at each moment is respectively and The time smoothing term is then expressed as:

[0073] (10)

[0074] Combining the binarization tendency constraint and the smoothing constraint, the final opacity constraint term is defined as:

[0075] (11)

[0076] in, and Let represent the weights of the binarization constraint and the temporal smoothing constraint, respectively. The opacity constraint, color reconstruction loss, depth consistency loss, and spatial smoothing term are combined to form the joint optimization objective function:

[0077] (12)

[0078] in, This indicates the weight of the opaque constraint in the total loss.

[0079] The joint optimization objective function is used to optimize the attributes of all Gaussian elements, HexPlane features, gated network and deformation decoder parameters to obtain the optimized attributes of Gaussian elements, HexPlane features, gated network and deformation decoder parameters.

[0080] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method of the present invention can simultaneously improve the problems of insufficient spatiotemporal feature representation and insufficient local sampling density in complex dynamic scenes, and suppress redundant Gaussians, thereby significantly improving the local details and overall quality of reconstruction. Experiments on multiple public datasets (N3D, D-NeRF, HyperNeRF) and a self-built complex hand-object interaction dataset (HIMoRe) have verified the effectiveness of the proposed method.

[0081] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a schematic diagram illustrating an application of the present invention in immersive content presentation in virtual reality / augmented reality.

[0084] Figure 2 This is a schematic diagram illustrating an application of the present invention in remote immersive conferencing and 3D video communication.

[0085] Figure 3 This is a schematic diagram illustrating the application of the present invention in free-viewpoint replay of sports events and film and television special effects;

[0086] Figure 4 This is a schematic diagram illustrating the differences between low-quality and high-quality dynamic reconstructions in the hand-object interaction area, provided as an embodiment of the present invention.

[0087] Figure 5 This is a schematic diagram illustrating the implementation principle of a dynamic scene volumetric video reconstruction method based on feature group gating fusion, provided in an embodiment of the present invention.

[0088] Figure 6 The flowchart illustrates a dynamic scene volumetric video reconstruction method based on feature group gating fusion, as provided in an embodiment of the present invention.

[0089] Figure 7 A schematic diagram of an error-guided Gaussian sampling process provided in an embodiment of the present invention;

[0090] Figure 8 This is a schematic diagram comparing the reconstruction effects of different algorithms on the N3D dataset, provided as an embodiment of the present invention.

[0091] Figure 9 This is a schematic diagram showing the reconstruction performance of different algorithms on the HIMoRe dataset, provided as an embodiment of the present invention.

[0092] Figure 10 A schematic diagram illustrating the visual comparison results of different methods on the D-NeRF dataset provided in this embodiment of the invention;

[0093] Figure 11 This invention provides a visual effect comparison diagram of whether or not error-guided sampling is introduced. Detailed Implementation

[0094] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0095] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0096] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0097] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0098] To improve the detail representation capability of volumetric video reconstruction in complex dynamic regions and alleviate the problems of detail blurring and structural loss caused by the disconnect between fixed feature fusion, denser strategies and error distribution, and redundant Gaussian sampling, this invention proposes a dynamic reconstruction method based on group-gated fusion and error-guided sampling (GG-4DGS). This method is based on the 4DGS dynamic Gaussian framework and mainly includes three mutually cooperating core modules:

[0099] First, a group-gated HexPlane feature fusion module is designed. This module divides the six spatial-temporal feature planes into several groups according to the spatiotemporal relationships they represent, and uses a lightweight gating network to adaptively calculate the fusion weights of each feature group based on the local features corresponding to the current sampling position. This allows for greater reliance on spatial structure features in static regions and enhanced temporal correlation features in dynamically changing regions, thereby improving the representation ability of complex motion regions.

[0100] Secondly, an error-guided Gaussian sampling strategy is proposed. During training, this strategy locates high-error regions that the model has not yet fully fitted based on the pixel-level error distribution between the current rendering result and the real image, and supplements new Gaussian units in three-dimensional space along the camera ray of the corresponding pixel, so that the limited model capacity is concentrated on the complex regions that are under-represented.

[0101] Finally, an opacity constraint is introduced. This constraint adjusts the Gaussian opacity from two aspects: the binarization tendency of the opacity distribution and temporal smoothing. It reduces the repeated superposition of semi-transparent redundant Gaussians in local regions, improves edge sharpness, and suppresses temporal jitter. At the same time, it allows the Gaussian elements newly added by error-guided sampling to participate in local reconstruction more smoothly.

[0102] This invention can be widely applied to various immersive and 3D vision applications that use volumetric video as the core data format. Typical application scenarios include:

[0103] (1) Immersive content presentation of virtual reality and augmented reality. This invention can reconstruct dynamic 3D content with freely switchable perspectives from multi-view videos, allowing VR / AR devices to enjoy immersive viewing and interaction. Figure 1 This is a schematic diagram illustrating an application of the present invention in immersive content presentation in virtual reality / augmented reality.

[0104] (2) Remote immersive conferencing and 3D video communication. This invention can reconstruct participants into dynamic 3D models in real time, and then transmit them over the network to present them in 3D form at a remote location, achieving a remote immersive experience with a sense of space. Figure 2 This is a schematic diagram illustrating an application of the present invention in remote immersive conferencing and 3D video communication.

[0105] (3) Free-viewpoint playback of sports events and production of film and television special effects. This invention can dynamically reconstruct motion scenes shot around the camera, support free-viewpoint playback and bullet time effects, and improve the flexibility and expressiveness of content production. Figure 3 This is a schematic diagram illustrating the application of the present invention in free-viewpoint replay of sports events and film and television special effects.

[0106] In the aforementioned applications, the reconstruction quality of complex dynamic regions (such as the edges of moving human bodies and areas of hand-object interaction) directly determines the realism of the immersive experience. This invention, through targeted enhancement of details in complex dynamic regions, effectively improves the reconstruction results in these areas. Figure 4 This is a schematic diagram illustrating the differences between low-quality and high-quality dynamic reconstructions in the hand-object interaction area, provided as an embodiment of the present invention.

[0107] The implementation principle diagram of a dynamic scene volumetric video reconstruction method based on feature group gating fusion provided in this embodiment of the invention is shown below. Figure 5 As shown, this invention introduces three core modules on the basis of the 4DGS dynamic Gaussian framework: a grouped gated HexPlane feature fusion module, an error-guided Gaussian sampling strategy, and an opacity constraint, to form a complete dynamic scene volumetric video reconstruction method, GG-4DGS.

[0108] The processing flow of a dynamic scene volumetric video reconstruction method based on feature group gating fusion provided in this embodiment of the invention is as follows: Figure 6 As shown, the processing steps include the following:

[0109] Step S10: Standardized space Gaussian initialization and spatiotemporal feature query;

[0110] A Gaussian primitive typically refers to the basic unit used to represent a 3D scene in 3D Gaussian sputtering (3DGS) technology. It is essentially a 3D Gaussian ellipsoid with specific properties. It is not a solid sphere, but a semi-transparent ellipsoid. The entire 3D world is pieced together from millions of such primitives, and each Gaussian primitive carries information such as position, shape, color, and transparency.

[0111] HexPlane is a dynamic 3D scene representation method designed to address computational bottlenecks and memory consumption issues in dynamic new view compositing. HexPlane decomposes complex 4D spatiotemporal data (x, y, z, t) into six lightweight 2D planes, significantly reducing memory requirements and accelerating computation.

[0112] In a standard 3D scene space, a set of three-dimensional Gaussian elements are sampled. For each Gaussian element, local features of multiple planes are obtained by interpolating its three-dimensional position (x, y, z) and timestamp t from the six space-time planes (xy, xz, yz, xt, yt, zt) of HexPlane. All local features are divided into several feature groups according to their spatiotemporal relationships.

[0113] Step S20: Grouped gated HexPlane feature fusion;

[0114] A gating mechanism is a mechanism that uses a learnable gating network to generate weight coefficients and adaptively weight and fuse different feature channels or feature groups. The gating weights are typically constrained between 0 and 1, allowing the model to dynamically adjust the contribution of each feature based on the input content.

[0115] By using a lightweight gating network with local features at the current sampling location as input, the fusion weights of each feature group are adaptively calculated. Then, the contribution ratio of each feature group is controlled by the gating weights to complete the fusion. The local features in each feature group are weighted and fused to obtain more targeted spatiotemporal features. This enables adaptive adjustment of the importance of features in different regions and at different times.

[0116] The fused spatiotemporal features specifically include: local spatial geometry and appearance structure information provided by the spatial plane (xy, xz, yz), and dynamic deformation information that changes over time provided by the spatiotemporal plane (xt, yt, zt); after adaptive weighted fusion with gated weights, a feature vector representing the local spatiotemporal state of the Gaussian element at a given time is obtained.

[0117] Step S30: Deformation prediction and Gaussian transform based on fused spatiotemporal features;

[0118] The spatiotemporal features are fused and input into the deformation decoder, which predicts the position, rotation, and scale updates of each Gaussian element at each time step. The position offset Δx, Δy, Δz, rotation update Δr, and scale update Δs of each Gaussian element are predicted using the position prediction head, rotation prediction head, and scale prediction head in the deformation decoder.

[0119] The position, rotation, and scale updates of each Gaussian element at each time step are applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space.

[0120] The deformation decoder consists of a multi-resolution HexPlane feature field, a temporal coding network (TimeNet), and a deformation regression network (DeformNet). The HexPlane feature field contains six learnable two-dimensional feature planes. The temporal coding network maps timestamps to temporal features. DeformNet regresses the time-by-time updates of position, rotation, and scale from the fused mesh features using a multilayer perceptron and attribute prediction heads. In subsequent optimization, the parameters optimized for the deformation decoder refer to the learnable weights and biases of DeformNet and the temporal coding network (i.e., the weights and biases of each linear layer), which are different learnable parameter objects from the "HexPlane features" listed in the claims.

[0121] Step S40: Micro-rasterization rendering;

[0122] A set of deformed target Gaussian elements is selected in the 3D scene space, and the set of target Gaussian elements is rendered into a target view image through differentiable rasterization. This target view image is the synthesized dynamic scene image.

[0123] By comparing the target view image with the real image, a pixel-level error map between the target view image and the real image is obtained.

[0124] Step S50: Error-guided Gaussian sampling;

[0125] Based on the pixel-level error map between the target view image and the real image, a high-error region mask in the target view image is obtained through thresholding and morphological processing. The high-error region mask is projected back into the 3D scene space along the camera ray of the corresponding pixel. New Gaussian primitives are added to the high-error region in the 3D scene space with low opacity initialization, and a periodic triggering mechanism is used to participate in the optimization.

[0126] Step S60: Joint loss optimization (including opacity constraints).

[0127] The joint optimization objective is based on color reconstruction loss, combined with depth consistency loss, spatial smoothing term, and opacity constraint term proposed in this invention. This objective performs end-to-end optimization on all Gaussian properties, HexPlane features, gating network, and deformation decoder.

[0128] The newly added Gaussian sigma joins the existing Gaussian sigma in subsequent optimizations, and an opacity constraint is added to the total loss to suppress redundant Gaussian sigma. Through the synergistic effect of these modules, the quality of local detail representation and overall reconstruction in complex dynamic scenes is ultimately enhanced.

[0129] After optimization, given any target viewpoint and time, the corresponding dynamic scene image can be synthesized in real time by sequentially executing spatiotemporal feature query, grouped gating fusion, deformation prediction and differentiable rasterization, which supports volumetric video playback from free viewpoints.

[0130] The above step S20 specifically includes:

[0131] Group-gated HexPlane Feature Fusion Module: This invention introduces a group-gated fusion module based on the original HexPlane feature structure. This module divides the six spatial-temporal feature planes into several groups according to their expressed spatiotemporal relationships, and adaptively calculates the fusion weights of different feature groups based on the local features corresponding to the current sampling position. Compared with direct concatenation or fixed weighting fusion methods, group-gated fusion can dynamically adjust the contribution of different planar features in the final representation according to spatial location, temporal state, and the degree of local dynamic changes, thereby enhancing the model's ability to express complex motion regions and local detail regions.

[0132] In the spatiotemporal representation of 4DGS, HexPlane jointly encodes spatial location and temporal variation information through multiple two-dimensional planes. For a given sampling point (x, y, z, t), the model can sample corresponding features from six space-time planes, denoted as:

[0133]

[0134] in, Indicates the first Each feature plane represents a local feature vector at the current sampling position. Since the spatiotemporal relationships described by different planes are not entirely consistent, their roles in dynamic scene reconstruction also differ. Some planar features focus more on characterizing local spatial geometry, while others focus more on reflecting dynamic information that changes over time. If these features are fused in the same way at all locations, it is difficult to fully reflect the relative importance of different planar features in different regions, especially in areas of rapid motion, non-rigid deformation, or dense texture details, thus affecting the quality of the final reconstruction result. Therefore, this invention introduces a corresponding gating coefficient for each set of features, defined as:

[0135] (1)

[0136] in, This indicates a lightweight gating network. For activation function, Indicates the first Weights of group features. The gating network takes the local features at the current sampling position as input and outputs the fusion weights of the corresponding groups, enabling the model to adaptively adjust the contribution of each group feature according to different positions and time states.

[0137] To avoid excessively large or small weights for certain feature groups during training, this invention constrains the gating weights through an activation function, keeping them within a stable range. This design allows the model to rely more on spatial structure features in static regions and enhance the role of time-related features in dynamically changing regions, thereby improving the specificity of spatiotemporal feature representation. After obtaining the gating weights for each group, the final fused feature representation is as follows:

[0138] (2)

[0139] in, The fused spatiotemporal feature representation is shown. As can be seen from Equation (2), the final feature is controlled by gating weights to determine the contribution ratio of each group of features, rather than directly averaging the features of each plane. Based on this design, the model can adaptively weight different groups of features during the feature fusion stage, thereby achieving a better balance between static geometric stability and dynamic detail representation, and significantly enhancing the representation ability of complex motion regions.

[0140] Step S50 specifically includes: This invention proposes a Gaussian sampling strategy based on image error, used to supplement Gaussian primitives corresponding to high-error regions during training. This strategy locates regions where the model has not yet adequately fitted the model based on the difference between the currently rendered target image and the real image, and supplements new Gaussian primitives at the corresponding 3D positions. The basic process of this strategy is as follows: First, calculate the pixel error between the currently rendered target image and the real image; then, filter high-error regions according to the error distribution; finally, initialize new Gaussian primitives in 3D space along the camera ray of the corresponding pixel and add them to the current Gaussian set for subsequent optimization. Figure 7 This is a schematic diagram of an error-guided Gaussian sampling process provided in an embodiment of the present invention.

[0141] Specifically, in each training iteration, the model first renders the image from the target viewpoint based on the current Gaussian set, compares the rendered result with the real image, calculates the pixel-level error, and obtains an error map:

[0142] (3)

[0143] Where p represents the pixel position in the image plane, I(p) is the color value of the real image at pixel p, Î(p) is the pixel value rendered by the current model, and E(p) represents the reconstruction error at the corresponding position. This error map intuitively reflects the regions in the image space where the current model has not yet fully fitted the image. Positions with larger errors usually correspond to areas where the model's representation of local geometry, surface texture, or dynamic changes is still insufficient. Therefore, the error map can serve as a basis for selecting subsequent Gaussian unit supplementation positions, allowing newly added Gaussian units to more effectively target underrepresented local regions.

[0144] To avoid excessively dispersed error responses that lead to invalid generation of new Gaussian elements in noisy regions, this invention further performs threshold filtering on the obtained error map, retaining only regions with errors exceeding a set threshold. The high-error region mask can be represented as:

[0145] (4)

[0146] Where τ is the error screening threshold and M(p) is the binary mask function. When the error at a certain pixel location is higher than the threshold, that location is considered a high-error region; otherwise, it is not included in subsequent sampling. In this way, the model can limit the generation range of new Gaussian units to regions with significant errors, thereby avoiding the repeated addition of Gaussian units in regions that have already been well fitted.

[0147] Considering the potential for discrete noise responses in the error map, this invention further performs morphological processing on the high-error mask to remove small, isolated regions and retain spatially more coherent high-error regions. The processed mask is used to determine candidate locations for subsequently added Gaussian elements, enabling more stable localization of areas where the model representation is insufficient and providing more reliable regional constraints for subsequent 3D sampling.

[0148] After identifying the high-error region, this invention generates new Gaussian elements in three-dimensional space along the camera ray corresponding to the pixel. Let the camera center of the current viewpoint be o, and the direction of the unit ray corresponding to the pixel be d_p. Then, the camera ray originating from that pixel can be represented as:

[0149] (5)

[0150] Where λ represents the depth parameter along the ray direction. For pixels in the high-error region, this invention selects candidate positions in three-dimensional space along the corresponding ray and initializes new Gaussian primitives at these positions to map the high-error region in image space to the corresponding position in three-dimensional space. Let the sampling depth selected along the ray be λ*, then the center position of the newly added Gaussian primitive can be expressed as:

[0151] (6)

[0152] in, This represents the three-dimensional center position of the newly generated Gaussian element. For The specific selection can be determined by combining the depth distribution of existing Gaussian elements in the current scene, rendering depth information, or local neighborhood geometric priors.

[0153] After initializing the position, the scale, opacity, and other properties of the newly added Gaussian elements also need to be set. Let the scale and opacity of the newly added Gaussian elements be respectively... and Then its initialization form can be written as:

[0154] , (7)

[0155] in, and These represent the preset initial scale and initial opacity parameters, respectively. Initializing with low opacity ensures that newly added high-order primitives do not immediately dominate the rendering of local areas after being added to the scene; instead, their contribution gradually increases with subsequent parameter optimizations. This progressive participation mechanism helps maintain the overall stability of the original scene structure while supplementing local details.

[0156] To avoid frequent interruptions to the model's normal optimization during the sampling process, this invention does not perform error-guided sampling after each iteration, but rather triggers this operation periodically at specific stages of the training process. At this point, the model has completed preliminary geometric fitting, and the newly added Gaussian units are mainly used to supplement the representation of local high-error regions. In summary, the overall process of error-guided sampling includes five steps: error map calculation, high-error region screening, camera ray re-projection, initialization of newly added Gaussian units, and periodic sampling updates.

[0157] Step S60 specifically includes: In the dynamic Gaussian reconstruction process, the opacity parameter directly affects the role of Gaussian primitives in rendering. If there are many Gaussian primitives with intermediate opacity in a local area, they are prone to overlapping, thus affecting edge sharpness and result stability. From the rendering process perspective, Gaussian primitives with intermediate opacity continuously participate in color accumulation; if their number is too large, the color contribution source in a local area may become dispersed. Especially in areas with significant motion changes, if the opacity changes too much over time, it can easily cause discontinuities in the rendering results at adjacent time points. Therefore, this invention introduces opacity constraints during the training process to adjust the opacity distribution of Gaussian primitives.

[0158] Opacity Constraint: A regularization constraint imposed on the distribution and temporal variation of Gaussian opacity during training. Its purpose is to encourage opacity to distribute towards either 0 or 1, while maintaining smoothness between adjacent time steps, thereby reducing the repeated superposition of redundant semi-transparent Gaussians and improving edge sharpness and temporal stability.

[0159] In the standard Gaussian rendering process, let the... The opacity of each Gaussian element is The corresponding color is Then the color of the target pixel can be represented as:

[0160] (8)

[0161] in, Indicates the final rendered color. Indicates the first The effective opacity of each Gaussian element in the current viewing direction. Let represent the transmission term. As shown in equation (8), opacity directly affects the degree of participation of the current Gaussian element in color accumulation. Based on the above analysis, this invention constrains it from two aspects: opacity distribution and time variation.

[0162] First, this invention imposes a binarization tendency constraint on the opacity distribution of Gaussian elements. Let the opacity of the i-th Gaussian element be... If ∈ [0,1], then the constraint term is defined as:

[0163] (9)

[0164] From equation (9), it can be seen that when When the constraint term is close to 0 or 1, the constraint term is smaller; when When the value is in the middle range, the constraint term is larger. Therefore, this constraint causes the opacity to tend to distribute towards both ends, thereby reducing the repeated superposition of semi-transparent Gaussian units in local areas and improving edge sharpness.

[0165] Secondly, considering that the opacity changes of the same Gaussian element at adjacent time points can also affect the continuity of the results, this invention further introduces a smoothing constraint in the time dimension. Let the first... A high-ranking official in Time and The opacity at each moment is respectively and Then the time smoothing term can be expressed as:

[0166] (10)

[0167] This constraint limits excessive changes in the opacity of the same Gaussian element between adjacent time steps, thereby reducing temporal fluctuations and suppressing timing jitter. Combining the above two constraints, this invention defines the final opacity constraint as follows:

[0168] (11)

[0169] in, and These represent the weights of the binarization constraint and the temporal smoothing constraint, respectively. By adjusting these weights, the relative effects of the opacity distribution and temporal variation constraints during training can be controlled. In the overall training, this invention incorporates the opacity constraint into the total loss function, which, together with the color reconstruction loss, depth consistency loss, and spatial smoothing term, constitutes a joint optimization objective. The final total loss function is written as:

[0170] (12)

[0171] in, This represents the weight of the opacity constraint in the total loss. Adding this term imposes an additional constraint on the opacity parameter during training, thereby reducing the impact of long-term retention of redundant Gaussian elements in local regions. The opacity constraint also plays a moderating role for newly added Gaussian elements in error-guided sampling: these elements typically have low opacity during initialization and are subsequently updated gradually based on training errors. Adding the opacity constraint weakens the excessive influence of newly added Gaussian elements on local results in the early stages of training, allowing them to participate more smoothly in the reconstruction of local regions.

[0172] To verify the effectiveness of the proposed method in dynamic scene reconstruction, this invention conducts comparative experiments on multiple public datasets (N3D, D-NeRF, HyperNeRF) and a self-built HIMoRe dataset, and analyzes the results through quantitative indicators and visualization. At the same time, ablation experiments are conducted on the group gating module and the error-guided sampling strategy.

[0173] This invention evaluates the proposed method on publicly available Neural 3D Dataset (N3D), D-NeRF, HyperNeRF datasets, and a self-built HIMoRe dataset. The model implementation is based on an extension of the publicly available 4DGS code framework. While maintaining the original training process, a grouped gated feature fusion module and an error-guided sampling strategy are introduced. The feature plane dimension is set to 32, the feature compression ratio is set to 16, the optimizer is Adam, and the initial learning rate is set to... Except for the module proposed in this invention, all other training parameters are consistent with the original 4DGS method to ensure the fairness of the comparative experiment.

[0174] In the error-guided sampling process, a pixel-level error map is first calculated based on the current rendering result and the real image. A threshold of 0.2 is set on the normalized error map to extract high-error regions. Subsequently, a 3×3 morphological operation is performed on the error mask to remove isolated noise regions and obtain more stable sampling regions. New Gaussian elements are generated along the camera ray within these high-error regions, and their spatial position and attribute parameters are initialized.

[0175] All experiments were conducted on the same hardware environment, with consistent settings maintained during training and testing. Model performance was evaluated using three commonly used metrics: PSNR, SSIM, and LPIPS. PSNR and SSIM measure pixel-level reconstruction accuracy and structural consistency, while LPIPS assesses visual differences at the perceptual level. The final result was calculated by averaging all test frames as the quantitative metric.

[0176] Analysis of Experimental Results on the N3D Dataset: On the N3D dataset, this invention compares GG-4DGS with the original 4DGS and other representative dynamic reconstruction methods to evaluate the impact of the grouped gating fusion module and error-guided sampling strategy on the reconstruction quality of multi-view dynamic scenes. Table 1 shows the quantitative results of different methods on the N3D dataset. GG-4DGS achieved PSNR, SSIM, and LPIPS of 32.12 dB, 0.984, and 0.040, respectively. Among them, LPIPS achieved the best result in the table, indicating that the method of this invention has a more significant improvement in the recovery of local details at the perception level.

[0177] Table 1. Quantitative comparison results of GG-4DGS and different methods on the N3D dataset.

[0178] StreamRF NeurIPS 2022 28.26 − − NeRFPlayer TVCG'2023 30.69 0.966 0.111 HyperReel CVPR'2023 31.10 0.964 0.096 K-Planes CVPR'2023 31.63 0.982 0.090 HexPlane CVPR'2023 31.71 − 0.075 MixVoxels-L ICCV'2023 31.34 0.983 0.096 Dynamic3DGS 3DV'2024 30.67 0.981 0.099 4DGS CVPR'2024 31.02 0.935 0.150 Grid4D NeurIPS 2024 31.49 0.936 0.147 SpacetimeGS CVPR'2024 32.05 0.969 0.044 DeGauss ICCV'2025 31.56 0.947 0.047 Swift4D ICLR'2025 32.23 0.986 0.072 GG-4DGS (This invention) — 32.12 0.984 0.040

[0179] Note: In the table, data marked in bold represents the best performance, and data marked in underline represents the second-best performance.

[0180] Compared to the baseline method 4DGS, GG-4DGS improves PSNR from 31.02 dB to 32.12 dB, an increase of 1.10 dB; SSIM improves from 0.935 to 0.984, indicating a significant enhancement in structural consistency of the reconstruction results; and LPIPS decreases from 0.150 to 0.040, a reduction of 0.110. The simultaneous improvement in these three metrics demonstrates that the method of this invention does not only improve single-pixel errors, but also improves dynamic scene reconstruction results at both the structural preservation and perceptual quality levels. In particular, the largest decrease in LPIPS indicates that GG-4DGS recovers local textures, edge contours, and high-frequency detail areas more fully.

[0181] Considering the characteristics of the N3D dataset, which contains realistic multi-view dynamic scenes with continuous human movements, local texture changes, and moving edge regions, the original 4DGS uses a fixed spatiotemporal feature fusion method, which can easily lead to insufficient feature representation in areas with rapid local motion changes or complex textures. GG-4DGS adaptively weights different spatiotemporal feature planes through a grouped gating fusion module, enabling the model to adjust the contribution of different feature groups according to the spatial location and temporal state of the current sampling point. Simultaneously, error-guided sampling supplements Gaussian primitives in high-error regions based on the rendering error distribution, resulting in denser explicit representations of edges and complex textures. Therefore, the improvements in GG-4DGS mainly come from two aspects: firstly, improved spatiotemporal feature discrimination capability in complex regions; and secondly, improved Gaussian primitive coverage in local high-error regions.

[0182] To more intuitively compare the visual differences between different methods, Figure 8 This is a schematic diagram showing the reconstruction effects of different algorithms on the N3D dataset, as provided in an embodiment of the present invention. Figure 8 New perspective reconstruction results are presented on the N3D dataset. Figure 8 As can be seen, all methods can recover the main structure and overall appearance of the scene, but there are differences in the clarity of local dynamic areas. GG-4DGS retains more complete boundary information in the hand outline, finger gaps, tool edges, and flame jet areas, and the local texture transitions are more continuous. In contrast, although 4DGS can reconstruct the general content of the scene, it is more prone to boundary blurring and texture smoothing in the areas where the hand and tool contact each other, flame edges, and local rapidly changing areas. The method of this invention adds Gaussian units to these high-error areas through error-guided sampling, so that the local areas have a higher representation density, thereby alleviating the problems of flame boundary blurring and hand outline discontinuity.

[0183] (3) Analysis of Experimental Results on the HIMoRe Dataset: To further verify the reconstruction capability of GG-4DGS in complex hand interaction scenes, this invention conducted experiments on the self-built HIMoRe dataset. This dataset contains fine-grained finger movements, frequent hand-object occlusion, object contact, and local morphological changes. The dynamic process is more complex than that of N3D, thus better reflecting the model's ability to model local interaction areas. Table 2 shows the quantitative comparison results of different methods on the HIMoRe dataset.

[0184] Table 2. Quantitative comparison results of GG-4DGS and different methods on the HIMoRe dataset.

[0185] K-Planes CVPR'2023 26.74 0.860 0.214 HexPlane CVPR'2023 26.82 0.863 0.209 Grid4D NeurIPS 2024 26.96 0.869 0.199 4DGS CVPR'2024 26.90 0.866 0.203 DeGauss ICCV'2025 27.14 0.875 0.191 Swift4D ICLR'2025 27.71 0.887 0.174 GG-4DGS (This invention) — 27.79 0.884 0.178

[0186] Note: In the table, data marked in bold represents the best performance, and data marked in underline represents the second-best performance.

[0187] On the HIMoRe dataset, GG-4DGS achieves PSNR, SSIM, and LPIPS of 27.79 dB, 0.884, and 0.178, respectively. Among these, PSNR is the highest result in the table, exceeding Swift4D by 0.08 dB, DeGauss by 0.65 dB, and the original 4DGS by 0.89 dB, indicating that the proposed method can reduce overall pixel-level reconstruction error in complex hand interaction scenes. Regarding SSIM, GG-4DGS achieves 0.884, higher than 4DGS's 0.866 and DeGauss's 0.875, demonstrating that the proposed method better maintains the structural consistency of hand contours, object boundaries, and interaction areas. In terms of LPIPS, GG-4DGS achieves 0.178, lower than 4DGS's 0.203 and DeGauss's 0.191, indicating that its perceptual quality is close to the optimal method.

[0188] Compared to 4DGS, GG-4DGS improves all three metrics: PSNR by 0.89 dB, SSIM by 0.018, and LPIPS by 0.025. Since the main challenges in the HIMoRe dataset are concentrated on hand edges, object contact surfaces, and local occlusion regions, these improvements demonstrate that the method of this invention can more effectively handle local errors in complex interactive regions. In such scenarios, 4DGS typically requires using a unified spatiotemporal feature representation to simultaneously fit multiple significantly different local motion processes, easily leading to blurring at rapid finger movements and occlusion boundaries. GG-4DGS enhances the selective representation between different feature groups through a group gating mechanism and supplements high-error regions with Gaussian primitives through error-guided sampling, thus making it more suitable for handling frequently changing local dynamic regions in HIMoRe.

[0189] Figure 9 This is a schematic diagram showing the reconstruction effects of different algorithms on the HIMoRe dataset, as provided in an embodiment of the present invention. Figure 9 A visual comparison of the different methods on the HIMoRe dataset is presented. Figure 9As can be seen, GG-4DGS can recover the main structure of the scene, including the hand outline, the main shape of the manipulated object, and the spatial relationship of the interaction area. Further observation of local areas reveals that GG-4DGS exhibits better outline continuity at the contact points between the hand and the object, tool edges, and in structurally complex areas, with no obvious breaks in local edges. In areas of object surface texture and local deformation, GG-4DGS also retains more detailed information, making the reconstruction results closer to real-world observations. In contrast, 4DGS is prone to blurred boundaries and unclear finger outlines in areas of rapid hand movement, and it tends to blend the hand and object boundaries in areas where the hand occludes objects, resulting in incomplete local structures.

[0190] Combining quantitative results with visual comparisons reveals that GG-4DGS demonstrates a more significant advantage on the HIMoRe dataset than on N3D. This is because HIMoRe contains numerous high-error regions, such as finger edges, hand-object contact surfaces, occlusion transition areas, and local deformation regions. These areas are precisely where error-guided sampling plays a crucial role. Furthermore, the grouped gating fusion module can adaptively adjust the contributions of spatial and temporal features based on local dynamic changes, enabling the model to achieve stronger feature representation capabilities in complex interaction regions. Therefore, GG-4DGS not only improves the overall PSNR on HIMoRe but also enhances local visual quality.

[0191] (4) Experimental Results Analysis on D-NeRF and HyperNeRF Datasets: In addition to the multi-view real-scene dataset, this invention also conducted experiments on the synthetic dynamic scene dataset D-NeRF and the single-view non-rigid scene dataset HyperNeRF to further verify the applicability of GG-4DGS in different types of dynamic scenes. Table 3 shows the quantitative results of different methods on the D-NeRF dataset. The PSNR, SSIM, and LPIPS of GG-4DGS reached 34.15 dB, 0.98, and 0.02, respectively. Compared with the original 4DGS, the PSNR increased from 33.30 dB to 34.15 dB, an increase of 0.85 dB, indicating that the method of this invention can also reduce pixel errors and improve perception quality in synthetic dynamic scenes.

[0192] Table 3. Quantitative comparison results of GG-4DGS and different methods on the D-NeRF dataset.

[0193] TiNeuVox SIGGRAPH'2022 32.67 0.97 0.07 K-Planes CVPR'2023 31.61 0.97 − HexPlane CVPR'2023 31.04 0.97 0.04 3DGS SIGGRAPH'2023 23.19 0.93 0.08 4DGS CVPR'2024 33.30 0.98 0.03 4DGS-v2 ICLR'2024 34.09 0.98 0.02 SC-GS CVPR'2024 33.80 0.98 0.02 Grid4D NeurIPS 2024 34.20 0.99 0.01 GG-4DGS (This invention) — 34.15 0.98 0.02

[0194] Note: In the table, data marked in bold represents the best performance, and data marked in underline represents the second-best performance.

[0195] Considering the characteristics of the D-NeRF dataset, the dynamic targets and backgrounds are clearly separated, the motion patterns are relatively regular, and there are few local occlusions and complex interactions. Therefore, baseline methods such as 4DGS can already achieve high reconstruction quality. In this case, the improvement of GG-4DGS mainly lies in further correcting local error regions, rather than significantly changing the overall structural modeling capability. The grouped gating fusion module can optimize the combination of spatiotemporal features, and error-guided sampling can supplement regions with large local errors. However, since the number and complexity of high-error regions in D-NeRF are lower than in real interactive datasets, the advantages of the method in this invention are relatively mild on this dataset. Figure 10 Visual comparisons of different methods on the D-NeRF dataset are presented.

[0196] To further evaluate the performance of GG-4DGS in non-rigid dynamic scenes, experiments were also conducted on the HyperNeRF dataset. Table 4 presents the quantitative comparison results of different methods on the HyperNeRF dataset. GG-4DGS achieved PSNR of 25.38 dB and MS-SSIM of 0.852, compared to 4DGS, where PSNR increased from 25.20 dB to 25.38 dB and MS-SSIM increased from 0.845 to 0.852. This demonstrates that the method of this invention can still improve structural consistency and pixel accuracy in non-rigid scenes.

[0197] Table 4. Quantitative comparison results of GG-4DGS and different methods on the HyperNeRF dataset.

[0198] Nerfies ICCV'2021 22.2 0.803 HyperNeRF ACM TOG'2021 22.4 0.814 TiNeuVox SIGGRAPH'2022 24.3 0.836 3DGS SIGGRAPH'2023 19.7 0.680 4DGS CVPR'2024 25.2 0.845 Grid4D ICCV'2025 25.50 0.856 GG-4DGS (This invention) 2025 25.38 0.852

[0199] Note: In the table, data marked in bold represents the best performance, and data marked in underline represents the second-best performance.

[0200] The improvement of GG-4DGS on HyperNeRF is smaller than that of N3D and HIMoRe, mainly because the core challenge of HyperNeRF is complex non-rigid deformation and topological changes. These scenarios require the model to accurately establish the deformation correspondence between different time periods. The main improvements of this invention focus on feature fusion methods and local sampling density, without changing the overall deformation modeling framework of the original 4DGS. Therefore, grouped gated fusion and error-guided sampling can improve the expressive power of local regions, but their effectiveness is relatively limited for cross-time correspondence errors caused by large non-rigid deformations and topological changes. In summary, the advantages of GG-4DGS are more suitable for dynamic scenes with sufficient multi-view observation, complex local details, and frequent hand-object interactions.

[0201] (5) Ablation experiment analysis

[0202] To further analyze the impact of the grouped gating feature fusion module and the error-guided sampling strategy on the overall performance, ablation experiments were conducted on the N3D dataset. The experiments included introducing the grouped gating module alone, introducing the error-guided sampling module alone, and enabling both modules simultaneously, while keeping the original 4DGS structure unchanged, and comparing their reconstruction performance. The ablation results are shown in Table 5.

[0203] Table 5 Module Ablation Experiment Results

[0204] Introducing a gating module 31.46 0.965 0.062 Introducing guided sampling 31.38 0.963 0.066 The method of this invention (combination of both) 32.12 0.984 0.040

[0205] Note: Data marked in bold represents best performance.

[0206] As shown in Table 5, when only the group gating module is introduced, the model's PSNR, SSIM, and LPIPS are 31.46 dB, 0.965, and 0.062, respectively. This result indicates that group gating fusion can improve the feature representation capability of dynamic regions by adaptively adjusting the contributions of different spatial-temporal feature groups. Compared to fixed fusion methods, the gating mechanism enables the model to select more effective feature combinations under different spatial locations and temporal states, thus improving both structural consistency and perceptual quality.

[0207] When only the error-guided sampling module is introduced, the model's PSNR, SSIM, and LPIPS are 31.38 dB, 0.963, and 0.066, respectively. This result indicates that error-guided sampling can improve the representation density of local regions by supplementing high-error regions with Gaussian primitives, thereby improving the reconstruction results of edge and textured complex regions. However, compared to introducing the grouping gating module alone, the three metrics for introducing error-guided sampling alone are slightly lower, indicating that without a more effective feature fusion method, simply increasing the number of Gaussian primitives cannot fully solve the feature representation problem of dynamic regions. In other words, the sampling strategy can supplement local representation capacity, but further enhancement of spatiotemporal feature modeling capabilities is still needed to guide the addition of Gaussian primitives for effective optimization.

[0208] When both the group gating module and the error-guided sampling module are introduced simultaneously, the PSNR, SSIM, and LPIPS of GG-4DGS reach 32.12 dB, 0.984, and 0.040, respectively, significantly outperforming the results of using either module alone. Compared to introducing only the gating module, the complete method improves PSNR by 0.66 dB, SSIM by 0.019, and LPIPS by 0.022; compared to introducing only error-guided sampling, the complete method improves PSNR by 0.74 dB, SSIM by 0.021, and LPIPS by 0.026. This indicates a complementary relationship between the two modules: the group gating module enhances the spatiotemporal feature representation of complex dynamic regions, while error-guided sampling increases the Gaussian primitive coverage in high-error regions. The combination of the two modules simultaneously addresses both insufficient feature representation and insufficient local sampling.

[0209] Figure 11 This invention provides a visual effect comparison diagram of whether or not error-guided sampling is introduced. Without this strategy, the model's representation ability in local high-error regions is insufficient, especially in the watermelon-cutting scene of the HIMoRe dataset. The area where the hand and fruit are in contact simultaneously experiences rapid movement, occlusion changes, and boundary shape alterations. Relying solely on the initial Gaussian distribution, this area is prone to incomplete edges and missing local structures. After introducing error-guided sampling, the model supplements new Gaussian units in the hand edges, fruit contact surfaces, and locally occluded areas based on the error distribution between the rendered result and the real image. This allows these areas to obtain a more complete explicit representation, resulting in a more continuous outline of the hand area and a more complete structure of the fruit boundary and contact area.

[0210] In summary, the technical solution of this invention brings beneficial effects.

[0211] 1. Significantly improves the detail recovery quality of complex dynamic regions. The group-gated HexPlane feature fusion module proposed in this invention can adaptively adjust the fusion weights of the six spatial-temporal feature planes according to spatial location, temporal state, and degree of local dynamic change. This allows the model to rely more on spatial structural features in static regions and enhance temporal related features in dynamically changing regions. This effectively solves the problem of insufficient feature representation of complex moving regions caused by the fixed fusion method in existing methods, and achieves stronger representation capabilities in fast-moving edges, non-rigid deformation, and texture-dense regions.

[0212] 2. Achieving precise matching between Gaussian primitive spatial distribution and error distribution. The error-guided Gaussian sampling strategy proposed in this invention calculates the pixel-level error between the rendered image and the real image, and locates high-error regions through thresholding and morphological processing. Then, new Gaussian primitives are added in three-dimensional space along the camera ray, so that the limited model capacity is concentrated on the truly insufficient complex regions, effectively improving the problem of insufficient local representation density in motion edges, complex texture regions, and hand-object interaction regions.

[0213] 3. Effectively suppresses redundant Gaussian elements and improves temporal stability. The opacity constraint proposed in this invention adjusts the Gaussian opacity from two aspects: binarization tendency and temporal smoothing. This causes the opacity to distribute towards the two extremes close to 0 or 1, reducing the repeated superposition of semi-transparent redundant Gaussian elements in local regions and improving edge sharpness. At the same time, it restricts the abrupt change in opacity of the same Gaussian element at adjacent time steps, suppressing temporal jitter in the rendering results. This constraint also allows newly added Gaussian elements from error-guided sampling to participate in local reconstruction more smoothly, maintaining the stability of the overall scene structure.

[0214] 4. Three modules work synergistically to enhance overall performance. Ablation experiments show that grouped gated feature fusion and error-guided sampling have significant complementary effects in improving reconstruction accuracy, and the two synergistically optimize feature representation efficiency and point cloud spatial distribution. Experimental results on N3D, D-NeRF, HyperNeRF, and the self-built HIMoRe dataset show that the proposed method achieves stable improvements in PSNR, SSIM, and LPIPS: compared with the baseline method 4DGS, PSNR is improved by 1.10 dB and LPIPS is reduced by 0.110 dB on N3D, and PSNR is improved by 0.89 dB on HIMoRe. Visualization results also show that the proposed method has clearer and more complete reconstruction results in fine structures and dynamic regions.

[0215] 5. Compatible with existing frameworks and easy to implement in engineering. All three core modules of this invention can be extended and integrated without changing the overall 4DGS training process, maintaining the advantages of high efficiency and editability of explicit Gaussian representation. This facilitates its application in volumetric video applications such as virtual reality, augmented reality, remote immersive conferencing, free-viewpoint video, and film and television special effects.

[0216] To facilitate understanding of the improvements of the key aspects of this invention compared to the prior art, Table 6 compares this invention with the prior art 1 (4DGS) and the prior art 2 (a method based on spatiotemporal feature plane decomposition) from four dimensions: feature fusion method, densification strategy, opacity processing, and complex dynamic region representation.

[0217] Table 6. Comparison of Key Differences Between the Invention and Prior Art

[0218] Spatiotemporal feature fusion Fixed splicing / weighted Element-wise multiplication / concatenation Grouped gated adaptive weighting Denseization strategy Based on gradient, decoupled from error Based on gradient Error guidance, re-projection along the ray to fill in the missing points Opacity processing Unconstrained Unconstrained Binarization tendency + time smoothing constraint Redundant Gaussian Suppression weak weak Strong (joint constraints) Complex dynamic region details Vague and prone to missing Restricted Clear and complete Timing stability generally generally Significant improvement

[0219] Note: The comparison dimensions in the table reflect the core improvements of this invention compared to existing technologies.

[0220] 1. Alternative Implementation of Gated Network Structure. The lightweight gated network in the grouped gated fusion module can be implemented in various forms such as multilayer perceptron, one-dimensional convolution, or attention structure; the activation function can be Sigmoid, Softmax, or other functions that constrain the output within a finite interval. The grouping method of feature planes can also be flexibly adjusted as needed, for example, divided into two groups according to the spatial plane and the spatiotemporal plane, or grouped separately for each plane.

[0221] 2. Alternative Implementations for Error Measurement and Sampling Location Determination. In addition to pixel color differences, error map calculations can incorporate metrics such as structural similarity, perceptual error, or depth error. High-error regions can be selected using adaptive thresholds or quantile-based methods, in addition to fixed thresholds. The depth λ* of newly added Gaussian units can be determined based on existing Gaussian depth distributions, as well as rendering depth, multi-view consistency, or local geometric priors.

[0222] 3. Alternative implementations of opacity constraints. Besides using binarization tendency constraints... In addition to formal constraints, entropy constraints or other functions that take minimum values ​​at both ends and maximum values ​​in the middle can also be used; for time smoothing constraints, in addition to using the L1 norm of the first-order difference, the L2 norm or higher-order time differences can also be used. The weights of each constraint term can be adjusted according to the characteristics of the scenario.

[0223] 4. Alternative Implementation of the Backbone Framework. The three core modules of this invention are not limited to the 4DGS framework, but can also be ported to other dynamic Gaussian or dynamic radiation field methods that employ spatiotemporal feature plane decomposition and deformation field modeling, replacing or enhancing the corresponding feature fusion, densification, and opacity modeling steps.

[0224] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0225] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0226] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0227] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic scene volumetric video reconstruction based on feature group gating fusion, characterized in that, include: A set of three-dimensional Gaussian elements is sampled in the 3D scene space. Based on the three-dimensional position and timestamp of each Gaussian element, multiple local features are interpolated from the six space-time planes of HexPlane. All local features of all Gaussian elements are divided into several feature groups according to their spatiotemporal relationship. By using the local features of the current sampling location as input through a gating network, the fusion weights of each feature group are adaptively calculated. Based on the fusion weights of each feature group, the local features in each feature group are weighted and fused to obtain the fused spatiotemporal features. The fused spatiotemporal features are input into the deformation decoder, which predicts the update amounts of the position, rotation, and scale of each Gaussian element at each time step. The update amounts of the position, rotation, and scale of each Gaussian element at each time step are applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space. A set of deformed target Gaussian primitives in a 3D scene space is selected, and the set of target Gaussian primitives is rendered into a target view image through differentiable rasterization. The target view image is then used as a synthesized dynamic scene image.

2. The method according to claim 1, characterized in that, The method further includes: The target view image is compared with the real image to obtain a pixel-level error map between the target view image and the real image. Based on the pixel-level error map between the target view image and the real image, a high error region mask in the target view image is obtained through threshold filtering and morphological processing. The high error region mask is projected back onto the 3D scene space along the camera ray of the corresponding pixel. New Gaussian primitives are added to the high error region in the 3D scene space with a low opacity initialization method. The color reconstruction loss is the main component, which is combined with the depth consistency loss, spatial smoothing term, and opacity constraint term to form a joint optimization objective function. This joint optimization objective function is then used to optimize the attributes of all Gaussian units, HexPlane features, gating network, and deformation decoder parameters.

3. The method according to claim 1 or 2, characterized in that, The process involves sampling a set of three-dimensional Gaussian primitives in a 3D scene space. Based on the 3D position and timestamp of each Gaussian primitive, multiple local features are interpolated from the six space-time planes of HexPlane. All local features of all Gaussian primitives are then divided into several feature groups according to their spatiotemporal relationships, including: In a standardized 3D scene space, a set of three-dimensional Gaussian primitives are sampled to obtain the position, shape, color, and transparency information of each Gaussian primitive. For each Gaussian primitive, its three-dimensional position (x, y, z) and timestamp t are used to interpolate the local features of multiple planes from the six space-time planes (xy, xz, yz, xt, yt, zt) of HexPlane. All local features of all Gaussian primitives are divided into several feature groups according to their spatiotemporal relationships.

4. The method according to claim 3, characterized in that, The process involves using a gating network with local features at the current sampling location as input, adaptively calculating the fusion weights for each feature group, and then weighting and fusing the local features in each feature group according to these weights to obtain fused spatiotemporal features. This includes: For a given Gaussian element's 3D position (x, y, z) and timestamp t, HexPlane samples the corresponding local features from six space-time planes, denoted as: in, Indicates the first Local features of each feature plane at the current sampling position; A corresponding gating coefficient is introduced for each set of local features, which is defined as follows: （1） in, This indicates a lightweight gating network. For activation function, Indicates the first Weights of local features in a group; We perform weighted fusion of local features in each feature group to obtain fused spatiotemporal features: （2） in, This represents the fused spatiotemporal features, which specifically include: local spatial geometry and appearance structure information provided by the spatial planes (xy, xz, yz), and dynamic deformation information that changes over time provided by the spatiotemporal planes (xt, yt, zt).

5. The method according to claim 4, characterized in that, The process involves inputting the fused spatiotemporal features into a deformation decoder, which predicts the position, rotation, and scale updates of each Gaussian element at each time step. These updates are then applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space. This includes: The spatiotemporal features are fused and input into the deformation decoder. The deformation decoder predicts the update amount of each Gaussian element at each time step. The position offset Δx, Δy, Δz, rotation update Δr, and scale update Δs of each Gaussian element are predicted by the distribution of the position prediction head, rotation prediction head, and scale prediction head in the deformation decoder. The position, rotation, and scale updates of each Gaussian element at each time step are applied to each Gaussian element in the 3D scene space to obtain the deformed Gaussian elements in the 3D scene space.

6. The method according to claim 2, characterized in that, The process involves comparing the target view image with a real image to obtain a pixel-level error map between the target view image and the real image. Based on the pixel-level error map, a high-error region mask in the target view image is obtained through threshold filtering and morphological processing. The high-error region mask is then projected back onto the 3D scene space along the camera ray of the corresponding pixel. New Gaussian primitives are added to the high-error regions in the 3D scene space using a low-opacity initialization method. In each training iteration, the target view image is compared with the real image to obtain a pixel-level error map between the target view image and the real image: （3） Where p represents the pixel position in the image plane, I(p) is the color value of the real image at pixel p, Î(p) is the pixel value rendered by the current model, and E(p) represents the pixel-level error map between the target view image and the real image at the corresponding position. Based on the pixel-level error map E(p), thresholding and morphological processing are performed on the target viewpoint image to retain regions with errors higher than a set threshold, thus obtaining a high-error region mask in the target viewpoint image: （4） in, Error filtering threshold, It is a binary mask function; When the error at a certain pixel position exceeds a threshold At this point, the location is considered a high-error region. After determining the high-error region, a new Gaussian element is generated in 3D space along the camera ray of the corresponding pixel. Let the camera center of the current viewpoint be o, and the direction of the unit ray corresponding to the pixel be d_p. Then, the camera ray originating from this pixel is represented as: （5） in, Let λ* represent the depth parameter along the ray direction. For pixels in the high-error region, candidate positions are selected in 3D space along the corresponding ray, and new Gaussian primitives are initialized at these positions to map the high-error region in image space to the corresponding position in 3D space. Let the sampling depth selected along the ray be λ*, then the center position of the newly added Gaussian primitive is represented as: （6） in, Let the three-dimensional center position of the newly generated Gaussian element be represented by the following: Let the scale and opacity of the newly added Gaussian element be respectively... and Then its initialization form is written as: , （7） in, and These represent the preset initial scale and initial opacity parameters, respectively.

7. The method according to claim 2, characterized in that, The aforementioned method, which primarily uses color reconstruction loss and combines it with depth consistency loss, spatial smoothing term, and opacity constraint term, constitutes a joint optimization objective function. This joint optimization objective function is used to optimize the attributes of all Gaussian units, HexPlane features, gated network, and deformation decoder parameters, including: In the dynamic Gaussian reconstruction process, an opacity constraint is introduced, let the first... The opacity of each Gaussian element is The corresponding color is The color of the target pixel is then represented as: （8） in, Indicates the final rendered color. Indicates the first The effective opacity of each Gaussian element in the current viewing direction. Indicates the transmission term; Imposing a binarization tendency constraint on the opacity distribution of Gaussian elements, let the th element... The opacity of each Gaussian element is If ∈ [0,1], then the constraint term is defined as: （9） From equation (9), it can be seen that when When the constraint term is close to 0 or 1, the constraint term is smaller; when When the range is in the middle, the constraint terms are larger; Introduce a smoothing constraint in the time dimension, let the... A high-ranking official in Time and The opacity at each moment is respectively and The time smoothing term is then expressed as: （10） Combining the binarization tendency constraint and the smoothing constraint, the final opacity constraint term is defined as: （11） in, and Let represent the weights of the binarization constraint and the temporal smoothing constraint, respectively. The opacity constraint, color reconstruction loss, depth consistency loss, and spatial smoothing term are combined to form the joint optimization objective function: （12） in, This indicates the weight of the opaque constraint in the total loss. The joint optimization objective function is used to optimize the attributes of all Gaussian elements, HexPlane features, gated network and deformation decoder parameters to obtain the optimized attributes of Gaussian elements, HexPlane features, gated network and deformation decoder parameters.

Images