Image rendering method based on dynamic gaussian splatting

Abstract

The application discloses an image rendering method based on dynamic Gaussian splatting, which starts from an adaptive density mechanism, dynamically adjusts the distribution of Gaussian points according to the space-time characteristics, and accurately captures the structural changes and texture details of a complex motion region. First, input multi-view image data to initialize Gaussian attributes with point cloud data; second, obtain a canonical space Gaussian point set through pre-training, train the canonical space through dynamic Gaussian splatting, and enhance the Gaussian distribution density of a motion salient region by using a motion perception key point mechanism; then, calculate an overall loss term, update the attributes and quantity of the Gaussian points; finally, obtain Gaussian model and deformation field network weights, generate multi-view images through a differentiable rasterization rendering pipeline, and output the final result. The application fully utilizes the space-time characteristics of dynamic objects to efficiently reconstruct a dynamic scene, effectively solves the artifact and noise problems in the dynamic scene rendering process, and outputs high-quality rendered images.

Description

Technical Field

[0001] This invention relates to the fields of dynamic scene reconstruction, computer graphics and computer vision, and specifically to an image rendering method based on dynamic Gaussian splashing. Background Technology

[0002] Dynamic scene rendering aims to recover time-varying multi-view images from information with temporal variations (such as multi-frame videos or multi-view temporal images). Unlike traditional static multi-view stereo methods, dynamic scene modeling requires handling complex changes in both spatial and temporal dimensions simultaneously, presenting greater challenges and wider applications, such as virtual reality, robot navigation, motion capture, and immersive video. Current dynamic scene rendering mainly includes the following directions: (1) methods based on temporal depth map estimation. (2) four-dimensional modeling methods based on volume fusion. (3) methods based on dynamic graphics models (such as dynamic point clouds, Gaussians, and meshes). (4) methods based on neural implicit representations.

[0003] With the development of dynamic perception technology and visual acquisition equipment, dynamic scene rendering has gradually become an important research direction in computer vision and graphics. Especially in applications such as video understanding, augmented reality, intelligent robots, and digital humans, the system not only needs to accurately perceive spatial structure, but also must capture and restore the motion state and deformation details of objects to achieve joint modeling of space and time.

[0004] Despite some progress in related research, efficiently and robustly recovering new perspective images of scenes from multiple frames of images or videos in real-world dynamic environments still faces several serious challenges:

[0005] (1) Computational bottlenecks under large-scale data: 3D reconstruction of dynamic scenes usually requires processing continuous video frames or multi-angle images to generate temporally consistent spatial geometric representations. Current methods often face problems of high computation time and long training cycles when processing high-resolution, long-term or multi-view dynamic data, making it difficult to meet the application requirements of real-time interaction or efficient deployment.

[0006] (2) High memory consumption, prone to overflow or system crash. For example, dynamic modeling involves a large amount of temporal data caching, especially in four-dimensional scene modeling, where memory usage is much higher than in static modeling tasks. In long sequences, complex motions or high-resolution scenes, the system is prone to problems such as insufficient video memory, data overflow, and rendering failure.

[0007] (3) The model lacks robustness and exhibits significant differences in different dynamic scenarios: some involve large deformations (such as human movements), some involve extreme movements (such as animal running), and others involve fine-grained changes (such as facial expressions, clothing, etc.). Most current methods rely on specific assumptions or priors, making it difficult to adapt to all types of dynamic changes. Their performance fluctuates greatly in different scenarios, and their transferability and generality need to be improved.

[0008] (4) Occlusion and geometric drift issues: Complex occlusion occurs between dynamic objects and within the objects themselves during motion, leading to problems such as image mismatch and depth discontinuity. Geometric drift, on the other hand, accumulates errors over long sequences, ultimately affecting the spatial consistency and accuracy of the entire rendering result.

[0009] To address these challenges, there is an urgent need to develop an efficient, robust, and scalable dynamic scene rendering method that can not only adapt to the processing of large-scale image sequences but also possess strong geometric modeling and motion representation capabilities to support the complete, continuous, and detailed reconstruction of dynamic objects.

[0010] In recent years, Gaussian splatting technology has achieved great success in the field of real-time rendering of dynamic scenes. Inspired by this, some researchers have applied Gaussian splatting technology to the problem of dynamic scene rendering and have made some research progress. The relevant research papers are: [1] "4D Gaussian Splatting for Real-Time Dynamic SceneRendering", [2] "Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting", [3] "Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting". Among them, [1] was published at CVPR 2024, [2] was published at ICLR 2024, and [3] was published at ECCV 2024. The focus of the above dynamic scene rendering methods is to extend the expression of static scenes from the spatial dimension to the spatiotemporal dimension to express dynamic scenes. Although these methods have made some progress, they are limited to Gaussian points themselves and ignore the influence of the adaptive density mechanism in 3D Gaussian splatting on the reconstruction of complex dynamic scenes. If the density control mechanism fails to focus on fast-moving objects in the scene, it will result in artifacts and noise.

[0011] In other words, when applied to dynamic scenes with complex changes, existing technologies still face the following challenges: (1) noise and artifacts are easily generated in areas where objects move quickly or are partially occluded, affecting the rendering quality; (2) the geometry and appearance modeling of dynamic areas is not refined enough, resulting in blurred local structures; (3) some methods have high computational burden and storage overhead, making it difficult to meet the needs of real-time applications. Summary of the Invention

[0012] Purpose of the invention: To address the shortcomings of existing technical solutions, this invention provides an image rendering method based on dynamic Gaussian splashing. This invention first separates moving objects in the scene based on the spatiotemporal characteristics of moving Gaussian points. Then, it enhances the distribution density of Gaussian points in the moving object region through an adaptive density mechanism. Simultaneously, it prunes noise points and Gaussian points with low contribution in the scene, optimizing the overall spatial distribution of Gaussian points. This makes the local details of the dynamic scene more refined and the structure clearer, thereby improving rendering quality and computational efficiency. It effectively overcomes the deficiencies of existing dynamic scene reconstruction methods and can promote the development of fields such as autonomous driving, virtual reality, game development, and augmented reality.

[0013] Technical solution: The present invention provides an image rendering method based on dynamic Gaussian splashing, comprising the following steps:

[0014] Step 1: Input multi-view image data and camera parameters Combine image data, camera parameters, and timestamp information into a tuple. Each tuple corresponds to a training sample and includes image data, camera pose, and time; among them, Indicates time The next shot Zhang Image Number of camera angles This represents the number of timestamps, which corresponds to the number of video frames. For the camera intrinsic parameter matrix, For the camera rotation matrix, Let be the camera translation vector;

[0015] Step 2: For the tuples in the initial frame Using motion inference structure methods to generate scenes in time The initial point cloud below Then, through the initial point cloud Perform Gaussian property initialization to obtain a set of Gaussian points. ;in, Indicates the first The three-dimensional coordinates of the points For the number of point clouds, The Gaussian point scaling matrix, For color coefficients, For opacity, For the point cloud rotation matrix, Indicates the first One Gaussian point;

[0016] Step 3: Set the Gaussian points Input differentiable rasterization pipeline rendering to obtain Update the set through backpropagation The various attributes are controlled using an adaptive density mechanism to determine the number of Gaussian points. This process involves 3000 iterations, ultimately generating a set of Gaussian points in the canonical space of the dynamic scene;

[0017] Step 4: Based on the canonical space Gaussian point set, the dynamic scene is reconstructed through a deformation field network. Simultaneously, motion-aware keypoints driven by spatiotemporal features are used to enhance the Gaussian distribution density of the motion region. An error pixel-guided pruning mechanism is combined to prune noise points. A structured feature anchor rigid constraint mechanism is used to constrain the spatiotemporal consistency of Gaussian points within their neighborhood, resulting in a local rigidity loss. This is then used to construct the overall loss function. ;

[0018] Step 5: Using the overall loss function Perform backpropagation to update the set of Gaussian points. Related attributes, spatiotemporal structure encoder Weights, multi-head deformable decoder The weights; simultaneously, an adaptive density mechanism is used based on the mask. , And density control of Gaussian points using position gradient thresholds;

[0019] Step 6: Output the final set of Gaussian points after 14,000 iterations of training. and spatiotemporal structure encoder Weights, multi-head deformable decoder The weights are used for real-time rendering; then the final rendered image is generated and output through a differentiable raster rendering pipeline.

[0020] This invention employs spatiotemporal-aware density control to dynamically adjust the distribution of Gaussian points, thereby improving the modeling accuracy of dynamic regions. It introduces a motion-aware keypoint mechanism based on spatiotemporal features to specifically enhance the Gaussian distribution density within the neighborhood of Gaussian points with significant motion trends. An error pixel-guided pruning module is used to locate and remove noise points that affect rendering quality. A structured feature anchor rigid constraint mechanism is designed to constrain local Gaussian point motion deviations, enhancing spatiotemporal consistency. Finally, a PLY file containing Gaussian point attribute information and model weights is generated, allowing for real-time rendering of the results through a rasterization rendering pipeline, producing high-quality rendered images.

[0021] Furthermore, the specific method of step 4 is as follows:

[0022] Step 4.1: In each iteration, the attribute information of the Gaussian points at different times is deformed using a deformation field network (detailed process of the deformation field network reconstructing the dynamic scene: when the deformation field network reconstructs the dynamic scene, it first uses a spatiotemporal structure encoder). Spatiotemporal characteristics Gaussian gas obtained using a multi-head deformable decoder. The deformation field network includes a spatiotemporal structure encoder. and multi-head deformation decoder ;

[0023] The spatiotemporal structure encoder Includes a multi-resolution plane and a compact multilayer perceptron The expression is as follows:

[0024]

[0025] in, Indicates the first Multi-resolution plane of layers, and Two-dimensional grid coordinates on a plane. This represents a multilayer perceptron. It is all the combinations of two-dimensional planes, that is, the various two-dimensional planes in the four-dimensional spacetime composed of three-dimensional space and time dimension;

[0026] Spatiotemporal structure encoder pass Gaussian properties are encoded across different two-dimensional planes, incorporating temporal information into these planes. Features are then extracted and multiplied from these multiple two-dimensional planes using bilinear interpolation, and the results from each layer are fused. Finally, the results are processed... Mapping to the final feature representation; final spatiotemporal features as follows:

[0027] ;

[0028] in, This represents the bilinear interpolation used to query the voxel features located at the four vertices of the grid;

[0029] After all the 3D Gaussian features have been encoded by the spatiotemporal structure encoder, a multi-head deformable decoder is used. Calculate any time The Gaussian property deformation values ​​were calculated using independent multilayer perceptron (MLP) methods to determine the positional deformation. Rotational deformation Scale deformation Opacity distortion and color distortion ;

[0030] The above These are independent multilayer perceptrons;

[0031] Gaussian values ​​are obtained from the above five deformation values. The corresponding time is the first The Gaussian properties of a Gaussian point are:

[0032] ;

[0033] Step 4.2: For the deformed Gaussian properties, first use the differentiable rasterization rendering pipeline to generate the rendered image at the corresponding time step. Then, the pruning module is guided by error pixels to... The error pixels are located, and the corresponding Gaussian points are pruned, as follows:

[0034] First, for each iteration, render images whose rendering metrics are below the structural similarity threshold... Combine real images Calculate error pixels Then, the image area with the largest error was selected, and the pixel blocks within it were filtered to select the top 128 pixels with the largest difference to form a rendering deviation set. : ,in These are the global coordinates of the error pixels;

[0035] When the error pixel coordinate set is obtained Then, through coordinate transformation, the set of rays originating from the camera position and pointing towards the error pixels is obtained: According to the pixel error threshold For 3D Gaussian coordinates By traversing each ray in the set, the final 3D Gaussian coordinates that need to be pruned are obtained. Ensure that only useless noise points that contribute little to the final rendering result are removed; the noise coordinate calculation formula is:

[0036] ;

[0037] in, Represents rays The starting point, i.e., the center of the camera. Represents rays The direction vector. This represents the vector from the starting point of the ray to the Gaussian point. This represents the length of the projection of the vector along the ray direction. Represents the projection vector;

[0038] By traversing all 3D Gaussian coordinates With threshold Comparison, less than the threshold The Gaussian points are considered as error pixels, and finally the Gaussian coordinate masks that need to be pruned are selected. (By traversing all 3D Gaussian coordinates) With threshold Comparison, less than the threshold The Gaussian points are considered as error pixels, i.e., coordinate masks.

[0039] Gaussian coordinate mask The Gaussian points that need to be pruned are marked, and then the corresponding Gaussian points are pruned through an adaptive density mechanism, thereby improving the quality of the rendered image.

[0040] Step 4.3: Transform the Gaussian set obtained in Step 4.1. The spatiotemporal consistency of Gaussian points within their neighborhood is constrained through a rigid constraint mechanism of structured feature anchors. The specific steps are as follows:

[0041] First, feature anchor points (i.e., cluster centers) are selected from Gaussian points through spatiotemporal clustering. Then, these anchor points are used as sampling points across the entire set of Gaussian points. Searching for it 1 nearest neighbor, calculate anchor point and The loss value is calculated based on the Euclidean distance between the nearest neighbors. Simultaneously, to further reduce the interference of distant points on local rigid constraints, an exponentially decaying weight is introduced, and the loss is weighted according to the distance to obtain the weight. The formula is: ;

[0042] in, This represents the hyperparameter that controls weight decay. Indicates the first The anchor point and the first Euclidean distance between the nearest neighbors;

[0043] Then, the weighted stiffness loss value is calculated, resulting in the final local stiffness loss. for:

[0044] ;

[0045] in, Indicates the number of anchor points. This represents the number of nearest neighbors for each anchor point. This represents the spatial coordinates of the Gaussian point.

[0046] The differentiable rasterization pipeline in this invention refers to the rendering pipeline in 3DGS. The raster rendering in step 3 is used for the initial training of the first frame of the scene (static scene), and step 4 is used for the training of the dynamic scene.

[0047] Furthermore, the specific method for enhancing the Gaussian distribution density of motion-sensing keypoint mechanisms driven by spatiotemporal features is as follows:

[0048] Assume at time... , , Below, the spatial coordinates of the Gaussian point are determined by... It means that the first The velocity of the Gaussian point Motion acceleration and direction of movement The calculation formula is as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] The mean values ​​of velocity, acceleration, and direction of motion , , and standard deviation , and The calculation formula is as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] During the iteration process, based on the motion threshold , , Gaussian points with significant motion characteristics are selected; those exceeding three thresholds are considered to have significant motion features. The motion thresholds are calculated as follows:

[0060] ;

[0061] ;

[0062] ;

[0063] , These are used to filter out Gaussian points with obvious motion trends, and then new points are sampled near these points to force the model to pay sufficient attention to the motion region;

[0064] in, This is a hyperparameter used to control the strictness of the screening process;

[0065] Subsequently, a mask is generated based on the motion threshold. Based on this mask, an adaptive density mechanism is used to locally compact the area in the neighborhood of the corresponding Gaussian point.

[0066] Furthermore, the overall loss function The expression is as follows:

[0067] ;

[0068] in, For image reconstruction loss, For the total variational loss, For entropy loss, This refers to the localized rigidity loss at the anchor point.

[0069] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0070] (1) The present invention solves the problems of unreasonable density distribution and poor spatiotemporal consistency in previous methods by using an image rendering method based on dynamic Gaussian splashing, and achieves high rendering quality and low storage overhead in different dynamic scenes.

[0071] (2) This invention proposes a motion perception keypoint mechanism driven by spatiotemporal features, which solves the problem that the adaptive density mechanism cannot enhance the Gaussian distribution density in dynamic regions. This mechanism obtains Gaussian points with motion tendency by statistically analyzing the motion information of objects at different times, and makes the adaptive density mechanism dense in the neighborhood of these Gaussian points, thereby improving the attention to moving objects.

[0072] (3) The present invention proposes a novel error pixel-guided pruning module, which can eliminate noise that has a significant impact on image rendering quality, thereby optimizing the distribution of Gaussian points in the scene and improving image rendering quality.

[0073] (4) The present invention designs a structured feature anchor point rigid constraint mechanism, which can effectively reduce the motion deviation of Gaussian points in the local range and improve the robustness of the method in different dynamic scenarios.

[0074] (5) This invention can solve the underfitting problem caused by existing dynamic scene reconstruction methods when processing sparse viewpoint image data, improve the real-time rendering speed after dynamic scene reconstruction, and reduce the memory overhead generated during training, so that the method can run on different devices.

[0075] (6) This invention can provide high-quality models for fields such as game production, virtual reality, augmented reality, autonomous driving and 4D asset generation. Attached Figure Description

[0076] Figure 1 This is an overall flowchart of the present invention;

[0077] Figure 2 These are samples of image data from different viewpoints at the same moment in the embodiment.

[0078] Figure 3 This refers to the initial point cloud model used in the embodiment.

[0079] Figure 4 This is the result of dynamic region segmentation in the embodiment;

[0080] Figure 5 These are the rendering results of the present invention in different scenarios;

[0081] Figure 6 This is the rendering result of the present invention on dynamic objects;

[0082] Figure 7 The rendering results of the present invention and existing methods in different scenarios are shown. Detailed Implementation

[0083] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0084] Existing dynamic Gaussian methods often neglect the impact of adaptive density mechanisms on rendering results when representing dynamic scenes, especially in data with sparse viewpoints or large scene changes, which often leads to noise and motion artifacts.

[0085] To address the aforementioned issues, this invention separates moving objects in a scene based on the spatiotemporal characteristics of moving Gaussian points. Then, it enhances the distribution density of Gaussian points in the moving object region through an adaptive density mechanism. Simultaneously, it prunes noise points and Gaussian points with low contribution in the scene, optimizing the overall spatial distribution of Gaussian points. This makes the local details of the dynamic scene more refined and the structure clearer, thereby improving rendering quality and computational efficiency.

[0086] like Figure 1 As shown, the image rendering method based on dynamic Gaussian splashing of the present invention includes the following steps:

[0087] Step 1: Input multi-view image data and camera parameters Combine image data, camera parameters, and timestamp information into a tuple. Each tuple corresponds to a training sample and includes image data, camera pose, and time; among them, Indicates time The next shot Zhang Image Number of camera angles This represents the number of timestamps, which corresponds to the number of video frames. For the camera intrinsic parameter matrix, For the camera rotation matrix, Let be the camera translation vector;

[0088] Step 2: For the tuples in the initial frame Using motion inference structure methods to generate scenes in time The initial point cloud below Then, through the initial point cloud Perform Gaussian property initialization to obtain a set of Gaussian points. ;in, Indicates the first The three-dimensional coordinates of the points For the number of point clouds, The Gaussian point scaling matrix, For color coefficients, For opacity, For the point cloud rotation matrix, Indicates the first One Gaussian point;

[0089] Step 3: Set the Gaussian points The input is a differentiable rasterization pipeline to generate the rendered image. Update the set through backpropagation The various attributes are controlled using an adaptive density mechanism to determine the number of Gaussian points. This process involves 3000 iterations, ultimately generating a set of Gaussian points in the canonical space of the dynamic scene;

[0090] Step 4: Based on the canonical space Gaussian point set, the dynamic scene is reconstructed through a deformation field network. Simultaneously, motion-aware keypoints driven by spatiotemporal features are used to enhance the Gaussian distribution density of the motion region. An error pixel-guided pruning mechanism is combined to prune noise points. A structured feature anchor rigid constraint mechanism is used to constrain the spatiotemporal consistency of Gaussian points within their neighborhood, resulting in a local rigidity loss. This is then used to construct the overall loss function. The specific method is as follows:

[0091] Step 4.1: In each iteration, the attribute information of the Gaussian point at different times is deformed through a deformation field network, which includes a spatiotemporal structure encoder. and multi-head deformation decoder ;

[0092] Spatiotemporal structure encoder Includes a multi-resolution plane and a compact multilayer perceptron The expression is as follows:

[0093]

[0094] in, Indicates the first Multi-resolution plane of layers, and Two-dimensional grid coordinates on a plane. This represents a multilayer perceptron. It is all the combinations of two-dimensional planes, that is, the various two-dimensional planes in the four-dimensional spacetime composed of three-dimensional space and time dimension;

[0095] Spatiotemporal structure encoder pass Gaussian properties are encoded across different two-dimensional planes, incorporating temporal information into these planes. Features are then extracted and multiplied from these multiple two-dimensional planes using bilinear interpolation, and the results from each layer are fused. Finally, the results are processed... Mapping to the final feature representation; final spatiotemporal features as follows:

[0096] ;

[0097] in, This represents the bilinear interpolation used to query the voxel features located at the four vertices of the grid;

[0098] After all the 3D Gaussian features have been encoded by the spatiotemporal structure encoder, a multi-head deformable decoder is used. Calculate any time The Gaussian property deformation values ​​were calculated using independent multilayer perceptron (MLP) methods to determine the positional deformation. Rotational deformation Scale deformation Opacity distortion and color distortion ;

[0099] The above These are independent multilayer perceptrons;

[0100] Gaussian values ​​are obtained from the above five deformation values. The corresponding time is the first The Gaussian properties of a Gaussian point are:

[0101] ;

[0102] Step 4.2: For the deformed Gaussian properties, first use the differentiable rasterization rendering pipeline to generate the rendered image at the corresponding time step. Then, the pruning module is guided by error pixels to... The error pixels are located, and the corresponding Gaussian points are pruned, as follows:

[0103] First, for each iteration, render images whose rendering metrics are below the structural similarity threshold... Combine real images Calculate error pixels Then, the image area with the largest error was selected, and the pixel blocks within it were filtered to select the top 128 pixels with the largest difference: ,in These are the global coordinates of the error pixels;

[0104] When the error pixel coordinate set is obtained Then, through coordinate transformation, the set of rays originating from the camera position and pointing towards the error pixels is obtained: According to the pixel error threshold For 3D Gaussian coordinates By traversing each ray in the set, the final 3D Gaussian coordinates that need to be pruned are obtained. Ensure that only those useless noise points that contribute little to the final rendering result are removed;

[0105] The formula for calculating noise coordinates is:

[0106] ;

[0107] in, Represents rays The starting point, i.e., the center of the camera. Represents rays The direction vector. This represents the vector from the starting point of the ray to the Gaussian point. This represents the length of the projection of the vector along the ray direction. Represents the projection vector;

[0108] By traversing all 3D Gaussian coordinates With threshold Comparison, less than the threshold The Gaussian points are considered as error pixels, and finally the Gaussian coordinate masks that need to be pruned are selected. ;

[0109] Gaussian coordinate mask The Gaussian points that need to be pruned are marked, and then the corresponding Gaussian points are pruned through an adaptive density mechanism, thereby improving the quality of the rendered image.

[0110] Step 4.3: Transform the Gaussian set obtained in Step 4.1. The spatiotemporal consistency of Gaussian points within their neighborhood is constrained through a rigid constraint mechanism of structured feature anchors. The specific steps are as follows:

[0111] First, feature anchors are selected from Gaussian points using spatiotemporal clustering. Then, these anchors are used as sampling points across the entire Gaussian point set. Searching for it 1 nearest neighbor, calculate anchor point and The loss value is calculated based on the Euclidean distance between the nearest neighbors; simultaneously, an exponentially decaying weight is introduced to weight the loss according to the distance. The formula is: ;

[0112] in, This represents the hyperparameter that controls weight decay. Indicates the first The anchor point and the first Euclidean distance between the nearest neighbors;

[0113] Then, the weighted stiffness loss value is calculated, resulting in the final local stiffness loss. for:

[0114] ;

[0115] in, Indicates the number of anchor points. This represents the number of nearest neighbors for each anchor point. Represents the spatial coordinates of the Gaussian point;

[0116] Overall loss function The expression is as follows:

[0117] ;

[0118] in, For image reconstruction loss, For the total variational loss, For entropy loss, This represents the localized stiffness loss at the anchor point.

[0119] Step 5: Using the overall loss function Perform backpropagation to update the set of Gaussian points. The relevant attributes and weights of the deformation field network; simultaneously, based on the mask, an adaptive density mechanism is used. , And density control of Gaussian points using position gradient thresholds;

[0120] Step 6: Output the final set of Gaussian points after 14,000 iterations of training. and spatiotemporal structure encoder Weights, multi-head deformable decoder The weights are used for real-time rendering; then the final rendered image is generated and output through a differentiable raster rendering pipeline.

[0121] This invention first inputs multi-view image data and initializes Gaussian attributes with point cloud data; secondly, it obtains a canonical space Gaussian point set through pre-training, trains the canonical space through dynamic Gaussian splashing, and enhances the Gaussian distribution density in motion-aware keypoints using a motion-aware keypoint mechanism; then, it calculates the overall loss term, and updates the model weights by backpropagation combined with position gradient and density control; finally, it outputs the Gaussian model and deformation field network weights, and outputs multi-view images through a differentiable raster rendering pipeline.

[0122] The specific method used in this embodiment to enhance the Gaussian distribution density of motion-sensing keypoints driven by spatiotemporal features to improve motion perception density in salient regions is as follows:

[0123] Assume at time... , , Below, the spatial coordinates of the Gaussian point are determined by... It means that the first The velocity of the Gaussian point Motion acceleration and direction of movement The calculation formula is as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] The mean values ​​of velocity, acceleration, and direction of motion , , and standard deviation , and The calculation formula is as follows:

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] During the iteration process, based on the motion threshold , , Gaussian points with significant motion characteristics are selected; those exceeding three thresholds are considered to have significant motion characteristics. The motion thresholds are calculated as follows:

[0135] ;

[0136] ;

[0137] ;

[0138] , These are used to filter out Gaussian points with obvious motion trends, and then new points are sampled near these points to force the model to pay sufficient attention to the motion region;

[0139] in, This is a hyperparameter used to control the strictness of the screening process;

[0140] Subsequently, a mask is generated based on the motion threshold. The adaptive density mechanism is guided to achieve local compaction in the neighborhood of the corresponding Gaussian point.

[0141] In this embodiment, the input image data sample is as follows: Figure 2 As shown, Figure 2 The image shows four images taken at the same time from different perspectives. In this embodiment, the initial point cloud model is as follows: Figure 3 As shown, the initial point cloud model reconstructed from the image data has a high degree of geometric consistency with the real scene.

[0142] In this embodiment, the Gaussian distribution density of motion-sensitive keypoints driven by spatiotemporal features is enhanced, such as... Figure 4 As shown, this invention extracts the spatiotemporal features of Gaussian points by statistically analyzing physical motion quantities such as velocity, acceleration, and direction changes, and dynamically adjusts the distribution of Gaussian points to achieve local density in regions of significant motion, thereby ensuring that the model pays more attention to dynamic regions. The final rendering results in different scenarios are shown below. Figure 5 As shown. The rendering result on the dynamic area in the scene is as follows. Figure 6 As shown. The rendering results compared to existing technologies in different scenarios are as follows. Figure 7 As shown.

Claims

1. An image rendering method based on dynamic Gaussian splashing, characterized in that, Includes the following steps: Step 1: Input multi-view image data and camera parameters Combine image data, camera parameters, and timestamp information into a tuple. Each tuple corresponds to a training sample and includes image data, camera pose, and time; among them, Indicates time The next shot Zhang Image Number of camera angles This represents the number of timestamps, which corresponds to the number of video frames. For the camera intrinsic parameter matrix, For the camera rotation matrix, Let be the camera translation vector; Step 2: For the tuples in the initial frame Using motion inference structure methods to generate scenes in time The initial point cloud below Then, through the initial point cloud Perform Gaussian property initialization to obtain a set of Gaussian points. ;in, Indicates the first The three-dimensional coordinates of the points For the number of point clouds, The Gaussian point scaling matrix, For color coefficients, For opacity, For the point cloud rotation matrix, Indicates the first One Gaussian point; Step 3: Set Gaussian points The input is a differentiable rasterization pipeline to generate the rendered image. Update the set through backpropagation The various attributes are controlled using an adaptive density mechanism to determine the number of Gaussian points. This process involves 3000 iterations, ultimately generating a set of Gaussian points in the canonical space of the dynamic scene; Step 4: Based on the canonical space Gaussian point set, the dynamic scene is reconstructed through a deformation field network. Simultaneously, motion-aware keypoints driven by spatiotemporal features are used to enhance the Gaussian distribution density of the motion region. An error pixel-guided pruning mechanism is combined to prune noise points. A structured feature anchor rigid constraint mechanism is used to constrain the spatiotemporal consistency of Gaussian points within their neighborhood, resulting in a local rigidity loss. This is then used to construct the overall loss function. ; Step 5: Using the overall loss function Perform backpropagation to update the set of Gaussian points. The relevant attributes and weights of the deformation field network; simultaneously, based on the mask, an adaptive density mechanism is used. , And density control of Gaussian points using position gradient thresholds; Step 6: Output the final set of Gaussian points after 14,000 iterations of training. and spatiotemporal structure encoder Weights, multi-head deformable decoder The weights are used for real-time rendering; then the final rendered image is generated and output through a differentiable raster rendering pipeline.

2. The image rendering method based on dynamic Gaussian splashing according to claim 1, characterized in that, The specific method for step 4 is as follows: Step 4.1: In each iteration, the attribute information of the Gaussian point at different times is deformed through a deformation field network, which includes a spatiotemporal structure encoder. and multi-head deformation decoder ; Spatiotemporal structure encoder Includes a multi-resolution plane and a compact multilayer perceptron The expression is as follows:

3. Among them, Indicates the first Multi-resolution plane of layers, and Two-dimensional grid coordinates on a plane. This represents a multilayer perceptron. It is all the combinations of two-dimensional planes, that is, the various two-dimensional planes in the four-dimensional spacetime composed of three-dimensional space and time dimension; Spatiotemporal structure encoder pass Gaussian properties are encoded across different two-dimensional planes, incorporating temporal information into these planes. Features are then extracted and multiplied from these multiple two-dimensional planes using bilinear interpolation, and the results from each layer are fused. Finally, the results are processed... Mapped to the final feature representation; final Spatiotemporal characteristics as follows: ； in, This represents the bilinear interpolation used to query the voxel features located at the four vertices of the grid; After all the 3D Gaussian features have been encoded by the spatiotemporal structure encoder, a multi-head deformable decoder is used. Calculate any time The Gaussian property deformation values ​​were calculated using independent multilayer perceptron (MLP) methods to determine the positional deformation. Rotational deformation Scale deformation Opacity distortion and color distortion ; The above These are independent multilayer perceptrons; Gaussian values ​​are obtained from the above five deformation values. The corresponding time is the first The Gaussian properties of a Gaussian point are: ； Step 4.2: For the deformed Gaussian properties, first use the differentiable rasterization rendering pipeline to generate the rendered image at the corresponding time step. Then, the pruning module is guided by error pixels to... The error pixels are located, and the corresponding Gaussian points are pruned, as follows: First, for each iteration, render images whose rendering metrics are below the structural similarity threshold... Combine real images Calculate error pixels Then, the image area with the largest error was selected, and the pixel blocks within it were filtered to select the top 128 pixels with the largest difference to form a rendering deviation set. : ,in These are the global coordinates of the error pixels; When the error pixel coordinate set is obtained Then, through coordinate transformation, the set of rays originating from the camera position and pointing towards the error pixels is obtained: According to the pixel error threshold For 3D Gaussian coordinates By traversing each ray in the set, the final 3D Gaussian coordinates that need to be pruned are obtained. Ensure that only those useless noise points that contribute little to the final rendering result are removed; The formula for calculating noise coordinates is: ； in, Represents rays The starting point, i.e., the center of the camera; Represents rays The direction vector. This represents the vector from the starting point of the ray to the Gaussian point. This represents the length of the projection of the vector along the ray direction. Represents the projection vector; By traversing all 3D Gaussian coordinates With threshold Comparison, less than the threshold The Gaussian points are considered as error pixels, and finally the Gaussian coordinate masks that need to be pruned are selected. ; Gaussian coordinate mask The Gaussian points that need to be pruned are marked, and then the corresponding Gaussian points are pruned through an adaptive density mechanism, thereby improving the quality of the rendered image. Step 4.3: Transform the Gaussian set obtained in Step 4.

1. The spatiotemporal consistency of Gaussian points within their neighborhood is constrained through a rigid constraint mechanism of structured feature anchors. The specific steps are as follows: First, feature anchors are selected from Gaussian points using spatiotemporal clustering. Then, these anchors are used as sampling points across the entire Gaussian point set. Searching for it 1 nearest neighbor, calculate anchor point and The loss value is calculated based on the Euclidean distance between the nearest neighbors; simultaneously, an exponentially decaying weight is introduced to weight the loss according to the distance. The formula is: ; in, This represents the hyperparameter that controls weight decay. Indicates the first The anchor point and the first Euclidean distance between the nearest neighbors; Then, the weighted stiffness loss value is calculated, resulting in the final local stiffness loss. for: ； in, Indicates the number of anchor points. This represents the number of nearest neighbors for each anchor point. This represents the spatial coordinates of the Gaussian point.

4. The image rendering method based on dynamic Gaussian splashing according to claim 1, characterized in that, The specific method for enhancing the Gaussian distribution density of motion-sensing keypoint mechanisms driven by spatiotemporal features is as follows: Assume at time... , , Below, the spatial coordinates of the Gaussian point are determined by... It means that the first The velocity of the Gaussian point Motion acceleration and direction of movement The calculation formula is as follows: ； ； ； The mean values ​​of velocity, acceleration, and direction of motion , , and standard deviation , and The calculation formula is as follows: ； ； ； ； ； ； During the iteration process, based on the motion threshold , , Gaussian points with significant motion characteristics are selected; Gaussian points with three thresholds are considered to have significant motion characteristics. The motion threshold is calculated as follows: ； ； ； , These are used to filter out Gaussian points with obvious motion trends, and then new points are sampled near these points to force the model to pay sufficient attention to the motion region; in, This is a hyperparameter used to control the strictness of the screening process; Subsequently, a mask is generated based on the motion threshold. The adaptive density mechanism is guided to achieve local compaction in the neighborhood of the corresponding Gaussian point.

5. The image rendering method based on dynamic Gaussian splashing according to claim 1, characterized in that, The overall loss function The expression is as follows: ； in, For image reconstruction loss, For the total variational loss, For entropy loss, This refers to the localized rigidity loss at the anchor point.

Images