A Dynamic Scene Deblurring Method and System Based on 4D Gaussian and Pseudo-True Value Supervision

Abstract

This invention proposes a dynamic scene deblurring method and system based on 4D Gaussian representation and pseudo-ground value supervision. The method includes: obtaining a sharp image and a rendered depth map from a 4D Gaussian representation; predicting the motion velocity of dynamic pixels using a velocity MLP network based on the rendered depth map; constructing a fuzzy weight network and predicting the contribution weight of each sampling point on the sampling trajectory; fusing the pseudo-sharp image with the original blurred input using a dynamic region mask to obtain a hybrid pseudo-ground value image; synthesizing a physically blurred image through weighted integration; and constructing a dual-domain reconstruction loss and geometry-motion regularization constraints by combining hybrid pseudo-ground value supervision and gradient decoupling strategies to jointly optimize the scene representation, motion parameters, and fuzzy model, thereby obtaining a sharp dynamic scene representation. This invention achieves explicit modeling of the physical blurring process through 4D Gaussian representation and a fuzzy weight network, improving the realism and view consistency of motion blur synthesis.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction and image restoration, and particularly to a dynamic scene deblurring method and system based on 4D Gaussian and pseudo-truth value supervision. Background Technology

[0002] 3D reconstruction of dynamic scenes from multiple perspectives is an important research direction in the field of computer vision, and it is widely used in immersive media, autonomous driving, robot navigation and other scenarios. However, in the actual acquisition process, the coupling between object motion and camera exposure time causes motion blur in the acquired images. This blur presents different distribution patterns in multiple perspectives, which seriously destroys the geometric consistency across perspectives and poses a fundamental challenge to subsequent 3D reconstruction.

[0003] Existing deblurring techniques are mainly divided into two categories: one is based on two-dimensional image processing, which estimates the blur kernel and recovers the clear image by using information from a single frame or neighboring frames. However, this type of method lacks explicit modeling of three-dimensional geometry and is difficult to guarantee texture consistency across multiple viewpoints. The other type is based on three-dimensional representation reconstruction methods, such as Neural Radiation Field (NeRF) and its variants. Although they achieve excellent results in static scenes, when processing dynamically blurred videos, they often assume that the input image is clear or only slightly degraded, and cannot cope with the complex blur patterns caused by the continuous movement of local objects within the exposure time.

[0004] In recent years, dynamic reconstruction methods based on 3D Gaussian sputtering (3DGS) have attracted widespread attention due to their efficient rendering speed and explicit scene representation. However, existing methods have two key limitations when dealing with blurred inputs: first, most methods still rely on the assumption of clear images and cannot directly model the physical integration process within the exposure time; second, some methods introduce implicit deformation fields to fit motion trajectories, but lack an explicit decoupling mechanism for the coupling relationship between motion and geometry. This can easily lead to the interpretation of local motion by distorting the static background during the optimization process, resulting in geometric distortion and temporal inconsistencies in the reconstruction results. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a dynamic scene deblurring method and system based on 4D Gaussian representation and pseudo-truth value supervision, so as to solve the above-mentioned technical problems.

[0006] This invention proposes a dynamic scene deblurring method based on 4D Gaussian representation and pseudo-truth value supervision, the method comprising the following steps:

[0007] Step 1: Input a multi-view motion-blurred video sequence, where each frame serves as the original blurred input image; construct a 4D Gaussian representation of the dynamic scene based on the multi-view motion-blurred video sequence; render a sharp rendered image and a rendered depth map from the 4D Gaussian representation; obtain a dynamic region mask from the original blurred input image using a pre-trained instance segmentation model, and define dynamic pixels based on the dynamic region mask; obtain a monocular depth prior, and fuse the rendered depth map with the monocular depth prior to construct a hybrid depth map; predict the motion velocity of the dynamic pixels in the camera coordinate system using a velocity MLP network based on the rendered depth map to obtain the sampling trajectory of the dynamic pixels during the exposure time;

[0008] Step 2: Construct a fuzzy weighted network. Using the motion velocity and blending depth map in the camera coordinate system as input, predict the contribution weight of each sampling point on the sampling trajectory, and use the contribution weight to perform weighted integration on the clear rendering image along the sampling trajectory to synthesize a physically blurred image.

[0009] Step 3: Generate a pseudo-sharp image from the original blurred input image using a pre-trained deblurring network; fuse the pseudo-sharp image with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; construct a sharpness reconstruction loss based on the hybrid pseudo-ground value image to constrain the sharp rendering image, and construct a re-blurring loss based on the physically blurred image and the original blurred input image.

[0010] Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

[0011] This invention also proposes a dynamic scene deblurring system based on 4D Gaussian and pseudo-truth value supervision, the system comprising:

[0012] The scene representation and trajectory modeling module is used for:

[0013] The system takes a multi-view motion-blurred video sequence as input, with each frame serving as the original blurred input image. A 4D Gaussian representation of the dynamic scene is constructed based on the multi-view motion-blurred video sequence. A sharp rendered image and a rendered depth map are obtained from the 4D Gaussian representation. A pre-trained instance segmentation model is used to obtain a dynamic region mask from the original blurred input image, and dynamic pixels are defined based on the dynamic region mask. A monocular depth prior is obtained, and the rendered depth map is fused with the monocular depth prior to construct a hybrid depth map. Based on the rendered depth map, a velocity MLP network is used to predict the motion velocity of the dynamic pixels in the camera coordinate system, thus obtaining the sampling trajectory of the dynamic pixels within the exposure time.

[0014] The fuzzy weight prediction and fuzzy synthesis module is used for:

[0015] A fuzzy weighted network is constructed, taking the motion velocity and blending depth map in the camera coordinate system as input, predicting the contribution weight of each sampling point on the sampling trajectory, and using the contribution weight to perform weighted integration on the sharp rendered image along the sampling trajectory to synthesize a physically blurred image.

[0016] The optimization module is used for:

[0017] A pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image is fused with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; a sharpness reconstruction loss is constructed based on the hybrid pseudo-ground value image to constrain the sharp rendered image, and a re-blurring loss is constructed based on the physically blurred image and the original blurred input image;

[0018] Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] 1. This invention addresses the optimization ambiguity caused by the coupling of motion and geometry. The method constructs a hybrid pseudo-truth image to provide a reliable monitoring signal for clarifying dynamic regions. Simultaneously, it introduces gradient decoupling in the calculation of the re-blurring loss, strictly limiting the gradient of the dynamic blur fitting to the dynamic region. This combination effectively resolves the motion-geometry optimization ambiguity, fundamentally avoiding the inherent problem of distorting the static background structure to fit dynamic blur in traditional methods, thus ensuring the geometric integrity and stability of the static scene.

[0021] 2. By simultaneously constructing reconstruction losses in both the clear and blurred image domains, and supplementing them with multiple regularization constraints such as depth consistency, rigid body velocity, and temporal smoothing, the method of this invention constructs an end-to-end joint optimization framework. This framework collaboratively optimizes scene geometry, object motion trajectories, and blur kernel parameters, significantly improving the deblurring quality of dynamic regions, especially in restoring high-frequency texture details and maintaining the sharpness of motion edges, while ensuring the spatiotemporal continuity of the synthesized video.

[0022] 3. Quantitative evaluation results on publicly available multi-view dynamic blur datasets show that the proposed method outperforms existing mainstream technologies in core metrics such as PSNR, SSIM, and LPIPS. In particular, in the most challenging dynamic regions, the proposed method recovers richer details and more consistent geometric structures, verifying its excellent performance and strong generalization ability in handling complex motion blur scenes.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the steps of the dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision proposed in this invention.

[0025] Figure 2 This is a diagram illustrating the overall framework of the dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision proposed in this invention.

[0026] Figure 3 This is a schematic diagram of the overall framework of the dynamic scene deblurring system based on 4D Gaussian and pseudo-truth value supervision proposed in this invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0028] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0029] Please see Figure 1 This invention proposes a dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision, which includes the following steps:

[0030] Step 1: Input a multi-view motion-blurred video sequence, where each frame serves as the original blurred input image; construct a 4D Gaussian representation of the dynamic scene based on the multi-view motion-blurred video sequence; render a sharp rendered image and a rendered depth map from the 4D Gaussian representation; obtain a dynamic region mask from the original blurred input image using a pre-trained instance segmentation model, and define dynamic pixels based on the dynamic region mask; obtain a monocular depth prior, and fuse the rendered depth map with the monocular depth prior to construct a hybrid depth map; predict the motion velocity of the dynamic pixels in the camera coordinate system using a velocity MLP network based on the rendered depth map to obtain the sampling trajectory of the dynamic pixels during the exposure time.

[0031] Please see Figure 2 In step 1, a 4D Gaussian representation of the dynamic scene is constructed based on the multi-view motion-blurred video sequence; a sharp rendered image and a rendered depth map are obtained from the 4D Gaussian representation; a dynamic region mask is obtained from the original blurred input image using a pre-trained instance segmentation model, and dynamic pixels are defined based on the dynamic region mask; a monocular depth prior is obtained, and the rendered depth map is fused with the monocular depth prior to construct a hybrid depth map; based on the rendered depth map, the motion velocity of the dynamic pixels in the camera coordinate system is predicted through a velocity MLP network to obtain the sampling trajectory of the dynamic pixels during the exposure time. Specifically, the steps include the following:

[0032] A motion structure recovery algorithm is used to process multi-view motion-blurred video sequences to recover the camera pose parameters and sparse 3D point cloud of the scene corresponding to each frame; where each frame of the view motion-blurred video sequence is used as the original blurred input image.

[0033] Based on the camera pose parameters and the sparse 3D point cloud of the scene, the dynamic scene is parameterized into a set of 4D Gaussian elements with time attributes.

[0034] Using the primitives in the four-dimensional Gaussian primitive set, a 4D Gaussian representation of the dynamic scene is constructed. The 4D Gaussian representation of the dynamic scene is then rendered by differentiable rasterization to obtain a clear rendered image. The 4D Gaussian representation of the dynamic scene is then rendered by differentiable rasterization again to obtain a rendering depth map.

[0035] Using a pre-trained instance segmentation and tracking model, foreground moving object segmentation is performed on each frame of a multi-view motion-blurred video sequence to generate a dynamic region mask;

[0036] Dynamic pixel locations are distinguished using dynamic region masks to obtain a set of dynamic pixel locations;

[0037] A pre-trained monocular depth estimation network is used to extract monocular depth priors from multi-view motion-blurred video sequences, and the monocular depth priors are fused with the rendered depth map through learnable fusion coefficients to construct a hybrid depth map.

[0038] Given a camera pose, differentiable rasterization rendering or hybrid depth back projection calculation is performed using the 4D Gaussian representation of the dynamic scene to obtain the world coordinates corresponding to the dynamic pixels.

[0039] The world coordinates and blended depth map corresponding to the dynamic pixels are input into the velocity MLP network to regress the instantaneous motion velocity in the world coordinate system in order to obtain the motion vector in the world coordinate system.

[0040] Transform the motion vector in the world coordinate system to the camera coordinate system of the current viewpoint to obtain the camera coordinate velocity;

[0041] Discretize the exposure time to obtain the discrete sampling time;

[0042] By using the blended depth map and the camera intrinsic matrix, each pixel in the dynamic pixel position set is back-projected onto the camera coordinate system to obtain the initial 3D position;

[0043] Based on the camera coordinate velocity, the initial 3D position corresponding to each discrete sampling moment is calculated and reprojected onto the two-dimensional imaging plane to obtain the sampling point coordinates corresponding to the motion trajectory.

[0044] It should be noted that the instance segmentation and tracking model used in this embodiment is Track-Anything, which is an existing technology, and its specific structure will not be described in detail here.

[0045] Based on the camera pose parameters and the sparse 3D point cloud of the scene, the dynamic scene is parameterized into a set of four-dimensional Gaussian primitives with temporal attributes; wherein, each primitive attribute of the four-dimensional Gaussian primitive set consists of the following parameters: spatial position vector. Covariance matrix Optical opacity scalar and view-related color vectors (Using spherical harmonic basis function encoding); where, Representing three-dimensional real space, It represents a 3×3 real matrix space.

[0046] Among them, covariance matrix The decomposition and construction process using rotation and scaling components yields the following relationship:

[0047] ;

[0048] in, Describe the covariance matrix. This represents the rotation moments obtained from learnable quaternions through orthogonalization. This represents the diagonal scaling matrix constructed from the three-dimensional scaling parameters via an exponential mapping. This represents the matrix transpose operator;

[0049] In the step of using a dynamic region mask to distinguish the positions of dynamic pixels in order to obtain a set of dynamic pixel positions, the corresponding relationship in the process is as follows:

[0050] ;

[0051] in, This represents the set of pixel locations that are all marked as dynamic regions. Represents the two-dimensional pixel coordinates on the image plane. Represents a two-dimensional real number space. Indicates a dynamic region mask;

[0052] In the steps of extracting monocular depth priors from multi-view motion-blurred video sequences using a pre-trained monocular depth estimation network, and fusing the monocular depth priors with the rendered depth map through learnable fusion coefficients to construct a hybrid depth map, the corresponding relationships in the process are as follows:

[0053] ;

[0054] in, Represents a mixed depth map. Indicates the blending depth value. Represents the learnable fusion coefficient. This represents the relative depth map extracted from the original blurred frame using a pre-trained monocular depth estimation network. This represents the geometric depth map obtained by rendering the current 4D Gaussian model through a differentiable rasterization pipeline.

[0055] The world coordinates and blended depth map corresponding to the dynamic pixels are input into the velocity MLP network to regress the instantaneous motion velocity in the world coordinate system, so as to obtain the motion vector in the world coordinate system. The corresponding relationship in the process is as follows:

[0056] ;

[0057] in, This represents the velocity vector in the world coordinate system. This refers to a lightweight multilayer perceptron. Represents the three-dimensional coordinates of the world coordinate system;

[0058] In the step of transforming the motion vector from the world coordinate system to the camera coordinate system of the current viewpoint to obtain the camera coordinate velocity, the corresponding relationship in the process is as follows:

[0059] ;

[0060] in, This represents the rotation matrix extracted from the camera extrinsic parameters of the current frame. The camera coordinate system representing the current viewpoint;

[0061] In the step of discretizing the exposure time, the corresponding relationship in the process is as follows:

[0062] ;

[0063] in, Indicates the first exposure based on the center of exposure. Time offset at each sampling moment This indicates the exposure duration parameter. This represents the total number of discrete sampling points within the exposure time.

[0064] In the step of back-projecting each pixel in the dynamic pixel position set to the camera coordinate system using the blended depth map and the camera intrinsic parameter matrix to obtain the initial 3D position, the corresponding relationship in the process is as follows:

[0065] ;

[0066] in, Indicates the initial 3D position. Represents the camera intrinsic parameter matrix. Represents two-dimensional pixel coordinates on the image plane homogeneous coordinates;

[0067] In the steps of calculating the initial 3D position corresponding to each discrete sampling moment based on the camera coordinate velocity, and reprojecting it onto the two-dimensional imaging plane to obtain the coordinates of the sampling points corresponding to the motion trajectory, the corresponding relationship in the process is as follows:

[0068] ;

[0069] in, Indicates the first The reprojection coordinates of each sampling time point on the image plane This represents the projection mapping function from the camera coordinate system to the image plane.

[0070] It should be noted that, Indicates the first The reprojection coordinates of each sampling moment on the image plane, that is, the corresponding coordinates of the sampling points of the motion trajectory.

[0071] Step 2: Construct a fuzzy weighted network. Using the motion velocity and blending depth map in the camera coordinate system as input, predict the contribution weight of each sampling point on the sampling trajectory. Then, use the contribution weight to perform a weighted integral on the clear rendered image along the sampling trajectory to synthesize a physically blurred image.

[0072] In step 2, a blur weighted network is constructed. Taking the motion velocity and blending depth map in the camera coordinate system as input, the contribution weight of each sampling point on the sampling trajectory is predicted. The contribution weight is then used to perform a weighted integral on the sharp rendered image along the sampling trajectory to synthesize a physically blurred image. Specifically, the following steps are included:

[0073] A fuzzy weighted network is constructed; wherein the fuzzy weighted network is a multilayer perceptron containing two fully connected hidden layers, each with 128 neurons, and the activation function is ELU;

[0074] The motion velocity of each dynamic pixel in the camera coordinate system is concatenated with the blended depth map to form a 4-dimensional input feature vector.

[0075] A 4-dimensional input feature vector is input into a fuzzy weight network. After the fuzzy weight network outputs, a Softmax operation is applied to map the fuzzy weight network output into a normalized weight vector.

[0076] For each dynamic pixel, color values ​​are sampled from the sharp-rendered image and weighted integrals are performed using the normalized weight vector and the coordinates of the sampling point corresponding to the motion trajectory to synthesize a physically blurred image.

[0077] A 4-dimensional input feature vector is fed into a fuzzy weight network. After the fuzzy weight network outputs, a Softmax operation is applied to map the output of the fuzzy weight network into a normalized weight vector. The corresponding relationship in this process is as follows:

[0078]

[0079] ;

[0080] in, Indicates the first The fuzzy contribution weights corresponding to each sampling time point This indicates the total number of discrete sampling points within the exposure time. The weight vector, Represents the normalization function. This represents the forward propagation function of a fuzzy weighted network.

[0081] For each dynamic pixel, color values ​​are sampled from the sharp rendered image and weighted integrals are performed using the normalized weight vector and the coordinates of the sampling points corresponding to the motion trajectory to synthesize a physically blurred image. The corresponding relationship in this process is as follows:

[0082] ;

[0083] in, Indicates the first from the sharpened rendered image Reprojection coordinates on the image plane at each sampling time The color value sampled at that location, Represents the synthesized physically blurred image Two-dimensional pixel coordinates on the image plane The color value at that location.

[0084] Step 3: Generate a pseudo-sharp image from the original blurred input image using a pre-trained deblurring network; fuse the pseudo-sharp image with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; construct a sharpness reconstruction loss based on the hybrid pseudo-ground value image to constrain the sharp rendering image, and construct a re-blurring loss based on the physically blurred image and the original blurred input image.

[0085] Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

[0086] In step 3, a pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image is then fused with the original blurred input image using the dynamic region mask to obtain a hybrid pseudo-ground value image; a sharpness reconstruction loss is constructed based on the hybrid pseudo-ground value image to constrain the sharp rendered image, and a re-blurring loss is constructed based on the physically blurred image and the original blurred input image. Specifically, this includes the following steps:

[0087] A pre-trained deblurring network is used to generate a pseudo-sharp image from the original blurred input image; based on a dynamic region mask, the pseudo-sharp image is fused with the original blurred input image element by element to obtain a hybrid pseudo-ground image.

[0088] Based on the sharp rendered image and the mixed pseudo-real image, a sharpness reconstruction loss is constructed to constrain the sharp rendered image;

[0089] A re-blurring loss is constructed based on the physically blurred image and the original blurred input image.

[0090] It should be noted that during the construction of the heavy fuzzy loss, a gradient decoupling strategy is implemented for the static region to prevent the static geometry from being incorrectly optimized.

[0091] Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by geometry-motion regularization constraints, the parameters in the 4D Gaussian representation, the parameters of the fuzzy weight network, and the parameters of the velocity MLP network are jointly optimized, specifically including the following steps:

[0092] Based on the rendered depth map and monocular depth prior, a depth consistency loss is constructed;

[0093] The object-level coarse velocity true value is obtained from the displacement estimation of the center of the dynamic object; based on the camera coordinate velocity and the object-level coarse velocity true value, adaptive rigid body velocity constraints are calculated.

[0094] The time smoothing regularization loss is calculated based on the normalized weight vector.

[0095] Based on sharpness reconstruction loss, re-blurring loss, depth consistency loss, adaptive rigid body velocity constraint and time smoothing regularization loss, and by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate, a weighted joint approach is adopted to optimize the parameters of the 4D Gaussian representation, the parameters of the fuzzy weight network and the parameters of the velocity MLP network.

[0096] It should be noted that the pre-trained deblurring network used in this embodiment is NAFNet, which is an existing technology, and its specific structure will not be described in detail here.

[0097] The original blurred input image is extracted from a multi-view motion-blurred video sequence; a pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image and the original blurred input image are fused element-wise according to a dynamic region mask to obtain a hybrid pseudo-ground image. The corresponding relationship in this process is as follows:

[0098] ;

[0099] in, This represents the fused hybrid pseudo-true image. This indicates a pre-trained deblurring network. This represents the original blurred input image extracted from a multi-view motion-blurred video sequence; This represents the element-wise multiplication operation;

[0100] In the step of constructing the sharpness reconstruction loss based on the sharp rendered image and the mixed pseudo-ground image, the corresponding relationship in the process is as follows:

[0101]

[0102] in, This represents the L1 norm loss, used to measure the absolute difference in pixel intensity between two images; The structural similarity index measures the similarity between two images in terms of brightness, contrast, and structure. The weighting coefficients that balance L1 loss and SSIM loss; This indicates the loss in sharpness reconstruction;

[0103] In the step of constructing the reblurring loss based on the physically blurred image and the original blurred input image, the corresponding process has the following relationship:

[0104] ;

[0105] in, This represents the original blurred input image; This indicates element-wise multiplication; Indicates heavy fuzziness loss;

[0106] It should be noted that dynamic region masking Ensure that the loss is calculated only within the dynamic pixel area;

[0107] In calculating the heavy fuzzy loss Subsequently, a gradient decoupling strategy is introduced during backpropagation. Specifically, this is achieved by setting the gradient to zero for static regions through gradient truncation, allowing gradients to be applied only to dynamic regions via the mask. The gradients within the defined dynamic region are backpropagated to the optimizable parameters. The core of this strategy is that the fuzzy fitting of the dynamic region is achieved only by adjusting the motion parameters and the 4D Gaussian representation of the dynamic foreground, while the geometric structure of the static background (such as position, covariance, etc.) is not affected by the dynamic fuzzy loss. This avoids the model distorting the static background to fit the dynamic fuzz, fundamentally decoupling the optimization ambiguity between motion and geometry, and ensuring the geometric integrity and stability of the static scene.

[0108] In the step of constructing the depth consistency loss based on the rendered depth map and monocular depth prior, the corresponding process has the following relationship:

[0109] ;

[0110] in, This represents the depth map obtained by rendering the current 4D Gaussian model using differentiable rasterization. This represents the depth prior map extracted through a pre-trained monocular depth estimation network (such as Depth-Anything V2); It represents a very small positive integer, used to avoid numerical instability caused by taking the logarithm of zero; Represents the natural logarithm function; This represents the depth consistency loss;

[0111] In the steps of obtaining the object-level coarse velocity true value from the displacement estimation of the center of the dynamic object, and calculating the adaptive rigid body velocity constraint based on the camera coordinate velocity and the object-level coarse velocity true value, the corresponding process has the following relationship:

[0112] ;

[0113] in, This represents the pixel-by-pixel motion velocity predicted by the model in the camera coordinate system. This represents the object-level coarse velocity true value estimated from the displacement of the center of a dynamic object, obtained by applying a dynamic mask. The displacement of the object's center within the frame is calculated between adjacent frames; This represents the adaptive rigid body velocity constraint loss;

[0114] In the step of calculating the temporal smoothing regularization loss based on the normalized weight vector, the corresponding process has the following relationship:

[0115] ;

[0116] in, Indicates the first The fuzzy contribution weights corresponding to each sampling time point Denotes the square of the L2 norm. This represents the time-smoothing regularization loss.

[0117] There is also a total loss function, and the corresponding process has the following relationship:

[0118] ;

[0119] in, Represents the total loss function. , , , , These represent the corresponding sharpness reconstruction losses. Heavy fuzzy loss Depth consistency loss Adaptive rigid body velocity constraint loss Time smoothing regularization loss The weighting coefficients are used to balance the contribution of each loss to the overall optimization objective.

[0120] It should be noted that the total loss function As the overall optimization objective, the Adam optimizer is used to perform end-to-end joint optimization of the parameters in the 4D Gaussian representation (including the spatial location of each Gaussian primitive, covariance matrix, opacity, and color coefficients), blur weight network parameters, velocity MLP network parameters, adaptive exposure time parameters, and depth fusion coefficients, iterating a preset number of times until the loss converges. After optimization, a clear and high-quality 4D dynamic Gaussian scene representation is obtained. This representation can render clear images at any viewpoint and at any time, and supports downstream applications such as new viewpoint compositing and high frame rate temporal interpolation.

[0121] Please see Figure 3 This invention also proposes a dynamic scene deblurring system based on 4D Gaussian and pseudo-truth value supervision, the system comprising:

[0122] The scene representation and trajectory modeling module is used for:

[0123] The system takes a multi-view motion-blurred video sequence as input, with each frame serving as the original blurred input image. A 4D Gaussian representation of the dynamic scene is constructed based on the multi-view motion-blurred video sequence. A sharp rendered image and a rendered depth map are obtained from the 4D Gaussian representation. A pre-trained instance segmentation model is used to obtain a dynamic region mask from the original blurred input image, and dynamic pixels are defined based on the dynamic region mask. A monocular depth prior is obtained, and the rendered depth map is fused with the monocular depth prior to construct a hybrid depth map. Based on the rendered depth map, a velocity MLP network is used to predict the motion velocity of the dynamic pixels in the camera coordinate system, thus obtaining the sampling trajectory of the dynamic pixels within the exposure time.

[0124] The fuzzy weight prediction and fuzzy synthesis module is used for:

[0125] A fuzzy weighted network is constructed, taking the motion velocity and blending depth map in the camera coordinate system as input, predicting the contribution weight of each sampling point on the sampling trajectory, and using the contribution weight to perform weighted integration on the sharp rendered image along the sampling trajectory to synthesize a physically blurred image.

[0126] The optimization module is used for:

[0127] A pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image is fused with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; a sharpness reconstruction loss is constructed based on the hybrid pseudo-ground value image to constrain the sharp rendered image, and a re-blurring loss is constructed based on the physically blurred image and the original blurred input image;

[0128] Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

[0129] To quantitatively evaluate the effectiveness of the proposed method in dynamic scene deblurring and geometric reconstruction tasks, comparative experiments were conducted on the publicly available multi-view dynamic blur dataset DynaMoDe-NeRF. The mainstream dynamic reconstruction methods Swift4D, DeGauss, and 4DGS were selected as baselines, and the SSIM, PSNR, and LPIPS metrics were used to evaluate the results in both the overall and dynamic regions. The experimental results are shown in Table 1, with the best results highlighted in bold.

[0130] Table 1: Quantitative evaluation results of the method of the present invention on dynamic region reconstruction and fuzzy synthesis tasks.

[0131]

[0132]

[0133] As shown in Table 1, the method of this invention outperforms existing baseline methods in terms of PSNR and SSIM in all four scenes, especially in terms of LPIPS in dynamic regions, which is reduced by an average of more than 35%, indicating that this method can more accurately recover the high-frequency texture and edge structure of moving objects. In terms of overall region metrics, this method achieves the best PSNR and SSIM values ​​in the Monkey, Sphere, and Cube scenes, and performs comparably to 4DGS in the Lychee scene. The above results verify the effectiveness and superiority of the proposed 4D Gaussian scene representation, fuzzy weight network, hybrid pseudo-ground value supervision, and gradient decoupling strategy in dynamic scene deblurring tasks.

[0134] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0135] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0136] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision, characterized in that, The method includes the following steps: Step 1: Input a multi-view motion-blurred video sequence, where each frame is used as the original blurred input image; construct a 4D Gaussian representation of the dynamic scene based on the multi-view motion-blurred video sequence; render a sharp rendered image and a rendered depth map from the 4D Gaussian representation; obtain a dynamic region mask from the original blurred input image using a pre-trained instance segmentation model, and define dynamic pixels based on the dynamic region mask; obtain a monocular depth prior, and fuse the rendered depth map with the monocular depth prior to construct a hybrid depth map; Based on the rendered depth map, the velocity MLP network is used to predict the motion velocity of dynamic pixels in the camera coordinate system, so as to obtain the sampling trajectory of dynamic pixels during the exposure time. Step 2: Construct a fuzzy weighted network. Using the motion velocity and blending depth map in the camera coordinate system as input, predict the contribution weight of each sampling point on the sampling trajectory, and use the contribution weight to perform weighted integration on the clear rendering image along the sampling trajectory to synthesize a physically blurred image. Step 3: Generate a pseudo-sharp image from the original blurred input image using a pre-trained deblurring network; fuse the pseudo-sharp image with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; construct a sharpness reconstruction loss based on the hybrid pseudo-ground value image to constrain the sharp rendering image, and construct a re-blurring loss based on the physically blurred image and the original blurred input image. Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

2. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 1, characterized in that, In step 1, a 4D Gaussian representation of the dynamic scene is constructed based on the multi-view motion-blurred video sequence; a sharp rendered image and a rendered depth map are rendered from the 4D Gaussian representation; a dynamic region mask is obtained from the original blurred input image using a pre-trained instance segmentation model, and dynamic pixels are defined based on the dynamic region mask; a monocular depth prior is obtained, and the rendered depth map is fused with the monocular depth prior to construct a hybrid depth map; Based on the rendered depth map, a velocity MLP network is used to predict the motion velocity of dynamic pixels in the camera coordinate system, so as to obtain the sampling trajectory of dynamic pixels during the exposure time. The specific steps include the following: A motion structure recovery algorithm is used to process multi-view motion-blurred video sequences to recover the camera pose parameters and sparse 3D point cloud of the scene corresponding to each frame; where each frame of the view motion-blurred video sequence is used as the original blurred input image. Based on the camera pose parameters and the sparse 3D point cloud of the scene, the dynamic scene is parameterized into a set of 4D Gaussian elements with time attributes. Using the primitives in the four-dimensional Gaussian primitive set, a 4D Gaussian representation of the dynamic scene is constructed. The 4D Gaussian representation of the dynamic scene is then rendered by differentiable rasterization to obtain a clear rendered image. The 4D Gaussian representation of the dynamic scene is then rendered by differentiable rasterization again to obtain a rendering depth map. Using a pre-trained instance segmentation and tracking model, foreground moving object segmentation is performed on each frame of a multi-view motion-blurred video sequence to generate a dynamic region mask; Dynamic pixel locations are distinguished using dynamic region masks to obtain a set of dynamic pixel locations; A pre-trained monocular depth estimation network is used to extract monocular depth priors from multi-view motion-blurred video sequences, and the monocular depth priors are fused with the rendered depth map through learnable fusion coefficients to construct a hybrid depth map. Given a camera pose, differentiable rasterization rendering or hybrid depth back projection calculation is performed using the 4D Gaussian representation of the dynamic scene to obtain the world coordinates corresponding to the dynamic pixels. The world coordinates and blended depth map corresponding to the dynamic pixels are input into the velocity MLP network to regress the instantaneous motion velocity in the world coordinate system in order to obtain the motion vector in the world coordinate system. Transform the motion vector in the world coordinate system to the camera coordinate system of the current viewpoint to obtain the camera coordinate velocity; Discretize the exposure time to obtain the discrete sampling time; By using the blended depth map and the camera intrinsic matrix, each pixel in the dynamic pixel position set is back-projected onto the camera coordinate system to obtain the initial 3D position; Based on the camera coordinate velocity, the initial 3D position corresponding to each discrete sampling moment is calculated and reprojected onto the two-dimensional imaging plane to obtain the sampling point coordinates corresponding to the motion trajectory.

3. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 2, characterized in that, Based on the camera pose parameters and the sparse 3D point cloud of the scene, the dynamic scene is parameterized into a set of four-dimensional Gaussian elements with temporal attributes. The corresponding relationship in the process is as follows: ； in, Describe the covariance matrix. This represents the rotation moments obtained from learnable quaternions through orthogonalization. This represents the diagonal scaling matrix constructed from the three-dimensional scaling parameters via an exponential mapping. This represents the matrix transpose operator; In the step of using a dynamic region mask to distinguish the positions of dynamic pixels in order to obtain a set of dynamic pixel positions, the corresponding relationship in the process is as follows: ； in, This represents the set of pixel locations that are all marked as dynamic regions. Represents the two-dimensional pixel coordinates on the image plane. Represents a two-dimensional real number space. Indicates a dynamic region mask; In the steps of extracting monocular depth priors from multi-view motion-blurred video sequences using a pre-trained monocular depth estimation network, and fusing the monocular depth priors with the rendered depth map through learnable fusion coefficients to construct a hybrid depth map, the corresponding relationships in the process are as follows: ； in, Represents a mixed depth map. Indicates the blending depth value. Represents the learnable fusion coefficient. This represents the relative depth map extracted from the original blurred frame using a pre-trained monocular depth estimation network. This represents the geometric depth map obtained by rendering the current 4D Gaussian model through a differentiable rasterization pipeline.

4. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 3, characterized in that, The world coordinates and blended depth map corresponding to the dynamic pixels are input into the velocity MLP network to regress the instantaneous motion velocity in the world coordinate system, so as to obtain the motion vector in the world coordinate system. The corresponding relationship in the process is as follows: ； in, This represents the velocity vector in the world coordinate system. This refers to a lightweight multilayer perceptron. Represents the three-dimensional coordinates of the world coordinate system; In the step of transforming the motion vector from the world coordinate system to the camera coordinate system of the current viewpoint to obtain the camera coordinate velocity, the corresponding relationship in the process is as follows: ； in, This represents the rotation matrix extracted from the camera extrinsic parameters of the current frame. The camera coordinate system representing the current viewpoint; In the step of discretizing the exposure time, the corresponding relationship in the process is as follows: ； in, Indicates the first exposure based on the center of exposure. Time offset at each sampling moment This indicates the exposure duration parameter. This represents the total number of discrete sampling points within the exposure time. In the step of back-projecting each pixel in the dynamic pixel position set to the camera coordinate system using the blended depth map and the camera intrinsic parameter matrix to obtain the initial 3D position, the corresponding relationship in the process is as follows: ； in, Indicates the initial 3D position. Represents the camera intrinsic parameter matrix. Represents two-dimensional pixel coordinates on the image plane homogeneous coordinates; In the steps of calculating the initial 3D position corresponding to each discrete sampling moment based on the camera coordinate velocity, and reprojecting it onto the two-dimensional imaging plane to obtain the coordinates of the sampling points corresponding to the motion trajectory, the corresponding relationship in the process is as follows: ； in, Indicates the first The reprojection coordinates of each sampling time point on the image plane This represents the projection mapping function from the camera coordinate system to the image plane.

5. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 4, characterized in that, In step 2, a blur weighted network is constructed. Taking the motion velocity and blending depth map in the camera coordinate system as input, the contribution weight of each sampling point on the sampling trajectory is predicted. The contribution weight is then used to perform a weighted integral on the sharp rendered image along the sampling trajectory to synthesize a physically blurred image. Specifically, the following steps are included: A fuzzy weighted network is constructed; wherein the fuzzy weighted network is a multilayer perceptron containing two fully connected hidden layers, each with 128 neurons, and the activation function is ELU; The motion velocity of each dynamic pixel in the camera coordinate system is concatenated with the blended depth map to form a 4-dimensional input feature vector. A 4-dimensional input feature vector is input into a fuzzy weight network. After the fuzzy weight network outputs, a Softmax operation is applied to map the fuzzy weight network output into a normalized weight vector. For each dynamic pixel, color values ​​are sampled from the sharp-rendered image and weighted integrals are performed using the normalized weight vector and the coordinates of the sampling point corresponding to the motion trajectory to synthesize a physically blurred image.

6. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 5, characterized in that, A 4-dimensional input feature vector is fed into a fuzzy weight network. After the fuzzy weight network outputs, a Softmax operation is applied to map the output of the fuzzy weight network into a normalized weight vector. The corresponding relationship in this process is as follows: ； ； in, Indicates the first The fuzzy contribution weights corresponding to each sampling time point This indicates the total number of discrete sampling points within the exposure time. The weight vector, Represents the normalization function. This represents the forward propagation function of a fuzzy weighted network; For each dynamic pixel, color values ​​are sampled from the sharp rendered image and weighted integrals are performed using the normalized weight vector and the coordinates of the sampling points corresponding to the motion trajectory to synthesize a physically blurred image. The corresponding relationship in this process is as follows: ； in, Indicates the first from the sharpened rendered image Reprojection coordinates on the image plane at each sampling time The color value sampled at that location, Represents the synthesized physically blurred image Two-dimensional pixel coordinates on the image plane The color value at that location.

7. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 6, characterized in that, In step 3, a pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image is then fused with the original blurred input image using the dynamic region mask to obtain a hybrid pseudo-ground value image; a sharpness reconstruction loss is constructed based on the hybrid pseudo-ground value image to constrain the sharp rendered image, and a re-blurring loss is constructed based on the physically blurred image and the original blurred input image. Specifically, this includes the following steps: A pre-trained deblurring network is used to generate a pseudo-sharp image from the original blurred input image; based on a dynamic region mask, the pseudo-sharp image is fused with the original blurred input image element by element to obtain a hybrid pseudo-ground image. Based on the sharp rendered image and the mixed pseudo-real image, a sharpness reconstruction loss is constructed to constrain the sharp rendered image; Based on the physically blurred image and the original blurred input image, a re-blurring loss is constructed. At the same time, during the construction of the re-blurring loss, a gradient decoupling strategy is implemented for static regions to prevent static geometry from being incorrectly optimized.

8. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 7, characterized in that, Based on sharpness reconstruction loss and reblurring loss, and supplemented by geometry-motion regularization constraints, the parameters of the 4D Gaussian representation, the blur weight network parameters, and the velocity MLP network parameters are jointly optimized by truncating the gradients in static regions and allowing only the backpropagation of gradients in dynamic regions. Specifically, the following steps are included: Based on the rendered depth map and monocular depth prior, a depth consistency loss is constructed; The object-level coarse velocity true value is obtained from the displacement estimation of the center of the dynamic object; based on the camera coordinate velocity and the object-level coarse velocity true value, adaptive rigid body velocity constraints are calculated. The time smoothing regularization loss is calculated based on the normalized weight vector. Based on sharpness reconstruction loss, re-blurring loss, depth consistency loss, adaptive rigid body velocity constraint and time smoothing regularization loss, and by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate, a weighted joint approach is adopted to optimize the parameters of the 4D Gaussian representation, the parameters of the fuzzy weight network and the parameters of the velocity MLP network.

9. The dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision according to claim 8, characterized in that, The original blurred input image is extracted from a multi-view motion-blurred video sequence; a pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image and the original blurred input image are fused element-wise according to a dynamic region mask to obtain a hybrid pseudo-ground image. The corresponding relationship in this process is as follows: ； in, This represents the fused hybrid pseudo-true image. This indicates a pre-trained deblurring network. This represents the original blurred input image extracted from a multi-view motion-blurred video sequence; This represents the element-wise multiplication operation; In the step of constructing the sharpness reconstruction loss based on the sharp rendered image and the mixed pseudo-ground image, the corresponding relationship in the process is as follows: ； in, Indicates L1 norm loss, Represents the structural similarity index. This represents the weighting coefficients that balance the L1 loss and the SSIM loss. This indicates the loss in sharpness reconstruction; In the step of constructing the reblurring loss based on the physically blurred image and the original blurred input image, the corresponding process has the following relationship: ； in, This represents the original blurred input image. This indicates element-wise multiplication. Indicates heavy fuzziness loss; In the step of constructing the depth consistency loss based on the rendered depth map and monocular depth prior, the corresponding process has the following relationship: ； in, This represents the depth map obtained by rendering the current 4D Gaussian model using differentiable rasterization. This represents the depth prior map extracted using a pre-trained monocular depth estimation network. To represent a very small positive number, Represents the natural logarithm function. This represents the depth consistency loss; In the steps of obtaining the object-level coarse velocity true value from the displacement estimation of the center of the dynamic object, and calculating the adaptive rigid body velocity constraint based on the camera coordinate velocity and the object-level coarse velocity true value, the corresponding process has the following relationship: ； in, This represents the pixel-by-pixel motion velocity predicted by the model in the camera coordinate system. This represents the object-level coarse velocity true value estimated from the displacement of the center of a dynamic object. This represents the adaptive rigid body velocity constraint loss; In the step of calculating the temporal smoothing regularization loss based on the normalized weight vector, the corresponding process has the following relationship: ； in, Indicates the first The fuzzy contribution weights corresponding to each sampling time point Denotes the square of the L2 norm. This represents the time-smoothing regularization loss.

10. A dynamic scene deblurring system based on 4D Gaussian and pseudo-truth value supervision, characterized in that, The system employs the dynamic scene deblurring method based on 4D Gaussian and pseudo-truth value supervision as described in any one of claims 1-9, and the system comprises: The scene representation and trajectory modeling module is used for: The system takes a multi-view motion-blurred video sequence as input, with each frame serving as the original blurred input image. A 4D Gaussian representation of the dynamic scene is constructed based on the multi-view motion-blurred video sequence. A sharp rendered image and a rendered depth map are obtained from the 4D Gaussian representation. A pre-trained instance segmentation model is used to obtain a dynamic region mask from the original blurred input image, and dynamic pixels are defined based on the dynamic region mask. A monocular depth prior is obtained, and the rendered depth map is fused with the monocular depth prior to construct a hybrid depth map. Based on the rendered depth map, a velocity MLP network is used to predict the motion velocity of the dynamic pixels in the camera coordinate system, thus obtaining the sampling trajectory of the dynamic pixels within the exposure time. The fuzzy weight prediction and fuzzy synthesis module is used for: A fuzzy weighted network is constructed, taking the motion velocity and blending depth map in the camera coordinate system as input, predicting the contribution weight of each sampling point on the sampling trajectory, and using the contribution weight to perform weighted integration on the sharp rendered image along the sampling trajectory to synthesize a physically blurred image. The optimization module is used for: A pseudo-sharp image is generated from the original blurred input image using a pre-trained deblurring network; the pseudo-sharp image is fused with the original blurred input image according to the dynamic region mask to obtain a hybrid pseudo-ground value image; a sharpness reconstruction loss is constructed based on the hybrid pseudo-ground value image to constrain the sharp rendered image, and a re-blurring loss is constructed based on the physically blurred image and the original blurred input image; Based on the sharpness reconstruction loss and the re-blurring loss, and supplemented by the geometry-motion regularization constraint, the parameters in the 4D Gaussian representation, the parameters of the blur weight network, and the parameters of the velocity MLP network are jointly optimized by truncating the gradient of the static region and allowing only the gradient of the dynamic region to backpropagate. After a preset number of iterations, the algorithm converges to obtain a sharp dynamic scene representation.

Images