Motion deblurring method and system based on taylor expansion and 3d gaussian sputtering

Abstract

This invention proposes a motion deblurring method and system based on Taylor expansion and 3D Gaussian sputtering. The method includes: extracting the dynamic scene from a multi-view motion-blurred video sequence and parameterizing it into an optimizable time-varying 3D Gaussian sputtering representation; constructing a motion model based on Taylor expansion; obtaining an initial velocity field by differentiating and projecting the Taylor expansion motion model, and then correcting it to obtain a corrected 2D pixel-level velocity field; synthesizing a synthetic physically blurred image consistent with the input video frame by simulating the physical imaging process during camera exposure; finally, training and optimization are performed, and after optimization, a final 3D Gaussian scene representation is obtained. This final 3D Gaussian scene representation is then used to render a clear image, thus achieving motion deblurring. This invention improves the physical interpretability and accuracy of motion modeling by explicitly modeling continuous motion trajectories using Taylor expansion.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction, and in particular to a motion deblurring method and system based on Taylor expansion and 3D Gaussian sputtering. Background Technology

[0002] Motion blur is a common problem in dynamic scene imaging, especially when shooting handheld cameras or fast-moving objects. How to recover clear temporal information from blurred videos while maintaining scene geometric consistency has always been a research challenge in computer vision and 3D reconstruction. Traditional deblurring methods are mostly based on 2D image processing and lack explicit modeling of the coupling relationship between 3D geometry and motion, easily leading to texture distortion and geometric warping.

[0003] In recent years, methods based on Neural Radiation Field (NeRF) and 3D Gaussian Sputtering (3DGS) have made significant progress in dynamic scene reconstruction. However, most existing methods assume that the input is a sharp image, making it difficult to directly process blurry videos; or they employ implicit motion modeling, which lacks physical interpretability and is prone to optimization ambiguities caused by geometry-motion coupling.

[0004] Therefore, a dynamic deblurring method that can explicitly model the physical blurring process, combine higher-order motion priors, and possess strong geometric consistency is needed to achieve high-quality reconstruction from blurred videos to clear 3D dynamic scenes. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a motion deblurring method and system based on Taylor expansion and 3D Gaussian sputtering to solve the above-mentioned technical problems.

[0006] This invention proposes a motion deblurring method based on Taylor expansion and 3D Gaussian sputtering, the method comprising the following steps:

[0007] Step 1: Extract the dynamic scene from the multi-view motion-blurred video sequence and parameterize the dynamic scene into an optimizable time-varying 3D Gaussian sputtering representation;

[0008] Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties;

[0009] Step 2: Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field.

[0010] Step 3: Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage;

[0011] Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure.

[0012] Step 4: Construct a total loss function that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions based on the synthesized physically blurred image consistent with the input video frame and the clear rendered image. Iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a clear image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

[0013] This invention also proposes a motion deblurring system based on Taylor expansion and 3D Gaussian sputtering, wherein the system applies the motion deblurring method based on Taylor expansion and 3D Gaussian sputtering as described above, and the system includes:

[0014] The motion modeling module is used for:

[0015] Extract dynamic scenes from multi-view motion-blurred video sequences and parameterize the dynamic scenes into an optimizable time-varying 3D Gaussian sputtering representation;

[0016] Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties;

[0017] The model derivation module is used for:

[0018] Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field.

[0019] Image synthesis module, used for:

[0020] Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage;

[0021] Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure.

[0022] The iterative optimization module is used for:

[0023] Based on the synthesized physically blurred image consistent with the input video frame and the sharp rendered image, a total loss function is constructed that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions. This function is used to iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a sharp image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] This invention improves the physical interpretability and accuracy of motion modeling by explicitly modeling continuous motion trajectories through Taylor expansion; it achieves geometrically consistent motion deblurring by combining 3D Gaussian sputtering with mixed geometric context; it introduces learnable physical blur synthesis to accurately simulate the imaging process during exposure; and it verifies the superior performance of this invention on dynamic deblurring tasks on multiple public datasets, significantly improving sharpness and reconstruction quality.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0027] Figure 1 The flowchart below shows the motion deblurring method based on Taylor expansion and 3D Gaussian sputtering proposed in this invention.

[0028] Figure 2 This is a schematic diagram of the overall framework of the method of the present invention.

[0029] Figure 3 This is a diagram of the Peano remainder correction network architecture of the method of the present invention.

[0030] Figure 4 This is a schematic diagram of the motion deblurring system based on Taylor expansion and 3D Gaussian sputtering proposed in this invention. Detailed Implementation

[0031] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0032] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0033] Please see Figure 1 and Figure 2 This embodiment provides a motion deblurring method based on Taylor expansion and 3D Gaussian sputtering, the method including the following steps:

[0034] Step 1: Extract the dynamic scene from the multi-view motion-blurred video sequence and parameterize the dynamic scene into an optimizable time-varying 3D Gaussian sputtering representation;

[0035] Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties;

[0036] In a preferred embodiment of the present invention, the extraction of scene dynamic parameters from a multi-view motion-blurred video sequence specifically includes the following steps:

[0037] Based on multi-view motion-blurred video sequences, sparse point clouds and camera extrinsic parameters of each frame are extracted using structure from motion (SfM) techniques, such as COLMAP.

[0038] The center location and covariance matrix are calculated based on the sparse point cloud.

[0039] The viewpoint-related color is calculated based on the camera's extrinsic parameters and the sparse point cloud.

[0040] Given opacity; scene dynamics are composed of center position, covariance matrix, opacity, and viewpoint-dependent color.

[0041] In this embodiment, the sparse point cloud provides the initial geometry of the scene, and the camera extrinsic parameters provide the position and orientation of each frame in the world coordinate system.

[0042] By directly using the world coordinates of each point in the sparse point cloud as the center position of the initial Gaussian unit;

[0043] The covariance matrix is ​​obtained by calculating the average distance from each sparse point cloud to its k nearest neighbors and initializing an isotropic covariance matrix based on the average distance.

[0044] Initialize the opacity of all Gaussian elements to a constant close to 0 (e.g., 0.1), and represent it as opacity;

[0045] Each point in the sparse point cloud is assigned a color value, which is then assigned to the zeroth-order coefficient of the spherical harmonic function to obtain the viewpoint-dependent color.

[0046] Camera intrinsic parameters (focal length, principal point, distortion coefficients) are obtained through calibration during the motion structure restoration process or read from video metadata; camera extrinsic parameters (rotation matrix and translation vector) are estimated by motion structure restoration techniques and represent the pose of each frame in the world coordinate system.

[0047] In a preferred embodiment of the present invention, the dynamic scene is parameterized into an optimizable time-varying 3D Gaussian sputtering representation, specifically as follows:

[0048] A set of 3D Gaussian primitives is constructed using dynamic scene parameters, and a time dimension is introduced into the 3D Gaussian primitives to form a set of 4D Gaussian primitives that change over time. This yields an optimizable time-varying 3D Gaussian sputtering representation; each 4D Gaussian element consists of a time-varying center position. Covariance matrix Opacity Viewpoint-related colors definition;

[0049] covariance matrix By rotation matrix With scaling matrix The definition and the corresponding process have the following relationship:

[0050] ;

[0051] in, This indicates the transpose operation. Represents the scaling matrix. Represents the rotation matrix;

[0052] Rotation matrix Learnable quaternions (Or rotation vector) representation, converted to a 3×3 rotation matrix through orthogonalization, controls the orientation of 4D Gaussian elements in three-dimensional space, rotation matrix The optimization process involves updating the scaling matrix using gradient descent. From learnable 3D scaling vectors The scaling factor is determined by an exponential mapping to ensure a positive scaling value, and its formula is: ;

[0053] in, This indicates that a diagonal matrix is ​​constructed using the elements within the parentheses as its diagonal. These represent the learnable scaling parameters of the Gaussian element in the three orthogonal axes, used to control the size of the 4D Gaussian element in the three axes.

[0054] In a preferred embodiment of the present invention, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties according to the optimizable time-varying 3D Gaussian sputtering representation, specifically as follows:

[0055] Utilizing in reference time The nth-order Taylor series expanded at a given point models the center position of the 4D Gaussian element, and the corresponding process has the following relationship:

[0056] ;

[0057] in, This represents the learnable k-th order Taylor coefficients. Let n represent the Peano remainder used to capture nonlinear residuals, and n represent the order of the Taylor expansion. The center position of the 4D Gaussian primitive is represented by a three-dimensional vector function with respect to time t, which describes the trajectory of the current 4D Gaussian primitive in three-dimensional space as time changes. This represents the factorial of k, and t represents the current time and reference time. For each 4D Gaussian primitive, the parameters are independently learnable, representing the time reference point for the Taylor expansion of the current 4D Gaussian primitive's own properties (including position, opacity, rotation, scaling, etc.). During the optimization process, Compared with other Taylor coefficients Together, they are updated through gradient descent, thereby adaptively adjusting the motion modeling center of each 4D Gaussian primitive, improving the flexibility and accuracy of motion trajectory fitting.

[0058] Step 2: Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field.

[0059] As a preferred embodiment of the present invention, based on the Taylor expansion motion model, the initial velocity field is obtained through analytical differentiation and differentiable projection, and the initial velocity field is corrected by residual using a neural network to obtain a corrected 2D pixel-level velocity field. Specifically, the following steps are included:

[0060] Differentiating the polynomial of the center position of the 4D Gaussian element defined by Taylor coefficients yields the basic 3D velocity field. The corresponding process has the following relationship:

[0061] ;

[0062] in, This represents the basic 3D velocity field at time t. Represents the (k+1)th order Taylor coefficients. Taylor polynomial approximation of the center position of the 4D Gaussian element. ;

[0063] By using differentiable velocity rasterization, the opacity of the fundamental 3D velocity field and the 4D Gaussian primitives is combined. and transmittance Combined, projected onto a 2D image plane, to generate a rendering speed map. The corresponding process has the following relationship:

[0064] ;

[0065] in, Represents the basic 3D velocity field. This represents a rendering speed graph. This represents the cumulative transmittance up to the elementary element q. , Indicates the first j The opacity of each 4D Gaussian pixel, ranging from [0, 1], determines the contribution of the current 4D Gaussian pixel to the final pixel color. q This represents the index of the currently processed 4D Gaussian unit, i.e., the first element after sorting by depth. q Four 4D Gaussian elements. During the differentiable rasterization process, the 4D Gaussian elements are ordered from far to near in depth and α-blending is performed sequentially. Indicates the pixel u An ordered set of all contributing 4D Gaussian elements;

[0066] The rendering speed map and the current rendering time are input into the Peano residual correction network to predict the speed residual. ;

[0067] The corrected 2D pixel-level velocity field is obtained by adding the rendered velocity map to the velocity residual. The corresponding process has the following relationship:

[0068] ;

[0069] in, Represents the velocity residual. This represents the corrected 2D pixel-level velocity field.

[0070] like Figure 3 As shown, the Peano remainder correction network employs a multilayer perceptron architecture to compensate for the high-frequency nonlinear motion residuals caused by Taylor expansion truncation. The Peano remainder correction network receives the base velocity as input. The feature is transformed through multiple fully connected layers with time t, and the final output is the velocity residual.

[0071] The Peano residual correction network consists of three hidden layers with feature dimensions of 64, 128, and 64, respectively. All hidden layers are followed by the ReLU activation function, and the output layer is a linear layer (without an activation function) to allow prediction of both positive and negative velocity residuals.

[0072] Step 3: Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage;

[0073] Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure.

[0074] As a preferred embodiment of the present invention, based on an optimizable time-varying 3D Gaussian sputtering representation and a corrected 2D pixel-level velocity field, a synthesized physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure. Specifically, the process includes the following steps:

[0075] Prior depth is extracted from video frames of a multi-view motion-blurred video sequence using a pre-trained monocular depth estimation model.

[0076] Utilizing the optimizable time-varying 3D Gaussian sputtering representation of the current optimization phase, the depth of the current model rendering is generated at a given camera pose through a differentiable rasterization process;

[0077] By using the prior depth and the current model rendering depth, a blended depth map is constructed. The corresponding process has the following relationship:

[0078] ;

[0079] in, Represents the fusion coefficient. Represents a mixed depth map. Denotes the prior depth, prior depth It is extracted from the input video frames by a pre-trained monocular depth estimation model (such as Depth-Anything V2), providing a geometrically stable depth reference. This indicates the current rendering depth of the model. The scene geometry reconstructed by the current 3D Gaussian sputtering representation, which is optimized in the current optimization stage, is rendered and generated under a given camera pose through a differentiable rasterization process.

[0080] The dynamic object mask is extracted from the video, and the dynamic pixel set is defined based on the dynamic object mask. ,in, This represents a dynamic object mask, where u represents a dynamic pixel. Represents a dynamic set of pixels;

[0081] By using the blending depth and camera parameters, dynamic pixels are back-projected into 3D space, and based on the motion information corresponding to the camera coordinate system, the 2D reprojection coordinates at each sampling time are calculated to obtain the sampling coordinates of the blur kernel.

[0082] The camera coordinate system velocity and blending depth map are input into the blur weight network to predict the contribution weight of each sampling point to the final blur color.

[0083] The process of generating a sharp rendered image from the currently optimizable time-varying 3D Gaussian sputtering representation through differentiable rasterization rendering follows the following relationship:

[0084] ;

[0085] in, Represents pixels u The clear color in the area, Indicates the first q The color of a 4D high-resolution pixel. Indicates the first q The opacity of a 4D high-resolution element;

[0086] By using the contribution weights of the final blurred color to perform weighted integration on the sharp rendered image at the sampling coordinates of the blur kernel, a synthetic physically blurred image consistent with the input video frame is synthesized.

[0087] In a preferred embodiment of the present invention, dynamic pixels are back-projected into 3D space using blending depth and camera parameters, and the 2D reprojection coordinates corresponding to each sampling time are calculated based on the motion information in the camera coordinate system to obtain the sampling coordinates of the blur kernel. The specific steps include the following:

[0088] The motion trajectory of dynamic pixels during the exposure time is discretized into N samples, and the time offset of each sample is calculated. The corresponding process has the following relationship:

[0089] ;

[0090] in, Indicates the exposure time. i Indicates the first i One sample, N represents the total number of samples. Indicates the first i The time offset of each sample; N samples are used to discretize the continuous motion trajectory within the exposure time window. Each sample corresponds to a timestamp, which is used to calculate the 3D position and 2D reprojection coordinates at that moment.

[0091] By back-projecting dynamic pixels onto the camera coordinate system using the blending depth and camera intrinsic parameter matrix, the initial 3D position is obtained. The corresponding process has the following relationship:

[0092] ;

[0093] in, This represents the homogeneous coordinates of the dynamic pixel u. This represents the initial 3D position; K represents the camera intrinsic parameter matrix, which is obtained through calibration during the motion structure recovery process or read from video metadata.

[0094] The corrected 2D pixel-level velocity field is rigorously transformed to the camera coordinate system using a view rotation matrix to obtain the camera coordinate system velocity. The corresponding process has the following relationship:

[0095] ;

[0096] in, Indicates the velocity of the camera coordinate system; This represents the view rotation matrix, which is extracted from the rotation part of the camera extrinsic parameters of the current frame and is used to transform the velocity vector in the world coordinate system to the current camera coordinate system.

[0097] Based on the camera coordinate system velocity, the 3D position corresponding to the time offset of each sample is calculated and reprojected back onto the image plane to obtain the sampling coordinates of the blur kernel. The corresponding process has the following relationship:

[0098] ;

[0099] in, The perspective projection function represents the projection from 3D camera coordinates to 2D image coordinates. The sampling coordinates of the blur kernel are used to sample colors from the sharp rendered image in subsequent steps and synthesize a physically blurred image through weighted integration, thereby simulating the motion blur effect during camera exposure.

[0100] In a preferred embodiment of the present invention, the camera coordinate system velocity and the blending depth map are input into the blur weight network to predict the contribution weight of each sampling point to the final blurred color. The corresponding process has the following relationship:

[0101] ;

[0102] in, This indicates the contribution weight of the final blurred color. This indicates that Softmax normalization is performed along the time dimension to ensure... ; This represents a fuzzy weighted network.

[0103] By performing a weighted integral on the sharp rendered image at the sampling coordinates of the blur kernel, a synthesized physically blurred image consistent with the input video frame is synthesized. The process of synthesizing a physically blurred image consistent with the input video frame has the following relationship:

[0104] ;

[0105] in, This represents the color value at the corresponding coordinates of a clearly rendered image. Represents the pixel color of the synthesized physically blurred image that matches the input video frame. This indicates the contribution weight of the final blurred color.

[0106] Step 4: Construct a total loss function that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions based on the synthesized physically blurred image consistent with the input video frame and the clear rendered image. Iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a clear image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

[0107] In a preferred embodiment of the present invention, the total loss function that simultaneously constrains the fuzzy synthesis accuracy of the dynamic region and the reconstruction consistency of the static region has the following relationship:

[0108] ;

[0109] in, Represents the total loss function. Indicates the static region reconstruction loss. This represents the dynamic region re-blurring loss. Represents the dynamic region re-blurring loss Weighting coefficients; Represents static region reconstruction loss Weighting coefficients, weighting coefficients and Used to balance the constraint strength between dynamic and static regions;

[0110] Among them, dynamic region re-blurring loss Used to force the synthesized blurred image in dynamic regions To approximate a realistic fuzzy input, the corresponding process has the following relationship:

[0111] ;

[0112] in, This represents the blurred image synthesized by the model. Represents a realistic blurred input image. Indicates a dynamic region. It is an L1 norm. This indicates element-wise multiplication;

[0113] Among them, static area reconstruction loss A sharp image used to force rendering in static areas. C Approximating realistic fuzzy input Since there is no motion in the static region, its ideal clear image should be consistent with the input blurred image. This loss term effectively prevents the static geometry from being distorted by the error gradient of the dynamic region during the optimization process, ensuring the stability of the background structure. The corresponding process has the following relationship:

[0114] ;

[0115] in, C This indicates a sharp image that is forcibly rendered in a static area. Indicates a static region.

[0116] After the above optimization process, a high-quality, clear 3D Gaussian scene representation is finally obtained. This representation itself is the result of deblurring and also constitutes a powerful intermediate representation. Based on this, the system can render clear images at any specified viewpoint and time, achieving free synthesis of new viewpoints. Furthermore, thanks to the learned continuous Taylor motion model, high frame rate temporal interpolation can be achieved by calculating the Gaussian state at any intermediate moment, generating smooth slow-motion or frame-interpolated videos. If needed, the 3D Gaussians at different times can also be converted into dynamic 3D mesh sequences for a wider range of downstream applications.

[0117] Please refer to Figure 4 This embodiment also provides a motion deblurring system based on Taylor expansion and 3D Gaussian sputtering, wherein the system applies the motion deblurring method based on Taylor expansion and 3D Gaussian sputtering as described above, and the system includes:

[0118] The motion modeling module is used for:

[0119] Extract dynamic scenes from multi-view motion-blurred video sequences and parameterize the dynamic scenes into an optimizable time-varying 3D Gaussian sputtering representation;

[0120] Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties;

[0121] The model derivation module is used for:

[0122] Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field.

[0123] Image synthesis module, used for:

[0124] Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage;

[0125] Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure.

[0126] The iterative optimization module is used for:

[0127] Based on the synthesized physically blurred image consistent with the input video frame and the sharp rendered image, a total loss function is constructed that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions. This function is used to iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a sharp image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

[0128] To verify the effectiveness of our proposed method in deblurring dynamic scenes, we conducted experiments on the publicly available multi-view dynamic blur dataset (DynaMoDe-NeRF) and made quantitative comparisons with current mainstream dynamic reconstruction methods. The results are shown in Table 1, where PSNR and SSIM are positive indicators, with higher values ​​indicating better reconstruction quality; LPIPS are negative indicators, with lower values ​​indicating higher perceptual quality. Experiments show that our proposed method significantly outperforms existing methods in PSNR and SSIM in dynamic regions, and reduces LPIPS by an average of approximately 20% in dynamic regions. This demonstrates that our proposed method, through explicit Taylor expansion modeling of continuous motion trajectories combined with velocity rasterization and residual term correction, can more accurately recover the geometric structure and motion consistency of moving objects, thereby achieving high-quality motion deblurring.

[0129] Table 1. Quantitative comparison results of the method of the present invention and existing technologies on multi-view dynamic deblurring tasks.

[0130]

[0131] Continued from the table above

[0132]

[0133] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A motion deblurring method based on Taylor expansion and 3D Gaussian sputtering, characterized in that, The method includes the following steps: Step 1: Extract the dynamic scene from the multi-view motion-blurred video sequence and parameterize the dynamic scene into an optimizable time-varying 3D Gaussian sputtering representation; Parameterizing a dynamic scene into an optimizable time-varying 3D Gaussian sputtering representation specifically includes: constructing a set of 3D Gaussian primitives using dynamic scene parameters, and introducing a time dimension into the 3D Gaussian primitives to form a set of 4D Gaussian primitives that vary over time. This yields an optimizable time-varying 3D Gaussian sputtering representation; each 4D Gaussian element consists of a time-varying center position. Covariance matrix Opacity Viewpoint-related colors definition; Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory with Gaussian properties, specifically: Utilizing in reference time The nth-order Taylor series expanded at a given point models the center position of the 4D Gaussian element, and the corresponding process has the following relationship: ； in, Represents the learnable k-th order Taylor coefficients. Let n represent the Peano remainder used to capture nonlinear residuals, and n represent the order of the Taylor expansion. Indicates the center position of the 4D Gaussian element. Let t represent the factorial of k, and t represent the current time. Step 2: Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field. Step 3: Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage; Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure. Specifically, the process includes the following steps: Prior depth is extracted from video frames of a multi-view motion-blurred video sequence using a pre-trained monocular depth estimation model. Utilizing the optimizable time-varying 3D Gaussian sputtering representation of the current optimization phase, the depth of the current model rendering is generated at a given camera pose through a differentiable rasterization process; By using the prior depth and the current model rendering depth, a blended depth map is constructed. The corresponding process has the following relationship: ； in, Represents the fusion coefficient. Represents a mixed depth map. Indicates the prior depth; Indicates the current rendering depth of the model; Dynamic object masks are extracted from multi-view motion-blurred video sequences, and dynamic pixel sets are defined based on these masks. ,in, This represents a dynamic object mask, where u represents a dynamic pixel. Represents a dynamic set of pixels; By using the blending depth and camera parameters, dynamic pixels are back-projected into 3D space, and based on the motion information corresponding to the camera coordinate system, the 2D reprojection coordinates at each sampling time are calculated to obtain the sampling coordinates of the blur kernel. The camera coordinate system velocity and blending depth map are input into the blur weight network to predict the contribution weight of each sampling point to the final blur color. The process of generating a sharp rendered image from the currently optimizable time-varying 3D Gaussian sputtering representation through differentiable rasterization rendering follows the following relationship: ； in, This represents the sharp color at pixel u. This represents the color of the q-th 4D Gaussian pixel. This represents the opacity of the q-th 4D Gaussian pixel; By using the contribution weight of the final blurred color to perform a weighted integral on the clear rendered image at the sampling coordinates of the blur kernel, a synthetic physical blur image consistent with the input video frame is synthesized. Step 4: Construct a total loss function that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions based on the synthesized physically blurred image consistent with the input video frame and the clear rendered image. Iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a clear image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

2. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 1, characterized in that, In step 1, the scene dynamic parameters of the multi-view motion-blurred video sequence are extracted, specifically including the following steps: Based on multi-view motion-blurred video sequences, sparse point clouds and camera extrinsic parameters of each frame are extracted using motion structure restoration techniques. The center location and covariance matrix are calculated based on the sparse point cloud. The viewpoint-related color is calculated based on the sparse point cloud. Given opacity; scene dynamics are composed of center position, covariance matrix, opacity, and viewpoint-dependent color.

3. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 2, characterized in that, In step 1, the covariance matrix By rotation matrix With scaling matrix The definition and the corresponding process have the following relationship: ； in, This indicates the transpose operation. Represents the scaling matrix. This represents the rotation matrix.

4. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 3, characterized in that, In step 2, based on the Taylor expansion motion model, the initial velocity field is obtained through analytical differentiation and differentiable projection. A neural network is then used to correct the residuals of the initial velocity field, resulting in a corrected 2D pixel-level velocity field. Specifically, the steps include: Differentiating the polynomial of the center position of the 4D Gaussian element defined by Taylor coefficients yields the basic 3D velocity field. The corresponding process has the following relationship: ； in, Indicates time t The basic 3D velocity field at that location. Represents the (k+1)th order Taylor coefficients. Taylor polynomial approximation of the center position of the 4D Gaussian element. ; By using differentiable velocity rasterization, the opacity of the fundamental 3D velocity field and the 4D Gaussian primitives is combined. and transmittance Combined, projected onto a 2D image plane, to generate a rendering speed map. The corresponding process has the following relationship: ； in, Represents the basic 3D velocity field. This represents a rendering speed graph. Indicates accumulation to the primitive. q Previous transmittance, , Indicates the first j The opacity of each 4D Gaussian pixel, ranging from [0, 1], determines the contribution of the current 4D Gaussian pixel to the final pixel color. q This represents the index of the currently processed 4D Gaussian unit. During the differentiable rasterization process, the 4D Gaussian units are sorted from far to near in depth and α-mixed sequentially. Indicates the pixel u An ordered set of all contributing 4D Gaussian elements; The rendering speed map and the current rendering time are input into the Peano residual correction network to predict the speed residual. ; The corrected 2D pixel-level velocity field is obtained by adding the rendered velocity map to the velocity residual. The corresponding process has the following relationship: ； in, Represents the velocity residual. This represents the corrected 2D pixel-level velocity field.

5. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 4, characterized in that, The dynamic pixels are back-projected into 3D space using blending depth and camera parameters. Based on the motion information in the camera coordinate system, the 2D reprojection coordinates at each sampling time are calculated to obtain the sampling coordinates of the blur kernel. The specific steps include the following: The motion trajectory of dynamic pixels during the exposure time is discretized into N samples, and the time offset of each sample is calculated. The corresponding process has the following relationship: ； in, Indicates the exposure time. i Indicates the first i One sample, , N Represents the total number of samples. Indicates the first i Time offset of each sample; By back-projecting dynamic pixels onto the camera coordinate system using the blending depth and camera intrinsic parameter matrix, the initial 3D position is obtained. The corresponding process has the following relationship: ； in, Represents dynamic pixels u homogeneous coordinates Indicates the initial 3D position; K This represents the camera intrinsic parameter matrix, which is obtained through calibration during the motion structure recovery process or read from video metadata. The corrected 2D pixel-level velocity field is rigorously transformed to the camera coordinate system using a view rotation matrix to obtain the camera coordinate system velocity. The corresponding process has the following relationship: ； in, Indicates the velocity of the camera coordinate system; This represents the view rotation matrix, which is extracted from the rotation part of the camera extrinsic parameters of the current frame and is used to transform the velocity vector in the world coordinate system to the current camera coordinate system. Based on the camera coordinate system velocity, the 3D position corresponding to the time offset of each sample is calculated and reprojected back onto the image plane to obtain the sampling coordinates of the blur kernel. The corresponding process has the following relationship: ； in, The perspective projection function represents the projection from 3D camera coordinates to 2D image coordinates. The sampling coordinates represent the fuzzy kernel.

6. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 5, characterized in that, The camera coordinate system velocity and blending depth map are input into the blur weight network to predict the contribution weight of each sampling point to the final blur color. The corresponding process has the following relationship: ； in, This indicates the contribution weight of the final blurred color. This indicates that Softmax normalization is performed along the time dimension to ensure... ; This represents a fuzzy weighted network. By performing a weighted integral on the sharp rendered image at the sampling coordinates of the blur kernel, a synthetic physically blurred image consistent with the input video frame is synthesized. The corresponding process has the following relationship: ； in, This represents the color value at the corresponding coordinates of a clearly rendered image. Represents the pixel color of the synthesized physically blurred image that matches the input video frame. This indicates the contribution weight of the final blurred color.

7. The motion deblurring method based on Taylor expansion and 3D Gaussian sputtering according to claim 6, characterized in that, In step 4, the total loss function that simultaneously constrains the fuzzy synthesis accuracy of the dynamic region and the reconstruction consistency of the static region has the following relationship: ； in, Represents the total loss function. Indicates the static region reconstruction loss. This represents the dynamic region re-blurring loss. Represents the dynamic region re-blurring loss Weighting coefficients; Represents static region reconstruction loss Weighting coefficients, weighting coefficients and Used to balance the constraint strength between dynamic and static regions; Among them, dynamic region re-blurring loss Used to force the synthesized blurred image in dynamic regions To approximate a realistic fuzzy input, the corresponding process has the following relationship: ； in, This represents the blurred image synthesized by the model. Represents a realistic blurred input image. Indicates a dynamic region. It is an L1 norm. This indicates element-wise multiplication; Among them, static area reconstruction loss A sharp image used to force rendering in static areas. C Approximating realistic fuzzy input Since there is no motion in the static region, its ideal clear image should be consistent with the input blurred image. This loss term effectively prevents the static geometry from being distorted by the error gradient of the dynamic region during the optimization process, ensuring the stability of the background structure. The corresponding process has the following relationship: ； in, C This indicates a sharp image that is forcibly rendered in a static area. Indicates a static region.

8. A motion deblurring system based on Taylor expansion and 3D Gaussian sputtering, characterized in that, The system applies the motion deblurring method based on Taylor expansion and 3D Gaussian sputtering as described in any one of claims 1 to 7, and the system comprises: The motion modeling module is used for: Extract dynamic scenes from multi-view motion-blurred video sequences and parameterize the dynamic scenes into an optimizable time-varying 3D Gaussian sputtering representation; Based on the optimizable time-varying 3D Gaussian sputtering representation, a motion model based on Taylor expansion is constructed to explicitly parameterize the motion trajectory of Gaussian properties; The model derivation module is used for: Based on the Taylor expansion motion model, the initial velocity field is obtained by analytical differentiation and differentiable projection, and the residual of the initial velocity field is corrected by a neural network to obtain the corrected 2D pixel-level velocity field. Image synthesis module, used for: Render a clear image using the time-varying 3D Gaussian sputtering representation that can be optimized at the current stage; Based on the optimizable time-varying 3D Gaussian sputtering representation and the corrected 2D pixel-level velocity field, a synthetic physically blurred image consistent with the input video frame is synthesized by simulating the physical imaging process during camera exposure. The iterative optimization module is used for: Based on the synthesized physically blurred image consistent with the input video frame and the sharp rendered image, a total loss function is constructed that simultaneously constrains the blur synthesis accuracy of dynamic regions and the reconstruction consistency of static regions. This function is used to iteratively optimize the optimizable time-varying 3D Gaussian sputtering representation. After optimization, the final 3D Gaussian scene representation is obtained, and a sharp image is rendered using the final 3D Gaussian scene representation to achieve motion deblurring.

Images