Monocular blurred video reconstruction method based on 4D Gaussian splattering

Abstract

A monocular blurred video reconstruction method based on 4D Gaussian splatting belongs to the fields of image restoration and three-dimensional reconstruction. This method addresses the problems of existing monocular blurred video reconstruction methods, such as low video visual quality and poor real-time rendering performance, when processing object motion blur. The present invention constructs a 4DGS model, simulates the physical formation process of camera motion blur and object motion blur, and synthesizes blurred images. This is used to calculate the loss with real blurred images to train the constructed 4DGS model, and then uses the trained 4DGS model to achieve video reconstruction. The present invention is primarily used to achieve 4D scene reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of image restoration and three-dimensional reconstruction. Background Art

[0002] In recent years, many impressive and outstanding works have emerged in the field of 3D reconstruction. Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) proposed implicit neural representation and explicit Gaussian ellipsoid representation respectively, which provide strong support for constructing high-quality static 3D scenes. With the increasing industrial demand, 4D reconstruction technology (i.e., dynamic scene reconstruction) has gradually become a research hotspot. In order to model dynamic objects, researchers have introduced implicit neural fields and explicit deformation methods to represent the motion of objects. Although these methods have made significant progress, most methods still rely on synchronized multi-view videos. In monocular videos, since the model only observes the dynamic object once at each time point, the performance of existing methods is often unsatisfactory. To address this problem, recent studies have introduced data-driven prior information such as depth maps, optical flow, trajectory and generative models to improve the performance of 4D reconstruction.

[0003] However, motion blur caused by camera shake and object motion during the exposure time often appears in videos. The above methods usually rely on clear videos as input for supervision. When processing blurred videos, the reconstruction results are not ideal, that is, the reconstructed video is still blurred. Therefore, how to restore clear reconstruction results from blurred monocular videos becomes a practical problem. The first step to solve the blur problem is to deal with camera motion blur, which is relatively simple. Some methods based on NeRF and 3DGS jointly optimize the 3D static scene representation and camera pose to deal with camera motion blur by calculating the reconstruction loss between the synthesized blurred image and the input blurred frame. However, object motion blur is more challenging because it requires estimating a continuous and clear dynamic representation within the exposure time. To the best of our knowledge, only DybluRF has attempted to use the NeRF method to solve this problem, but it has the disadvantages of low visual quality of the produced video and poor real-time performance of the rendering results. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems of low video visual quality and poor real-time rendering performance in existing monocular blurred video reconstruction methods when processing object motion blur; the present invention provides a monocular blurred video reconstruction method based on 4D Gaussian splashing.

[0005] A monocular blurred video reconstruction method based on 4D Gaussian splattering comprises the following steps:

[0006] S1. Constructing training samples: Each frame of a monocular blurry video is used as a real blurry image, and the real blurry image is used as a training sample. The monocular blurry video is also divided into K segments, and the image corresponding to the frame with the highest clarity in each segment is used as the standard frame image.

[0007] S2. Build a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models.

[0008] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment;

[0009] The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in the segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of the t-th frame image in the segment according to the dynamic area Gaussian D t , the dynamic area Gaussian D of the t-1 frame image t-1 and the dynamic area Gaussian D of the t+1 frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment;

[0010] A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video;

[0011] The camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video based on the camera pose corresponding to the t-th frame image of the segment;

[0012] S3. First stage training: Use training samples to train the static area prediction model and the camera pose prediction model simultaneously. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training;

[0013] S4, second stage training: Use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model, and camera pose prediction model to complete the training;

[0014] S5, inference stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

[0015] Preferably, the dynamic area prediction model includes a standard dynamic Gaussian model of blur perception, a dynamic deformable field model, a continuous dynamic Gaussian estimation module, and an exposure time parameter model; wherein the standard dynamic Gaussian model of blur perception, the dynamic deformable field model, and the exposure time parameter model are all learnable parts; the standard frame image corresponding to each segment of the monocular blur video is used to establish the standard dynamic Gaussian of the blur perception standard dynamic Gaussian model in the dynamic area prediction model under each segment;

[0016] A dynamic deformable field model is used to perform deformable field transformation based on the standard dynamic Gaussian under each segment to obtain the dynamic area Gaussian of all frame images in the segment;

[0017] The Gaussian D of the dynamic area of ​​the t-1 frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 ;

[0018] The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video;

[0019] The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1 frame image corresponding to each segment according to the relative exposure time of each moment in the exposure time of the t-1 frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 , generating D t,1 to D t,N Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

[0020] Preferably,

[0021]

[0022] Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating-point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

[0023]

[0024] w t is the exposure time corresponding to the t-th frame image of each segment.

[0025] Preferably, the static area prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model;

[0026] The camera pose predictor is used to predict the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment within the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 Also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N Also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time;

[0027] The interpolation module is used to calculate the exposure time P corresponding to the standard frame image of each segment in the SE(3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m, m is an integer, and 1<m<N.

[0028] Preferably,

[0029]

[0030] Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

[0031] Preferably, the camera pose predictor includes a main branch, a camera pose prediction branch at the start of exposure, and a camera pose prediction branch at the end of exposure;

[0032] The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, and the camera pose P t After inputting the high-frequency coding layer, the first four fully connected layers output the feature map A after feature extraction;

[0033] The output of the main branch is simultaneously fed into the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure;

[0034] The camera pose prediction branch at the start of exposure includes: the second first fully connected layer, the second first activation operation layer and the second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer and then the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ;

[0035] The camera pose prediction branch at the end of exposure includes: a cascaded third-first fully connected layer, a third-first activation operation layer, and a third-second fully connected layer. The feature map A is input into the third-first fully connected layer, and the third-second fully connected layer outputs the camera pose P at the end of exposure. t,N .

[0036] Preferably,

[0037]

[0038] in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, is the l1 loss function, is the structural similarity loss, and ⊙ is the pixel-wise multiplication operation.

[0039] Preferably,

[0040] During the first and second stage training, a multi-resolution consistency regularization term is introduced Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment;

[0041]

[0042] in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, is the clear image at the i-th moment within the exposure time rendered by the 4DGS model obtained by training the downsampled monocular blurry video, ||||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i=1,2…N, N is the total number of time steps within the exposure time;

[0043] During the first and second stage training, a multi-frame consistency regularization term is introduced Constrain the content consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment;

[0044]

[0045] Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, To estimate the optical flow through the pre-trained network Towards Alignment obtained, To estimate the optical flow through the pre-trained network Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

[0046] Preferably, during the second stage of training, the exposure time regularization term is introduced Distinguish the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model;

[0047]

[0048] Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each segment, and max() is the maximum value operation.

[0049] The beneficial effects brought by the present invention are:

[0050] The present invention estimates continuous camera poses and continuous dynamic Gaussians during exposure time, simulates camera motion and object motion during exposure time, and estimates a clear 4D scene representation.

[0051] The present invention first performs the first stage training on the static region prediction model and the camera pose prediction model in the 4DGS model simultaneously, in order to stabilize the training. Then, the second stage training is performed on the dynamic region prediction model, the static region prediction model and the camera pose prediction model in the 4DGS model simultaneously. In the process of training the 4DGS model, the physical formation process of camera motion blur and object motion blur is simulated to synthesize blurred images, so as to calculate the loss with the real blurred image to realize the training of the constructed 4DGS model, and use the trained 4DGS model to realize video reconstruction. In specific applications, the method of the present invention benefits from the explicit representation of the physical world and the fuzzy robust training method of the trained 4DGS model, so that the real-time rendering result is maintained while significantly improving the reconstructed visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the principle of the monocular blurred video reconstruction method based on 4D Gaussian splattering according to the present invention;

[0053] Figure 2 This is a schematic diagram of the camera pose predictor structure;

[0054] Figure 3 This is a visual quality comparison chart of the reconstruction results of the reconstruction method of the present invention and the existing model. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] 4D Gaussian splash:

[0058] A 3D Gaussian sphere (3DGS) can be described by five parameters {x, r, s, o, c}, where x represents its center position in the world coordinate system, r and s are the rotation matrix and scale matrix respectively, which are used to define the shape, α is the opacity, and the spherical harmonic coefficient c represents the perspective-dependent color. In 4D Gaussian splash (4DGS), static and dynamic areas are usually treated separately. The static area can usually be represented by a set of 3DGS, denoted as S. For the dynamic area, 4DGS usually selects a timestamp (such as the first timestamp) as the canonical frame, and represents the moving object corresponding to the frame as a canonical dynamic Gaussian C. Then, C deforms the Gaussian sphere through an implicit or explicit deformable field to represent the motion of the object. The dynamic Gaussian at time t is denoted as D t , which can be expressed as:

[0059]

[0060] in, is the deformation operation, Θ F yes The Gaussian at time t consists of the static Gaussian S and the dynamic Gaussian D t In general, 4DGS represents a scene as a static Gaussian S, a standard dynamic Gaussian C, and a deformation operation In addition, by providing the camera pose, the Gaussian D at time t can be t Project to 2D space and render to get the corresponding image.

[0061] The formation process of motion blur:

[0062] Motion blur is caused by camera shake and object motion and can be simply viewed as the integral of a sequence of potentially sharp images, i.e.:

[0063]

[0064] in, is a blurred image, I t (u, v) is the potential clear image at time t, (u, v) is the pixel position, τ is the camera exposure time, and φ is the normalization factor. To approximate the integration operation, the exposure time can be divided into N timestamps, and the blurred image can be regarded as the average of N clear images, that is:

[0065]

[0066] In the invention, the continuous camera motion and object motion within the exposure time are estimated, and the physical formation process of the above-mentioned motion blur is integrated into the model optimization, so as to reconstruct a high-quality 4DGS model from the blurred video.

[0067] The concept behind this invention is to design a monocular blurred video reconstruction method based on 4D Gaussian splatting. During 4DGS model training, the physical formation of camera and object motion blur is simulated to synthesize blurred images. This is then used to calculate the loss between the blurred images and the trained 4DGS model, and video reconstruction is then performed using the trained 4DGS model. The following specific implementation methods are provided:

[0068] Specific implementation method 1. Figure 1 This embodiment describes a monocular blurred video reconstruction method based on 4D Gaussian splattering, which includes the following steps:

[0069] S1. Constructing training samples: Each frame of a monocular blurry video is used as a real blurry image, and the real blurry image is used as a training sample. The monocular blurry video is also divided into K segments, and the image corresponding to the frame with the highest clarity in each segment is used as the standard frame image.

[0070] S2. Build a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models.

[0071] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment;

[0072] The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in the segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of the t-th frame image in the segment according to the dynamic area Gaussian D t , the dynamic area Gaussian D of the t-1 frame image t-1 and the dynamic area Gaussian D of the t+1 frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment;

[0073] A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video;

[0074] The camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video based on the camera pose corresponding to the t-th frame image of the segment;

[0075] S3. First stage training: Use training samples to train the static area prediction model and the camera pose prediction model simultaneously. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training;

[0076] S4, second stage training: Use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model, and camera pose prediction model to complete the training;

[0077] S5, inference stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

[0078] In this implementation, the overall framework of the monocular blurred video reconstruction method based on 4D Gaussian splattering is presented. In general, the camera motion and object motion during the exposure time are simulated by estimating the continuous camera pose and dynamic Gaussian during the exposure time. When processing the standard frame image of each fragment, the camera exposure time of the frame is first evenly divided into N time steps, that is, N moments are obtained. Then, the corresponding image is rendered according to the camera pose, dynamic Gaussian and static Gaussian; finally, all images within the exposure time are processed. Average to obtain a synthetic blurred image use and the real blurred image B t Calculate the reconstruction loss and implement 4DGS model training.

[0079] To estimate continuous dynamic Gaussian, the standard dynamic Gaussian selection strategy for fuzzy perception is:

[0080] The current 4DGS model usually only selects the dynamic area at one moment as the standard dynamic Gaussian. This approach is prone to insufficient rendering details when reconstructing scenes with large motion. To this end, the present invention selects multiple dynamic Gaussians according to the degree of blur of the image. Among them, each dynamic Gaussian is only responsible for several adjacent frames, thereby reducing the burden of modeling large motion. That is: divide the video into K segments, divide the video evenly into K segments, and use the image corresponding to the frame with the highest clarity in each segment as the standard frame image, and represent the content corresponding to the standard frame image as the dynamic standard Gaussian C of the video segment. j , j=1,2……K.

[0081] A variable standard dynamic Gaussian selection strategy for blur perception is proposed to better handle scenes with large motion. The video is evenly divided into K segments, and the blur degree of the dynamic area of ​​each frame is specifically calculated. It can be expressed as,

[0082]

[0083] Among them, b t is the blur degree of the dynamic area of ​​the t-th frame image, M t Is the mask indicating the dynamic area of ​​the t-th frame image. t is the Laplace representation of the t-th frame image, is ΔB t The mean of b. t The smaller the value, the clearer the image; conversely, the blurrier the image. In addition, in order to keep the boundaries of each video segment as clear as possible, in specific applications, the clearest frames can be found within the H frame range at the boundary of each video segment and redefined as the boundaries of the video segment. Finally, the content corresponding to the clearest frame in each segment is represented as the dynamic norm Gaussian C of the segment. j .

[0084] In order to simulate object motion blur, the estimation of continuous object motion representation is converted into an estimation of exposure time. A specific structure of the dynamic area prediction model is given, which is as follows:

[0085] See also Figure 1The dynamic area prediction model includes a blur-aware standard dynamic Gaussian model, a dynamic deformable field model, a continuous dynamic Gaussian estimation module, and an exposure time parameter model; among them, the blur-aware standard dynamic Gaussian model, the dynamic deformable field model, and the exposure time parameter model are all learnable parts;

[0086] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian model of blur perception in the dynamic area prediction model under each segment;

[0087] A dynamic deformable field model is used to perform deformable field transformation based on the standard dynamic Gaussian under each segment to obtain the dynamic area Gaussian of all frame images in the segment;

[0088] The Gaussian D of the dynamic area of ​​the t-1 frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 ;

[0089] The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video;

[0090] The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1 frame image corresponding to each segment according to the relative exposure time of each moment in the exposure time of the t-1 frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 , generating D t,1 to D t,N Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

[0091] In specific applications, the standard dynamic Gaussian model of fuzzy perception and the static area prediction model are both 3DGS models.

[0092] In order to realize the deformation of dynamic Gaussian, a set of rigid transformation matrices can be used to represent the motion change of three-dimensional Gaussian. Specifically, a Gaussian point in the standard dynamic space is represented as {x c ,r c ,s,o,c}, the corresponding Gaussian ball in the tth frame is represented as {x t ,r t ,s,o,c}, the corresponding rigid transformation matrix is ​​expressed as {A t ,E t}, the deformation process of the Gaussian sphere can be expressed as,

[0093] x t =A t x c +E t ,r t =A t r c .

[0094] x c is the position of the standard dynamic Gaussian, r c is the shape of a standard dynamic Gaussian; x t is the position of the dynamic Gaussian of the tth frame corresponding to the standard dynamic Gaussian, r t is the shape of the dynamic Gaussian of the tth frame corresponding to the standard dynamic Gaussian.

[0095] Specifically, continuous dynamic Gaussian is obtained by estimating exposure parameters, and continuous dynamic Gaussian is converted into Estimated conversion into corresponding exposure time parameters Calculation. Assume that the exposure parameter w at the i-th exposure moment is obtained t,i (ie D t,i and The normalized time interval between t,i It can be calculated according to the following formula:

[0096]

[0097] Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating-point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

[0098] Specifically, w t,i It can be expressed as:

[0099]

[0100] w t is the exposure time corresponding to the t-th frame image of each segment.

[0101] Specifically, considering that the motion of the object during the exposure time can be regarded as uniform linear motion, w t,1 and w t,N The absolute values ​​of can be regarded as the same, both are exposure time w t One half of .

[0102] In order to simulate camera motion blur, a camera pose prediction model is set up to estimate the continuous camera pose within the exposure time, and a specific structure of the static area prediction model is given, which is as follows:

[0103] See also Figure 1 ,The static region prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model;

[0104] The camera pose predictor is used to predict the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment within the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 Also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N Also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time;

[0105] The interpolation module is used to calculate the exposure time P corresponding to the standard frame image of each segment in the SE(3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m , m is an integer, and 1<m<N.

[0106] In order to estimate the continuous camera pose during the exposure time A camera pose predictor is set up to predict the camera pose P at the start of exposure t,1 and the camera pose P at the end t,N Then, in SE(3) algebraic space, t,1 and P t,N Perform linear interpolation to obtain the camera pose P at the mth time step t,m .P t,m It can be expressed as:

[0107]

[0108] Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

[0109] See also Figure 2 ,The specific camera pose predictor includes a main branch, a camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure;

[0110] The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, and the camera pose P t After inputting the high-frequency coding layer, the first four fully connected layers output the feature map A after feature extraction;

[0111] The output of the main branch is simultaneously fed into the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure;

[0112] The camera pose prediction branch at the start of exposure includes: the second first fully connected layer, the second first activation operation layer and the second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer and then the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ;

[0113] The camera pose prediction branch at the end of exposure includes: a cascaded third-first fully connected layer, a third-first activation operation layer, and a third-second fully connected layer. The feature map A is input into the third-first fully connected layer, and the third-second fully connected layer outputs the camera pose P at the end of exposure. t,N .

[0114] In specific applications, the number of channels of the first fully connected layer, the first two fully connected layers, and the first three fully connected layers of the main branch are all set to 64.

[0115] At the start of exposure, the number of channels of the second-first fully connected layer of the camera pose prediction branch is set to 64, and the number of channels of the second-second fully connected layer is set to 6.

[0116] At the end of exposure, the number of channels of the third fully connected layer of the camera pose prediction branch is set to 64, and the number of channels of the third and second fully connected layers is set to 6.

[0117] The activation function for all activation operations is the ReLU activation function.

[0118] In order to stabilize the first stage training process, a static reconstruction loss function is used during optimization. Train the static area prediction model and the camera pose prediction model. Specifically,

[0119]

[0120] In order to stabilize the second stage training process, the second stage loss function is used during optimization. The dynamic region prediction model, static region prediction model and camera pose prediction model are trained simultaneously. Specifically,

[0121]

[0122] in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, is the l1 loss function, is the structural similarity loss, and ⊙ is the pixel-wise multiplication operation.

[0123] This paper introduces exposure time regularization, multi-frame consistency regularization, and multi-resolution consistency regularization to constrain the unrestricted optimization problem. The exposure time regularization term is introduced to suppress trivial solutions, while the multi-frame consistency regularization term and multi-resolution consistency regularization term are introduced to mitigate rendered image artifacts.

[0124] Specifically, during the first and second stage training, a multi-resolution consistency regularization term is introduced Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment;

[0125]

[0126] in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, is the clear image at the i-th moment within the exposure time rendered by the 4DGS model obtained by training the downsampled monocular blurry video, ||||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i=1,2…N, N is the total number of time steps within the exposure time.

[0127] Since the reconstruction results at each resolution should be consistent and the blur at low resolution is easier to remove, a multi-resolution consistency constraint is introduced. The reconstruction results of the low-resolution model are used to guide the reconstruction of the high-resolution model to avoid image artifacts; the low-resolution model is a 4DGS model trained with downsampled monocular blurred video.

[0128] During the first and second stage training, a multi-frame consistency regularization term is introduced Constrain the content consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment;

[0129]

[0130] Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, To estimate the optical flow through the pre-trained network Towards Alignment obtained, To estimate the optical flow through the pre-trained network Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

[0131] Since the content of each frame should be consistent within a certain exposure time, a multi-frame consistency constraint is introduced. Constrain the content between adjacent frames to remain consistent and constrain the content of each frame to remain consistent with the first frame to alleviate image artifacts.

[0132] During the second stage of training, the exposure time regularization term is introduced Distinguish the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model;

[0133]

[0134] Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each segment, and max() is the maximum value operation.

[0135] Continuous Dynamic Gaussian There should be some distinction within a certain exposure time. In other words, w t The value of should not be too small. If w t Too small, D t,i will be almost the same as D t Consistent, leading to the situation of trivial solution, so the exposure time regularization term is introduced To alleviate the ordinary solution.

[0136] Principle analysis: By estimating the continuous camera pose and continuous dynamic Gaussian within the exposure time, the camera motion and object motion within the exposure time are simulated to estimate a clear scene representation. In addition, a blur-aware variable standard dynamic Gaussian strategy is introduced to better model scenes with large motion amplitudes, simplify the reconstruction process, and increase the reconstruction speed; at the same time, the exposure time regularization term, the multi-frame consistency regularization term, and the multi-resolution consistency regularization term are introduced to constrain the unrestricted optimization problem. Among them, the introduction of the exposure time regularization term To alleviate trivial solutions, introduce multi-frame consistency constraints and multi-resolution consistency constraints To alleviate image artifacts and further improve the reconstructed visual quality.

[0137] Verification test:

[0138] See also Figure 3 , Figure 3 A visual quality comparison chart of the reconstruction method of the present invention and the reconstruction results of the existing model is given. Figure 3 The reconstruction method of the present invention is closer to the clear truth image, and the reconstruction effect is significantly higher than other existing technologies.

[0139] Combined with the above, it is proved that the monocular blurred video reconstruction method of the present invention based on 4D Gaussian splashing is significantly better than the current optimal method.

[0140] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A monocular blurred video reconstruction method based on 4D Gaussian splashing, characterized by: The method comprises the following steps: S1. Constructing training samples: Each frame of a monocular blurry video is used as a real blurry image, and the real blurry image is used as a training sample. The monocular blurry video is also divided into K segments, and the image corresponding to the frame with the highest clarity in each segment is used as the standard frame image. S2. Build a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models. The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment; The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in the segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of the t-th frame image in the segment according to the dynamic area Gaussian D t , the dynamic area Gaussian D of the t-1 frame image t-1 and the dynamic area Gaussian D of the t+1 frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment; A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video; The camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video based on the camera pose corresponding to the t-th frame image of the segment; S3. First stage training: Use training samples to train the static area prediction model and the camera pose prediction model simultaneously. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training; S4, second stage training: Use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment, and a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment is generated. The clear images of all moments are averaged to obtain the synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model, and camera pose prediction model to complete the training; S5, inference stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

2. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 1, characterized in that: The dynamic region prediction model includes a blur-perceived standard dynamic Gaussian model, a dynamic deformable field model, a continuous dynamic Gaussian estimation module, and an exposure time parameter model. These three models are all learnable components. The standard frame image corresponding to each segment of a monocular blurry video is used to establish the standard dynamic Gaussian of the blur-perceived standard dynamic Gaussian model in the dynamic region prediction model for each segment. A dynamic deformable field model is used to perform deformable field transformation based on the standard dynamic Gaussian under each segment to obtain the dynamic area Gaussian of all frame images in the segment; The Gaussian D of the dynamic area of ​​the t-1 frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 ; The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video; The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1 frame image corresponding to each segment according to the relative exposure time of each moment in the exposure time of the t-1 frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic area Gaussian D of the t+1 frame image t+1 , generating D t,1 to D t,N Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

3. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 2, characterized in that: Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating-point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

4. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 3, characterized in that: w t is the exposure time corresponding to the t-th frame image of each segment.

5. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 1, characterized in that: The static region prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model; The camera pose predictor is used to predict the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment within the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 Also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N Also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time; The interpolation module is used to calculate the exposure time P corresponding to the standard frame image of each segment in the SE(3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m , m is an integer, and 1<m<N.

6. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 5, characterized in that: Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

7. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 5, characterized in that: The camera pose predictor includes a main branch, a camera pose prediction branch at the start of exposure, and a camera pose prediction branch at the end of exposure; The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, and the camera pose P t After inputting the high-frequency coding layer, the first four fully connected layers output the feature map A after feature extraction; The output of the main branch is simultaneously fed into the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure; The camera pose prediction branch at the start of exposure includes: the second first fully connected layer, the second first activation operation layer and the second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer and then the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ; The camera pose prediction branch at the end of exposure includes: a cascaded third-first fully connected layer, a third-first activation operation layer, and a third-second fully connected layer. The feature map A is input into the third-first fully connected layer, and the third-second fully connected layer outputs the camera pose P at the end of exposure. t,N .

8. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 1, characterized in that: in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, for loss function, is the structural similarity loss, and ⊙ is the pixel-wise multiplication operation.

9. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 1, characterized in that: During the first and second stage training, a multi-resolution consistency regularization term is introduced Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment; in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, is the clear image at the i-th moment within the exposure time rendered by the 4DGS model obtained by training the downsampled monocular blurry video, || ||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i = 1, 2...N, N is the total number of time steps within the exposure time; During the first and second stage training, a multi-frame consistency regularization term is introduced Constrain the content consistency of the clear image at all moments in the exposure time corresponding to the t-th frame image of each segment; Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, To estimate the optical flow through the pre-trained network Towards Alignment obtained, To estimate the optical flow through the pre-trained network Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

10. The monocular blurred video reconstruction method based on 4D Gaussian splattering according to claim 2, characterized in that: During the second stage of training, the exposure time regularization term is introduced Distinguish the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model; Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each segment, and max() is the maximum value operation.

Images