Monocular fuzzy video reconstruction method based on 4D Gaussian splashing

Abstract

The invention discloses a monocular fuzzy video reconstruction method based on 4D Gaussian splashing, and belongs to the field of image restoration and three-dimensional reconstruction. The problems that when an existing monocular blurred video reconstruction method is used for processing object motion blurring, the generated video visual quality is low, and the real-time performance of a rendering result is poor are solved. According to the method, the 4DGS model is constructed, the blurred image is synthesized by simulating the physical forming process of camera motion blurring and object motion blurring, so that the loss of the real blurred image is calculated to train the constructed 4DGS model, and the trained 4DGS model is utilized to realize video reconstruction. The method is mainly used for realizing 4D scene reconstruction.

Description

Technical Field

[0001] The invention belongs to the field of image restoration and three-dimensional reconstruction. Background Art

[0002] In recent years, many impressive works have emerged in the field of 3D reconstruction. Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) proposed implicit neural representation and explicit Gaussian ellipsoid representation respectively, which provide strong support for constructing high-quality static 3D scenes. With the increase in industrial demand, 4D reconstruction technology (i.e., dynamic scene reconstruction) has gradually become a research hotspot. In order to model dynamic objects, researchers have introduced implicit neural fields and explicit deformation methods to represent the motion of objects. Although these methods have made significant progress, most of them still rely on synchronized multi-view videos. In monocular videos, since the model only observes the dynamic object once at each time point, the results of existing methods are often unsatisfactory. To solve this problem, recent studies have introduced data-driven prior information, such as depth maps, optical flow, trajectories, and generative models to improve the results of 4D reconstruction.

[0003] However, motion blur caused by camera shake and object motion during exposure time often appears in videos. The above methods usually rely on clear videos as input for supervision. When processing blurry videos, the reconstruction results are not ideal, that is, the reconstructed video is still blurry. Therefore, how to restore clear reconstruction results from blurry monocular videos becomes a practical problem. The first step to solve the blur problem is to deal with camera motion blur, which is relatively simple. Some NeRF and 3DGS-based methods jointly optimize the 3D static scene representation and camera pose to deal with camera motion blur by calculating the reconstruction loss between the synthesized blurry image and the input blurry frame. However, object motion blur is more challenging because it requires estimating a continuous and clear dynamic representation within the exposure time. To the best of our knowledge, only DybluRF has attempted to solve this problem using the NeRF method, which has the disadvantages of low visual quality of the generated video and poor real-time performance of the rendering results. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of low video visual quality and poor real-time rendering results in the existing monocular blurred video reconstruction method when processing object motion blur; the present invention provides a monocular blurred video reconstruction method based on 4D Gaussian splashing.

[0005] A monocular blurred video reconstruction method based on 4D Gaussian splashing, the method comprises the following steps:

[0006] S1. Construct training samples: take each frame of a monocular blur video as a real blur image, and the real blur image as a training sample; divide the monocular blur video into K segments, and take the image corresponding to the frame with the highest clarity in each segment as the standard frame image;

[0007] S2. Construct a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models.

[0008] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment;

[0009] The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in each segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of all frame images in the segment according to the dynamic area Gaussian D of the t-th frame image in the segment. t , the dynamic area Gaussian D of the t-1th frame image t-1 and the dynamic region Gaussian D of the t+1th frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment;

[0010] A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video;

[0011] A camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video according to the camera pose corresponding to the t-th frame image of the segment;

[0012] S3, first stage training: use training samples to train the static area prediction model and the camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment in the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments in the exposure time corresponding to the t-th frame image of the segment is generated, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training;

[0013] S4, second stage training: use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment to generate a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model and camera pose prediction model to complete the training;

[0014] S5, reasoning stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

[0015] Preferably, the dynamic area prediction model includes a standard dynamic Gaussian model of blur perception, a dynamic deformable field model, a continuous dynamic Gaussian estimation module and an exposure time parameter model; wherein the standard dynamic Gaussian model of blur perception, the dynamic deformable field model and the exposure time parameter model are all learnable parts; the standard frame image corresponding to each segment under the monocular blur video is used to establish the standard dynamic Gaussian of the blur perception standard dynamic Gaussian model in the dynamic area prediction model under each segment;

[0016] A dynamic deformable field model is used to perform deformable field transformation according to a standard dynamic Gaussian under each segment to obtain dynamic area Gaussians of all frame images in the segment;

[0017] The dynamic area Gaussian D of the t-1th frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 ;

[0018] The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video;

[0019] The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1th frame image corresponding to each segment according to the relative exposure time at each moment within the exposure time corresponding to the t-1th frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 , generating D t,1 To D t,N Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

[0020] Preferably,

[0021]

[0022] Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

[0023]

[0024] w t is the exposure time corresponding to the t-th frame image of each fragment.

[0025] Preferably, the static area prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model;

[0026] The camera pose predictor is used to calculate the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment in the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 It is also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N It is also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time;

[0027] The interpolation module is used to interpolate the exposure time P corresponding to the standard frame image of each fragment in the SE (3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m, m is an integer, and 1<m<N.

[0028] Preferably,

[0029]

[0030] Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

[0031] Preferably, the camera pose predictor comprises a main branch, a camera pose prediction branch at the start of exposure, and a camera pose prediction branch at the end of exposure;

[0032] The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, the camera pose P t After inputting the high frequency coding layer, the first four fully connected layers output the feature map A after feature extraction;

[0033] The output of the main branch is sent to the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure at the same time;

[0034] The camera pose prediction branch at the start of exposure includes: a second first fully connected layer, a second first activation operation layer, and a second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer, and the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ;

[0035] The camera pose prediction branch at the end of exposure includes: a third fully connected layer, a third activation operation layer and a third second fully connected layer cascaded in sequence. The feature map A is input into the third first fully connected layer and then outputted by the third second fully connected layer. The camera pose P at the end of exposure t,N .

[0036] Preferably,

[0037]

[0038] in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, is the l1 loss function, is the structural similarity loss and ⊙ is the pixel-by-pixel multiplication operation.

[0039] Preferably,

[0040] During the first and second phases of training, a multi-resolution consistency regularization term is introduced. Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment; where:

[0041]

[0042] in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, is the clear image at the i-th moment in the exposure time rendered by the 4DGS model trained with the downsampled monocular blurry video, ||||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i=1,2…N, N is the total number of time steps in the exposure time;

[0043] During the first and second phases of training, a multi-frame consistency regularization term is introduced. Constrain the content consistency of the clear image at all moments in the exposure time corresponding to the t-th frame image of each segment; wherein,

[0044]

[0045] Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, The pre-trained optical flow estimation network will Towards Alignment obtained, The pre-trained optical flow estimation network will Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

[0046] Preferably, during the second stage of training, an exposure time regularization term is introduced Distinguishing the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model;

[0047]

[0048] Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each fragment, and max() is the maximum value operation.

[0049] The beneficial effects brought by the present invention are:

[0050] The present invention estimates the continuous camera pose and continuous dynamic Gaussian during the exposure time, simulates the camera motion and object motion during the exposure time, and estimates a clear 4D scene representation.

[0051] The present invention first performs the first stage training on the static area prediction model and the camera pose prediction model in the 4DGS model at the same time, in order to stabilize the training. Then, the second stage training is performed on the dynamic area prediction model, the static area prediction model and the camera pose prediction model in the 4DGS model at the same time. In the process of training the 4DGS model, the physical formation process of camera motion blur and object motion blur is simulated to synthesize a blurred image, so as to calculate the loss with the real blurred image to realize the training of the constructed 4DGS model, and use the trained 4DGS model to realize video reconstruction. In specific applications, the method of the present invention benefits from the explicit representation of the physical world and the fuzzy robust training method of the trained 4DGS model, so that the real-time rendering result is maintained while significantly improving the reconstructed visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 4D Gaussian splashing based monocular blurred video reconstruction method of the present invention;

[0053] Figure 2 It is a schematic diagram of the structure of the camera pose predictor;

[0054] Figure 3 This is a visual quality comparison diagram between the reconstruction method of the present invention and the reconstruction results of the existing model. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] 4D Gaussian Splash:

[0058] A 3D Gaussian sphere (3DGS) can be described by five parameters {x, r, s, o, c}, where x represents its center position in the world coordinate system, r and s are the rotation matrix and scale matrix, respectively, which are used to define the shape, α is the opacity, and the spherical harmonic coefficient c represents the perspective-dependent color. In 4D Gaussian splash (4DGS), static and dynamic areas are usually treated separately. The static area can usually be represented by a set of 3DGS, denoted as S. For the dynamic area, 4DGS usually selects a timestamp (such as the first timestamp) as the canonical frame, and represents the moving object corresponding to the frame as a canonical dynamic Gaussian C. Then, C deforms the Gaussian sphere through an implicit or explicit deformable field to represent the motion of the object. The dynamic Gaussian at time t is denoted by D t , which can be expressed as:

[0059]

[0060] in, is the deformation operation, Θ F yes The Gaussian at time t consists of a static Gaussian S and a dynamic Gaussian D t In general, 4DGS represents a scene as a static Gaussian S, a standard dynamic Gaussian C, and a deformation operation In addition, by providing the camera pose, the Gaussian D at time t can be t Project into 2D space and render to get the corresponding image.

[0061] The formation process of motion blur:

[0062] Motion blur is caused by camera shake and object motion and can be simply viewed as the integral of a sequence of latent sharp images, i.e.:

[0063]

[0064] in, is a blurred image, I t (u,v) is the potential clear image at time t, (u,v) is the pixel position, τ is the camera exposure time, and φ is the normalization factor. To approximate the integration operation, the exposure time can be divided into N timestamps, and the blurred image can be regarded as the average of N clear images, that is:

[0065]

[0066] In the invention, continuous camera motion and object motion within the exposure time are estimated, and the physical formation process of the above-mentioned motion blur is integrated into the model optimization, so as to reconstruct a high-quality 4DGS model from the blurred video.

[0067] The source of the invention concept: It aims to design a monocular blurred video reconstruction method based on 4D Gaussian splashing. In the 4DGS model training process, the physical formation process of camera motion blur and object motion blur is simulated to synthesize blurred images, so as to calculate the loss with the real blurred image to realize the training of the constructed 4DGS model, and use the trained 4DGS model to realize video reconstruction. The following specific implementation methods are provided:

[0068] Specific implementation method 1. Figure 1 The present embodiment is described. The monocular blurred video reconstruction method based on 4D Gaussian splashing described in the present embodiment comprises the following steps:

[0069] S1. Construct training samples: take each frame of a monocular blur video as a real blur image, and the real blur image as a training sample; divide the monocular blur video into K segments, and take the image corresponding to the frame with the highest clarity in each segment as the standard frame image;

[0070] S2. Construct a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models.

[0071] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment;

[0072] The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in each segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of all frame images in the segment according to the dynamic area Gaussian D of the t-th frame image in the segment. t , the dynamic area Gaussian D of the t-1th frame image t-1 and the dynamic region Gaussian D of the t+1th frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment;

[0073] A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video;

[0074] A camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video according to the camera pose corresponding to the t-th frame image of the segment;

[0075] S3, first stage training: use training samples to train the static area prediction model and the camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment in the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments in the exposure time corresponding to the t-th frame image of the segment is generated, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training;

[0076] S4, second stage training: use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment to generate a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model and camera pose prediction model to complete the training;

[0077] S5, reasoning stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

[0078] In this implementation, the overall framework of the monocular blurred video reconstruction method based on 4D Gaussian splashing is given. In general, the camera motion and object motion during the exposure time are simulated by estimating the continuous camera pose and dynamic Gaussian during the exposure time. When processing the standard frame image of each fragment, the camera exposure time of the frame is first evenly divided into N time steps, that is, N moments are obtained, and then the corresponding image is rendered according to the camera pose, dynamic Gaussian and static Gaussian; finally, all images during the exposure time are processed. Average to obtain a synthetic blurred image use and the real blurred image B t Calculate the reconstruction loss and implement 4DGS model training.

[0079] To estimate the continuous dynamic Gaussian, the standard dynamic Gaussian selection strategy for blur perception is:

[0080] The current 4DGS model usually only selects a dynamic area at a moment as the standard dynamic Gaussian. This method is prone to insufficient rendering details when reconstructing scenes with large movements. To this end, the present invention selects multiple dynamic Gaussians according to the blurriness of the image. Among them, each dynamic Gaussian is only responsible for several adjacent frames, thereby reducing the burden of modeling large movements. That is: divide the video into K segments, divide the video into K segments, and the image corresponding to the frame with the highest clarity in each segment is used as the standard frame image, and the content corresponding to the standard frame image is represented as the dynamic standard Gaussian C of the video segment. j , j=1,2……K.

[0081] A blur-aware variable standard dynamic Gaussian selection strategy is proposed to better handle scenes with large motion. The video is evenly divided into K segments, and the blur degree of the dynamic area of ​​each frame is specifically calculated. It can be written as,

[0082]

[0083] Among them, b t is the blur degree of the dynamic area of ​​the t-th frame image, M t is the mask indicating the dynamic area of ​​the t-th frame image. t is the Laplace representation of the t-th frame image, is ΔB t The mean value of b t The smaller the image, the clearer it is; otherwise, the image is more blurred. In addition, in order to keep the boundaries of each video segment as clear as possible, in specific applications, the clearest frames can be found within the H frame range at the boundary of each video segment and redefined as the boundaries of the video segment. Finally, the content corresponding to the clearest frame in each segment is represented as the dynamic norm Gaussian C of the segment. j .

[0084] In order to simulate object motion blur, the estimate of continuous object motion representation is converted into an estimate of exposure time, and a specific structure of the dynamic area prediction model is given, which is:

[0085] See also Figure 1The dynamic area prediction model includes a blur-aware standard dynamic Gaussian model, a dynamic deformable field model, a continuous dynamic Gaussian estimation module, and an exposure time parameter model; wherein the blur-aware standard dynamic Gaussian model, the dynamic deformable field model, and the exposure time parameter model are all learnable parts;

[0086] The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian model of blur perception in the dynamic area prediction model under each segment;

[0087] A dynamic deformable field model is used to perform deformable field transformation according to a standard dynamic Gaussian under each segment to obtain dynamic area Gaussians of all frame images in the segment;

[0088] The dynamic area Gaussian D of the t-1th frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 ;

[0089] The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video;

[0090] The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1th frame image corresponding to each segment according to the relative exposure time at each moment within the exposure time corresponding to the t-1th frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 , generating D t,1 To D t,N Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

[0091] In specific applications, the standard dynamic Gaussian model of fuzzy perception and the static area prediction model are both 3DGS models.

[0092] In order to realize the deformation of dynamic Gaussian, a set of rigid transformation matrices can be used to represent the motion change of three-dimensional Gaussian. Specifically, a Gaussian point in the standard dynamic space is represented as {x c ,r c ,s,o,c}, the corresponding Gaussian ball in the tth frame is expressed as {x t ,r t ,s,o,c}, the corresponding rigid transformation matrix is ​​expressed as {A t ,E t}, the deformation process of the Gaussian ball can be expressed as,

[0093] x t =A t x c +E t ,r t =A t r c .

[0094] x c is the position of the standard dynamic Gaussian, r c is the shape of a standard dynamic Gaussian; x t is the position of the dynamic Gaussian of the tth frame corresponding to the standard dynamic Gaussian, r t is the shape of the dynamic Gaussian of the tth frame corresponding to the standard dynamic Gaussian.

[0095] Specifically, the continuous dynamic Gaussian is obtained by estimating the exposure parameters, and the continuous dynamic Gaussian Estimated conversion into corresponding exposure time parameters Calculation. Assume that the exposure parameter w at the i-th exposure moment is obtained t,i (i.e. D t,i and The normalized time interval between t,i It can be calculated according to the following formula:

[0096]

[0097] Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

[0098] Specifically, w t,i It can be expressed as:

[0099]

[0100] w t is the exposure time corresponding to the t-th frame image of each fragment.

[0101] Specifically, considering that the motion of the object during the exposure time can be regarded as uniform linear motion, w t,1 and w t,N The absolute values ​​of can be regarded as the same, both of which are exposure time w t One half of .

[0102] In order to simulate camera motion blur, a camera pose prediction model is set up to estimate the continuous camera pose within the exposure time, and a specific structure of the static area prediction model is given, which is:

[0103] See also Figure 1 ,The static region prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model;

[0104] The camera pose predictor is used to calculate the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment in the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 It is also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N It is also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time;

[0105] The interpolation module is used to interpolate the exposure time P corresponding to the standard frame image of each fragment in the SE (3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m , m is an integer, and 1<m<N.

[0106] In order to estimate the continuous camera pose during the exposure time A camera pose predictor is set up to predict the camera pose P at the start of exposure t,1 and the camera pose P at the end t,N Then, in SE(3) algebraic space, t,1 and P t,N Perform linear interpolation to obtain the camera pose P at the mth time step t,m .P t,m It can be expressed as:

[0107]

[0108] Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

[0109] See also Figure 2 , the specific camera pose predictor includes a main branch, a camera pose prediction branch at the start of exposure, and a camera pose prediction branch at the end of exposure;

[0110] The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, the camera pose P t After inputting the high frequency coding layer, the first four fully connected layers output the feature map A after feature extraction;

[0111] The output of the main branch is sent to the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure at the same time;

[0112] The camera pose prediction branch at the start of exposure includes: a second first fully connected layer, a second first activation operation layer, and a second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer, and the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ;

[0113] The camera pose prediction branch at the end of exposure includes: a third fully connected layer, a third activation operation layer and a third second fully connected layer cascaded in sequence. The feature map A is input into the third first fully connected layer and then outputted by the third second fully connected layer. The camera pose P at the end of exposure t,N .

[0114] In specific applications, the number of channels of the first fully connected layer, the first second fully connected layer, and the first third fully connected layer of the main branch are all set to 64.

[0115] At the start of exposure, the number of channels of the second first fully connected layer of the camera pose prediction branch is set to 64, and the number of channels of the second second fully connected layer is set to 6.

[0116] At the end of exposure, the number of channels of the third fully connected layer of the camera pose prediction branch is set to 64, and the number of channels of the third and second fully connected layers is set to 6.

[0117] The activation function for all activation operations is the ReLU activation function.

[0118] In order to stabilize the first stage training process, a static reconstruction loss function is used during optimization. Train the static area prediction model and the camera pose prediction model. Specifically,

[0119]

[0120] In order to stabilize the second stage training process, the second stage loss function is used during optimization. The dynamic region prediction model, static region prediction model and camera pose prediction model are trained simultaneously. Specifically,

[0121]

[0122] in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, is the l1 loss function, is the structural similarity loss and ⊙ is the pixel-by-pixel multiplication operation.

[0123] The present invention introduces exposure time regularization term, multi-frame consistency regularization term and multi-resolution consistency regularization term to constrain the unrestricted optimization problem. In order to suppress trivial solutions, the exposure time regularization term is introduced; in order to alleviate the rendering image artifacts, the multi-frame consistency regularization term and multi-resolution consistency regularization term are introduced.

[0124] Specifically, during the first and second phases of training, a multi-resolution consistency regularization term is introduced Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment; where:

[0125]

[0126] in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, It is a clear image at the i-th moment in the exposure time rendered by the 4DGS model obtained by training the downsampled monocular blurry video, ||||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i=1,2…N, N is the total number of time steps in the exposure time.

[0127] Since the reconstruction results at each resolution should be consistent and the blur at low resolution is easier to remove, a multi-resolution consistency constraint is introduced. The reconstruction results of the low-resolution model are used to guide the reconstruction of the high-resolution model to avoid image artifacts; the low-resolution model is a 4DGS model trained with downsampled monocular blurred video.

[0128] During the first and second phases of training, a multi-frame consistency regularization term is introduced. Constrain the content consistency of the clear image at all moments in the exposure time corresponding to the t-th frame image of each segment; wherein,

[0129]

[0130] Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, The pre-trained optical flow estimation network will Towards Alignment obtained, The pre-trained optical flow estimation network will Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

[0131] During a certain exposure period, the content of each frame should be consistent. Therefore, a multi-frame consistency constraint is introduced. Constrain the contents of adjacent frames to remain consistent and constrain the contents of each frame to remain consistent with the first frame to alleviate image artifacts.

[0132] During the second stage of training, the exposure time regularization term is introduced Distinguishing the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model;

[0133]

[0134] Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each fragment, and max() is the maximum value operation.

[0135] Continuous Dynamic Gaussian There should be some distinction within a certain exposure time. In other words, w t The value of should not be too small. t Too small, D t,i will be almost the same as D t Consistent, leading to the situation of trivial solution, so the exposure time regularization term is introduced To alleviate the ordinary solution.

[0136] Principle analysis: By estimating the continuous camera pose and continuous dynamic Gaussian during the exposure time, the camera motion and object motion during the exposure time are simulated to estimate the clear scene representation. In addition, a blur-aware variable standard dynamic Gaussian strategy is introduced to better model scenes with large motion amplitudes, simplify the reconstruction process, and increase the reconstruction speed; at the same time, exposure time regularization terms, multi-frame consistency regularization terms, and multi-resolution consistency regularization terms are introduced to constrain unrestricted optimization problems. Among them, the introduction of the exposure time regularization term To alleviate trivial solutions, introduce multi-frame consistency constraints and multi-resolution consistency constraints To alleviate image artifacts and further improve the reconstructed visual quality.

[0137] Verification test:

[0138] See also Figure 3 , Figure 3 A visual quality comparison chart of the reconstruction method of the present invention and the reconstruction results of the existing model is given. Figure 3 It is shown that the reconstruction method of the present invention is closer to the clear truth image, and the reconstruction effect is significantly higher than other existing technologies.

[0139] Combined with the above, it is proved that the monocular blurred video reconstruction method of the present invention based on 4D Gaussian splashing is significantly better than the current optimal method.

[0140] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A monocular blurred video reconstruction method based on 4D Gaussian splashing, characterized in that: The method comprises the following steps: S1. Construct training samples: take each frame of a monocular blur video as a real blur image, and the real blur image as a training sample; divide the monocular blur video into K segments, and take the image corresponding to the frame with the highest clarity in each segment as the standard frame image; S2. Construct a 4DGS model. The 4DGS model includes a dynamic region prediction model, a static region prediction model, and a camera pose prediction model. All three are learnable models. The standard frame image corresponding to each segment in the monocular blurred video is used to establish the standard dynamic Gaussian of the dynamic area prediction model in each segment; The dynamic area prediction model predicts the dynamic area Gaussian of all frame images in each segment according to the standard dynamic Gaussian corresponding to each segment, and predicts the dynamic area Gaussian of all frame images in the segment according to the dynamic area Gaussian D of the t-th frame image in the segment. t , the dynamic area Gaussian D of the t-1th frame image t-1 and the dynamic region Gaussian D of the t+1th frame image t+1 , predict the dynamic area Gaussian at each moment in the exposure time corresponding to the t-th frame image in the segment; A static area prediction model is used to predict the static area Gaussian of the input monocular blurred video based on all frame images of the input monocular blurred video; A camera pose prediction model is used to predict the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video according to the camera pose corresponding to the t-th frame image of the segment; S3, first stage training: use training samples to train the static area prediction model and the camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is rendered with the camera pose at each moment in the exposure time corresponding to the t-th frame image of each segment, and a clear image of all moments in the exposure time corresponding to the t-th frame image of the segment is generated, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image in the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing a static reconstruction loss function Use it to modify the parameters of the static area prediction model and the camera pose prediction model to complete the training; S4, second stage training: use each training sample to train the dynamic area prediction model, static area prediction model and camera pose prediction model at the same time. During the training process, the static area Gaussian predicted by the static area prediction model is respectively rendered with the camera pose at each moment within the exposure time corresponding to the t-th frame image of each segment and the dynamic area Gaussian at the corresponding moment to generate a clear image of all moments within the exposure time corresponding to the t-th frame image of the segment, and the clear images of all moments are averaged to obtain a synthetic blurred image corresponding to the t-th frame image of the segment. And the synthetic blurred image corresponding to the t-th frame image of the segment The real blurred image B corresponding to the t-th frame image in this segment t Constructing the second stage loss function Use it to modify the parameters of the dynamic area prediction model, static area prediction model and camera pose prediction model to complete the training; S5, reasoning stage: Rendering is performed based on the camera pose at any given moment and the dynamic area Gaussian corresponding to the given moment predicted by the trained dynamic area prediction model, as well as the static area Gaussian predicted by the trained static area prediction model, to obtain a clear image corresponding to the given moment and complete the reconstruction.

2. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 1 is characterized in that: The dynamic area prediction model includes a blur perception standard dynamic Gaussian model, a dynamic deformable field model, a continuous dynamic Gaussian estimation module and an exposure time parameter model; wherein the blur perception standard dynamic Gaussian model, the dynamic deformable field model and the exposure time parameter model are all learnable parts; the standard frame image corresponding to each segment under the monocular blur video is used to establish the standard dynamic Gaussian of the blur perception standard dynamic Gaussian model in the dynamic area prediction model under each segment; A dynamic deformable field model is used to perform deformable field transformation according to a standard dynamic Gaussian under each segment to obtain dynamic area Gaussians of all frame images in the segment; The dynamic area Gaussian D of the t-1th frame image in this segment t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 ; The exposure time parameter model is used to predict the relative exposure time at each moment within the exposure time corresponding to the t-th frame image of each segment of the input monocular blurred video; The continuous dynamic Gaussian estimation module is used to estimate the dynamic area Gaussian D of the t-1th frame image corresponding to each segment according to the relative exposure time at each moment within the exposure time corresponding to the t-1th frame image of the segment. t-1 , the dynamic area Gaussian D of the t-th frame image t and the dynamic region Gaussian D of the t+1th frame image t+1 , generating D t,1 To D t,N ; Among them, D t,i is the dynamic area Gaussian at the i-th moment within the exposure time corresponding to the t-th frame image.

3. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 2 is characterized in that: Among them, w t,i It indicates that the t-th frame image is the distance from the ith exposure moment to the The relative exposure time of each exposure moment, w t,i is a floating point variable between (0,1), N is the total number of time steps within the exposure time, and ⊙ is a pixel-by-pixel multiplication operation.

4. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 3 is characterized in that: w t is the exposure time corresponding to the t-th frame image of each fragment.

5. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 1, characterized in that: The static region prediction model includes a camera pose predictor and an interpolation module; the camera pose predictor is a learnable model; The camera pose predictor is used to calculate the camera pose P corresponding to the t-th frame image of each segment of the input monocular blurred video. t , predict the camera pose P at the first moment in the exposure time corresponding to the frame image t,1 and the camera pose P at time N t,N ; Among them, the camera pose P at the first moment t,1 It is also used as the camera pose P at the start of exposure t,1 , the camera pose P at the Nth moment t,N It is also used as the camera pose P at the end of exposure t,N , N is the total number of time steps within the exposure time; The interpolation module is used to interpolate the exposure time P corresponding to the standard frame image of each fragment in the SE (3) algebraic space. t,1 and P t,N Linear interpolation is performed to obtain the camera pose P at the mth moment in the exposure time corresponding to the tth frame image of the segment. t,m , m is an integer, and 1<m<N.

6. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 5, characterized in that: Among them, exp() and log() are exponential function and logarithmic function respectively, and ⊙ is a pixel-by-pixel multiplication operation.

7. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 5, characterized in that: The camera pose predictor includes a main branch, a camera pose prediction branch at the start of exposure, and a camera pose prediction branch at the end of exposure; The main branch includes: a high-frequency coding layer, a first fully connected layer, a first activation operation layer, a first second fully connected layer, a first second activation operation, a first third fully connected layer, a first third activation operation layer and a first fourth fully connected layer, the camera pose P t After inputting the high frequency coding layer, the first four fully connected layers output the feature map A after feature extraction; The output of the main branch is sent to the camera pose prediction branch at the start of exposure and the camera pose prediction branch at the end of exposure at the same time; The camera pose prediction branch at the start of exposure includes: a second first fully connected layer, a second first activation operation layer, and a second second fully connected layer, which are cascaded in sequence. The feature map A is input into the second first fully connected layer, and the second second fully connected layer outputs the camera pose P at the start of exposure. t,1 ; The camera pose prediction branch at the end of exposure includes: a third fully connected layer, a third activation operation layer and a third second fully connected layer cascaded in sequence. The feature map A is input into the third first fully connected layer and then outputted by the third second fully connected layer. The camera pose P at the end of exposure t,N .

8. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 1, characterized in that: in, and Represent the synthetic blurred images The static area and the real blurred image B t The static area, M t is the mask of the dynamic area of ​​the t-th frame image, β is the loss function coefficient, for Loss function, is the structural similarity loss and ⊙ is the pixel-by-pixel multiplication operation.

9. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 1, characterized in that: During the first and second phases of training, a multi-resolution consistency regularization term is introduced. Constrain the resolution consistency of the clear image at all moments during the exposure time corresponding to the t-th frame image of each segment; where: in,(.) ↓ is the image downsampling operation, sg() is the gradient truncation operation, M t,i for The mask of the dynamic area, is the clear image at the i-th moment in the exposure time corresponding to the t-th frame image of any segment, is the clear image at the i-th moment in the exposure time rendered by the 4DGS model trained with the downsampled monocular blurry video, || ||1 is the l1 norm, ⊙ is the pixel-by-pixel multiplication operation, i=1,2…N, N is the total number of time steps in the exposure time; During the first and second phases of training, a multi-frame consistency regularization term is introduced. Constrain the content consistency of the clear image at all moments in the exposure time corresponding to the t-th frame image of each segment; wherein, Among them, M t,j and M t,1 They are and The mask of the dynamic area, is the clear image at the jth moment in the exposure time corresponding to the tth frame image of any segment, The pre-trained optical flow estimation network will Towards Alignment obtained, The pre-trained optical flow estimation network will Towards Alignment obtained, M t,i and M t,1 They are used to indicate and The mask of the dynamic area.

10. The monocular blurred video reconstruction method based on 4D Gaussian splashing according to claim 2, characterized in that: During the second stage of training, the exposure time regularization term is introduced Distinguishing the relative exposure time of each moment within the exposure time corresponding to the t-th frame image predicted by the exposure time parameter model; Among them, ∈ represents the threshold, w t is the exposure time corresponding to the t-th frame image of each fragment, and max() is the maximum value operation.

Images