Deblurring real dataset construction method based on 3D gaussian splatter

Abstract

A deblurring real dataset construction method based on 3D Gaussian splash belongs to the field of image restoration. The method aims at the problem that the existing image deblurring dataset is difficult to obtain a large number of real training data flexibly. The method comprises the following steps: collecting a clear image sequence and a blurred image sequence in the same scene; using 3D Gaussian splash to construct a three-dimensional reconstruction scene for the clear image sequence, and using the optimized camera pose obtained based on the current blurred image to render the three-dimensional reconstruction scene to generate a rendered clear image; and constructing a blurred-clear image pair from the current blurred image and the rendered clear image. The method improves the alignment accuracy between the rendered image and the blurred image.

Description

Technical Field

[0001] This invention relates to a method for constructing a deblurred real dataset based on 3D Gaussian splashing, belonging to the field of image restoration. Background Technology

[0002] Image deblurring is an important research direction in the field of image restoration, aiming to recover sharp images from blurred images caused by camera motion or the motion of moving objects. In recent years, deep learning-based methods have made significant progress in image deblurring tasks. These methods typically rely on a large amount of paired data of blurred and sharp images for supervised training, so the scale and quality of the training data have a significant impact on model performance.

[0003] Existing methods for constructing deblurred datasets mainly fall into two categories: synthetic data generation and real-scene acquisition. Synthetic methods typically generate blurred images by superimposing multiple frames of images, but the resulting blur differs from the actual imaging process of a real camera, resulting in insufficient realism. On the other hand, real-scene acquisition methods usually rely on complex devices such as robotic arms or beam splitters to obtain blurred-sharp image pairs. These systems are complex in structure and lack portability, limiting the flexibility and scalability of data acquisition.

[0004] Therefore, how to achieve flexible data collection while ensuring the realism of the fuzzy dataset, and how to construct a high-quality real defuzzy dataset, remains an urgent problem to be solved. Summary of the Invention

[0005] To address the problem that existing image deblurring datasets struggle to flexibly acquire large amounts of real training data, this invention provides a method for constructing a real deblurring dataset based on 3D Gaussian splashing.

[0006] The present invention provides a method for constructing a deblurred real dataset based on 3D Gaussian splashing, comprising:

[0007] Acquire clear and blurry image sequences within the same scene;

[0008] A 3D reconstructed scene is constructed using 3D Gaussian splashing on a clear image sequence, and the 3D reconstructed scene is rendered using an optimized camera pose obtained based on the current blurred image to generate a rendered clear image.

[0009] A blurred-sharp image pair is constructed from the current blurred image and the rendered sharp image.

[0010] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing, the method for obtaining the optimized camera pose based on the current blurred image is as follows:

[0011] The initial camera pose in a 3D scene is estimated using a 3D reconstruction system, and then optimized using a pose optimization method based on fuzzy perception to obtain the optimized camera pose.

[0012] According to the method for constructing a deblurred real dataset based on 3D Gaussian splashing of the present invention, the method for optimizing the initial pose of the camera using a pose optimization method based on fuzzy perception is as follows:

[0013] Set the camera's initial pose as The optimized camera pose is ;

[0014] in Let be the initial rotation matrix. Let be the initial translation vector. To optimize the rotation matrix, The optimized translation vector;

[0015] , ,

[0016] In the formula For rotation increments, This is the translation increment;

[0017] First initialize , Based on the obtained rendered clear image and blurry images Construct optimization constraints for rotation increments Translational increment Iterative updates will be performed.

[0018] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing is used to perform rotation increment... Translational increment The method for iterative updates is as follows:

[0019] ,

[0020] In the formula It is the overall loss function.

[0021] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing has an overall loss function. for:

[0022] ,

[0023] In the formula These are the weighting coefficients. For appearance consistency constraint loss, This represents the centroid alignment loss.

[0024] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing, appearance consistency constraint loss for:

[0025] ,

[0026] In the formula This indicates a downsampling operation.

[0027] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing, centroid alignment loss for:

[0028] ,

[0029] In the formula For fuzzy kernel estimates The center of mass, For fuzzy kernel estimates The geometric center;

[0030] ,

[0031] In the formula For fuzzy kernel, To achieve clear image gradients after rendering, For the gradient of the blurred image, For regularization parameters, For fuzzy kernel gradient, This is a convolution operation;

[0032] ,

[0033] In the formula The x-coordinate of the element's position in the fuzzy kernel estimate. The vertical coordinate of the element's position in the fuzzy kernel estimate. The height of the fuzzy kernel estimate, The width of the fuzzy kernel estimate.

[0034] According to the present invention, the method for constructing a deblurred real dataset based on 3D Gaussian splashing is as follows: the blurred image sequence is acquired by shooting with a handheld camera; the clear image sequence is acquired by shooting with a gimbal-stabilized camera; and the imaging parameters of the clear image sequence and the blurred image sequence are consistent.

[0035] According to the method for constructing a deblurred real dataset based on 3D Gaussian splashing of the present invention, there are overlapping regions between adjacent image frames in the clear image sequence.

[0036] According to the method for constructing a deblurred real dataset based on 3D Gaussian splashing of the present invention, both the clear image sequence and the blurred image sequence have a resolution of 4K.

[0037] The beneficial effects of this invention are as follows: The method of this invention includes a fuzzy data pair acquisition method and a fuzzy perception-based pose optimization method. Specifically, the fuzzy data pair acquisition method acquires dense, clear images of a scene using a gimbal device and acquires blurred images of the same scene using a handheld camera; based on the clear images, a 3D representation of the scene is constructed using 3D Gaussian splashing, and the pose of the blurred images is calibrated; then, corresponding clear images are generated through rendering as pairing data. The fuzzy perception-based pose optimization method optimizes the camera pose through appearance consistency constraints and centroid alignment constraints, thereby improving the alignment accuracy between the rendered image and the blurred image.

[0038] This invention combines real-scene video acquisition with 3D scene reconstruction technology to construct realistic blurred-sharp image pairing data without the need for complex dedicated hardware, thereby achieving the construction of a high-quality realistic deblurred dataset. First, this invention maintains consistent camera parameters between the two acquisition methods to ensure photometric consistency between the images. Then, it performs camera pose estimation on the blurred image in the 3D scene, and based on the estimated pose, renders a sharp image spatially aligned with the blurred image in the 3D scene, achieving the pairing of blurred and sharp images.

[0039] Compared with existing technologies, this invention can achieve flexible data acquisition while ensuring the realism of the blurred image, and generate a clear image precisely aligned with the blurred image through 3D scene reconstruction, thereby constructing a high-quality and diverse real-world image deblurring dataset. Experimental results show that the deblurring model trained on the dataset constructed based on the method of this invention has better generalization performance in real-world scenes. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method for constructing a deblurred real dataset based on 3D Gaussian splashing as described in this invention;

[0041] Figure 2 This is a schematic diagram of the framework of a pose optimization method based on fuzzy perception;

[0042] Figure 3 This is a comparison image of the blurred image effects obtained using the method of this invention and existing model methods. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Specific Implementation Method 1: Combination Figure 1 and Figure 2 As shown, this invention provides a method for constructing a deblurred real dataset based on 3D Gaussian splashing, including:

[0045] Acquire clear and blurry image sequences within the same scene;

[0046] A 3D reconstructed scene is constructed using 3D Gaussian splashing on a clear image sequence, and the 3D reconstructed scene is rendered using an optimized camera pose obtained based on the current blurred image to generate a rendered clear image.

[0047] A blurred-sharp image pair is constructed from the current blurred image and the rendered sharp image.

[0048] The method for acquiring fuzzy data pairs can be achieved by using portable devices to capture both fuzzy and clear videos, and then reconstructing the 3D scene using the clear frames to construct paired fuzzy-clear image data for subsequent model training.

[0049] Furthermore, combined with Figure 2 As shown, the method for obtaining the optimized camera pose based on the current blurred image is as follows:

[0050] The initial camera pose in a 3D scene is estimated using a 3D reconstruction system, and then optimized using a pose optimization method based on fuzzy perception to obtain the optimized camera pose.

[0051] This implementation adopts a pose optimization method based on fuzzy perception. When performing camera pose estimation, fuzzy degradation weakens the high-frequency texture information in the image, thereby reducing the accuracy of pose estimation based on feature matching, which leads to inaccurate pose estimation.

[0052] The method for optimizing the initial pose of the camera using a pose optimization method based on fuzzy perception is as follows:

[0053] Set the camera's initial pose as The camera pose can be obtained using GLOMAP based on a blurred image. Based on this, the optimized camera pose is... ;

[0054] in Let be the initial rotation matrix. Let be the initial translation vector. To optimize the rotation matrix, The optimized translation vector;

[0055] , ,

[0056] In the formula For rotation increments, This is the translation increment;

[0057] First initialize , Based on the optimized camera pose Under the condition of fixed 3D scene representation parameters, a corresponding clear image is rendered from the 3D scene; based on the obtained rendered clear image... and blurry images Construct optimization constraints for rotation increments Translational increment Iterative updates will be performed.

[0058] For rotation increment Translational increment The method for iterative updates is as follows:

[0059] ,

[0060] In the formula It is the overall loss function.

[0061] Overall loss function Camera pose is optimized by jointly optimizing appearance consistency loss and centroid alignment loss:

[0062] ,

[0063] In the formula These are the weighting coefficients. This is an appearance consistency constraint loss, used to ensure the consistency of the low-frequency structure between blurred and sharp images; This is the centroid alignment loss, used to reduce the spatial offset between the blurred and sharp images. It is achieved by constraining the centroid of the blur kernel to be consistent with its geometric center.

[0064] Appearance consistency constraint loss for:

[0065] ,

[0066] In the formula This indicates a downsampling operation.

[0067] Centroid alignment loss for:

[0068] ,

[0069] In the formula For fuzzy kernel estimates The center of mass, For fuzzy kernel estimates The geometric center;

[0070] First, the fuzzy kernel is estimated through the following optimization problem:

[0071] ,

[0072] In the formula For fuzzy kernel, To achieve clear image gradients after rendering, For the gradient of the blurred image, For regularization parameters, For fuzzy kernel gradient, This is a convolution operation; Represents the gradient operator;

[0073] ,

[0074] In the formula The x-coordinate of the element's position in the fuzzy kernel estimate. The vertical coordinate of the element's position in the fuzzy kernel estimate. The height of the fuzzy kernel estimate, The width of the fuzzy kernel estimate.

[0075] The above methods can effectively improve the accuracy of pose estimation for blurred images, thereby enhancing the spatial alignment accuracy between the rendered clear image and the blurred image.

[0076] In this embodiment, the blurred image sequence is acquired by shooting with a handheld camera; the clear image sequence is acquired by shooting with a gimbal-stabilized camera; the imaging parameters of the clear image sequence and the blurred image sequence are the same.

[0077] The clear image sequence has overlapping areas between adjacent image frames.

[0078] The blurred data acquisition method comprises two steps: data acquisition and reconstruction, and paired data alignment. In the data acquisition step, for each scene, video is acquired using both handheld and gimbal-stabilized shooting methods. Handheld shooting is used to obtain image sequences containing motion blur, while gimbal-stabilized shooting is used to obtain sharp image sequences. For sharp video acquisition, the camera is moved slowly and smoothly to ensure sufficient overlap between adjacent frames, thus meeting the requirements of 3D reconstruction. Both acquisition methods use the same imaging parameters, including ISO, aperture, shutter speed, and white balance, to ensure photometric consistency between blurred and sharp images. Based on the obtained sharp image sequences, a 3D representation of the scene is constructed using 3D Gaussian splashing, and this reconstructed 3D scene representation serves as the carrier for sharpness supervision in subsequent steps.

[0079] As an example, both the clear image sequence and the blurred image sequence have a resolution of 4K to obtain a high-density, high-resolution clear image sequence, thereby improving the quality of 3D reconstruction.

[0080] In the data alignment step, the first step is to use a 3D reconstruction system (GLOMAP) to align each blurred image. The camera pose in the 3D scene is estimated to obtain the initial pose; then, the initial pose is further optimized by a pose optimization method based on fuzzy perception to obtain a more accurate alignment result; finally, a clear image is generated from the 3D scene based on the optimized pose, and it is used as supervision information for training the deblurring model.

[0081] Figure 3 This section demonstrates the performance of different deblurring methods on various test sets. The first row shows the performance of each method on the RealBlur test set, the second row shows the performance of each method on the RBVD test set, the third row shows the performance of each method on the RSBlur test set, and the fourth row shows the performance of each method on the BSD test set.

[0082] In each column, the first column, Blur, is the original blurred image input; the second column, RealBlur, is the output performance of the model trained on the RealBlur dataset under the corresponding input; the third column is the output performance of the RBVD dataset; the fourth column is the RSBlur dataset; the fifth column is the BSD dataset; the sixth column is the GS-Blur dataset; the seventh column is the result of the GS-RealBlur dataset proposed in this invention; and the eighth column, GT, is the target image.

[0083] like Figure 3As shown, except for GS-RealBlur, the datasets perform relatively well in the same test distribution (In-Distribution), but their generalization ability is poor when facing new scenes, and they cannot effectively adapt to the changing real-world environment. The RealBlur dataset, which focuses on night scenes, performs well in night scenes, but its performance is significantly worse in other scenes, failing to effectively handle motion blur. The RBVD dataset, which focuses on camera motion blur, performs poorly in motion blur test scenes, unable to recover sufficient details in complex dynamic environments. BSD and RSBlur perform well in dynamic scenes, effectively handling motion blur and improving the edge sharpness of moving objects. However, their generalization ability in other scenes is poor; their performance drops significantly when encountering different scenes or complex backgrounds, and they cannot adapt to diverse environmental changes. Although GS-Blur provides various blur forms, because it is a simulated blur, it still differs from real-world display blur, and therefore its performance is unsatisfactory.

[0084] In contrast, the GS-RealBlur dataset of the method of this invention covers a variety of scenes and motion modes, and the blurred images are truly blurred. Therefore, it shows a significant advantage in deblurring effect, more complete detail restoration, and strong generalization ability, which can effectively cope with various complex environments.

[0085] In summary, compared with existing technologies, this invention achieves a more flexible and convenient method for acquiring realistic deblurred data by combining real-scene acquisition with 3D reconstruction. The data acquisition method does not rely on complex specialized equipment and can obtain realistic blurred images under lightweight acquisition conditions. The pose optimization method based on fuzzy perception improves the alignment accuracy between blurred and sharp images, thereby enhancing the overall quality of paired data. Experimental results show that the method of this invention has better generalization ability in deblurring tasks.

[0086] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A method for constructing a deblurred real dataset based on 3D Gaussian splashing, characterized in that... include, Acquire clear and blurry image sequences within the same scene; A 3D reconstructed scene is constructed using 3D Gaussian splashing on a clear image sequence, and the 3D reconstructed scene is rendered using an optimized camera pose obtained based on the current blurred image to generate a rendered clear image. A blurred-sharp image pair is constructed from the current blurred image and the rendered sharp image.

2. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 1, characterized in that, The method for obtaining the optimized camera pose based on the current blurred image is as follows: The initial camera pose in a 3D scene is estimated using a 3D reconstruction system, and then optimized using a pose optimization method based on fuzzy perception to obtain the optimized camera pose.

3. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 2, characterized in that, The method for optimizing the initial pose of the camera using a pose optimization method based on fuzzy perception is as follows: Set the camera's initial pose as The optimized camera pose is ; in Let be the initial rotation matrix. Let be the initial translation vector. To optimize the rotation matrix, The optimized translation vector; ， ， In the formula For rotation increments, This is the translation increment; First initialize , Based on the obtained rendered clear image and blurry images Construct optimization constraints for rotation increments Translational increment Iterative updates will be performed.

4. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 3, characterized in that, For rotational increments Translational increment The method for iterative updates is as follows: ， In the formula It is the overall loss function.

5. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 4, characterized in that, Overall loss function for: ， In the formula These are the weighting coefficients. For appearance consistency constraint loss, This represents the centroid alignment loss.

6. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 5, characterized in that, Appearance consistency constraint loss for: ， In the formula This indicates a downsampling operation.

7. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 6, characterized in that, Centroid alignment loss for: ， In the formula For fuzzy kernel estimates The center of mass, For fuzzy kernel estimates The geometric center; ， In the formula For fuzzy kernel, To achieve clear image gradients after rendering, For the gradient of the blurred image, For regularization parameters, For fuzzy kernel gradient, This is a convolution operation; ， In the formula The x-coordinate of the element's position in the fuzzy kernel estimate. The vertical coordinate of the element's position in the fuzzy kernel estimate. The height of the fuzzy kernel estimate, The width of the fuzzy kernel estimate.

8. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 1, characterized in that, The blurred image sequence was acquired by shooting with a handheld camera; the clear image sequence was acquired by shooting with a gimbal-stabilized camera; the imaging parameters of the clear image sequence and the blurred image sequence are the same.

9. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 8, characterized in that, The clear image sequence has overlapping areas between adjacent image frames.

10. The method for constructing a deblurred real dataset based on 3D Gaussian splashing according to claim 1, characterized in that, Both the clear image sequence and the blurry image sequence have a resolution of 4K.

Images