Three-dimensional scene reconstruction deblurring method, system and device and medium

Abstract

The invention discloses a three-dimensional scene reconstruction deblurring method, system, equipment and medium, and the method comprises the steps: firstly, processing a blurred image through employing a dense unconstrained three-dimensional reconstruction frame, generating an initial point cloud and a corresponding confidence score, then employing a confidence balance sampling strategy, sampling a preset number of points from the initial point cloud, and carrying out the sampling of a preset number of points; and finally, taking the high-quality sampling point cloud as initial input of a three-dimensional Gaussian primitive, performing alignment optimization on a potential sharp image and a camera track after the blurred image is decoupled, and outputting a deblurred three-dimensional scene model. By adopting the method, high-quality blurred image deblurring and new view angle synthesis in a low-illumination and dynamic scene can be realized, and the geometric accuracy and semantic definition of a reconstruction result are improved.

Description

Technical Field

[0001] This invention relates to the field of image deblurring technology, and in particular to a method, system, device and medium for deblurring three-dimensional scene reconstruction. Background Technology

[0002] With the continuous development of computer vision and graphics technologies, 3D scene reconstruction and novel perspective synthesis are showing increasingly important application value in fields such as autonomous driving and virtual reality. The core of achieving high-quality reconstruction lies in the accurate acquisition and utilization of scene information. Currently, the technological development of 3D scene reconstruction and novel perspective synthesis can be mainly divided into three categories, but each of them has certain limitations.

[0003] The first category comprises traditional structured light and multi-view geometric methods. These methods rely on tools like SfM (structured light) and COLMAP to calculate camera pose through local image feature matching, then use multi-view geometric relationships to generate an initial point cloud. However, in low-light or motion-blurred scenes, image features are blurred or noisy, easily leading to feature matching errors, inaccurate camera pose estimation, and sparse point clouds with displacement deviations. The second category is based on NeRF (a technique that uses neural networks to implicitly represent 3D scenes) and its variants, as well as 3D Gaussian scattering methods. These methods implicitly model multi-view images or combine point clouds with Gaussian distributions to achieve new perspective rendering. However, it relies on high-quality input data and cannot handle factors such as motion blur well. Geometric errors and missing semantic information in the input data will be amplified in the model, affecting the accuracy and consistency of reconstruction and rendering. The third type is deblurring technology assisted by event cameras. The principle is to take advantage of the high dynamic range and no motion blur of event cameras to decouple blurred images through event stream data to restore clear details. However, in practical applications, event camera data and RGB images differ significantly in terms of temporal sequence, spatial distribution, and signal-to-noise ratio. Multi-source data fusion faces technical bottlenecks such as information weight balancing and prior knowledge extraction, which limits the promotion and stable performance of the technology.

[0004] In summary, in the tasks of 3D scene reconstruction and new perspective synthesis under low light and dynamic environments, there are often problems such as insufficient point cloud quality, difficulty in handling motion blur, and insufficient fusion of multi-source data. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device, and medium for deblurring three-dimensional scene reconstruction, enabling high-quality deblurring and new perspective synthesis of blurred images in low-light and dynamic scenes, thereby improving the geometric accuracy and semantic clarity of the reconstruction results.

[0006] To achieve the above objectives, the present invention provides a method for deblurring a three-dimensional scene reconstruction, comprising: A dense, unconstrained stereo 3D reconstruction framework is used to process blurred images and generate initial point clouds and corresponding confidence scores. A confidence-balanced sampling strategy is adopted to sample a preset number of points from the initial point cloud to generate a high-quality sampled point cloud. The high-quality sampled point cloud is used as the initialization input of the three-dimensional Gaussian primitives. Alignment optimization is performed on the potential sharp image after decoupling the blurred image and the camera trajectory to output a deblurred three-dimensional scene model.

[0007] Optionally, the step of employing a confidence-balanced sampling strategy to sample a preset number of points from the initial point cloud to form a high-quality point cloud subset includes: The range of confidence scores is divided into multiple consecutive intervals; A fixed number of sampling points are assigned to each interval; Within each interval, a specified number of points are randomly selected with a probability proportional to the confidence score corresponding to the point cloud. Summarize all selected points within all intervals to generate a high-quality point cloud subset.

[0008] Optionally, the step of using the high-quality sampled point cloud as initialization input for three-dimensional Gaussian primitives, aligning and optimizing the decoupled latent sharp image and camera trajectory of the blurred image, and outputting a deblurred three-dimensional scene model includes: The high-quality sampled point cloud is used as the initialization input for the three-dimensional Gaussian primitives; Each blurred image is decomposed into multiple potentially sharp images arranged in a time sequence within the exposure time; Based on the time series, the camera motion trajectory causing blur is parametrically modeled to obtain the camera motion trajectory model; Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through a joint optimization process. The optimized 3D Gaussian primitives output a deblurred 3D scene model.

[0009] Optionally, the step of decomposing each blurred image into multiple potentially sharp images arranged in a time sequence within the exposure time includes: Based on the event stream data captured synchronously with the blurred image, an event-based dual integration method is used to inversely solve multiple potentially sharp images arranged in a time sequence within the exposure time from each blurred image.

[0010] Optionally, the step of parametrically modeling the camera motion trajectory causing blurring based on the time series to obtain a camera motion trajectory model includes: The camera motion trajectory that causes blurring is discretized into a pose sequence corresponding to the time series; Within the bundle adjustment framework, the initial and final adjustment poses of the pose sequence are parameterized, and interpolation calculations are performed on the intermediate poses between the initial and final adjustment poses to establish a camera motion trajectory model.

[0011] Optionally, the method of simultaneously updating the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model through a joint optimization process based on the initialized 3D Gaussian primitives includes: A potentially sharp image is rendered along the camera motion trajectory that causes blurring. The rendered potentially sharp image and the image are then converted to grayscale space and aligned. The alignment loss is then calculated. The rendered potentially sharp images are synthesized into a simulated blurred image. The simulated blurred image is then compared with the blurred image to calculate the reconstruction loss. Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through the alignment loss and the reconstruction loss.

[0012] Optionally, after the deblurred 3D scene model is output through the optimized 3D Gaussian primitives, the method further includes: Based on the deblurred 3D scene model, a corresponding image is generated under any given new camera pose through 3D Gaussian sputtering rendering, resulting in a new perspective synthetic image.

[0013] To achieve the above objectives, the present invention also provides a three-dimensional scene reconstruction and deblurring system, comprising: The point cloud and confidence score generation module is used to process blurred images using a dense unconstrained stereo 3D reconstruction framework to generate initial point clouds and corresponding confidence scores. The confidence-balanced sampling module is used to sample a preset number of points from the initial point cloud using a confidence-balanced sampling strategy to generate a high-quality sampled point cloud. The deblurred 3D reconstruction module is used to take the high-quality sampled point cloud as the initialization input of 3D Gaussian primitives, align and optimize the potential sharp image after decoupling the blurred image and the camera trajectory, and output a deblurred 3D scene model.

[0014] To achieve the above objectives, the present invention also provides a terminal device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the three-dimensional scene reconstruction deblurring method as described above.

[0015] To achieve the above objectives, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the three-dimensional scene reconstruction and deblurring method as described above.

[0016] Compared with existing technologies, this invention provides a 3D scene reconstruction deblurring method, system, device, and medium. Through a confidence-balanced sampling strategy, it ensures the effective extraction of useful geometric information in blurred image scenes to improve point cloud quality, solving the problems of point cloud sparsity and displacement deviation in traditional methods. It utilizes event flow to decouple blurred images and restores clear semantics through alignment methods, further improving reconstruction results. By optimizing the camera pose and exposure time model during scene reconstruction, it can more accurately restore details and structural information in blurred images. Simultaneously, optimizing the information weight balancing and event prior extraction of multi-source data during alignment effectively fuses event cameras and RGB images. This invention can improve the point cloud quality of 3D scene reconstruction in low-light and dynamic environments, efficiently handle motion blur, achieve full fusion of multi-source data, significantly improve the accuracy, structural consistency, and perceptual realism of new perspective rendering, and enhance the robustness and practicality of the technology in complex real-world scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a three-dimensional scene reconstruction and deblurring method provided in an embodiment of the present invention; Figure 2 This is another flowchart of a three-dimensional scene reconstruction and deblurring method provided in an embodiment of the present invention; Figure 3 These are visual comparison images of image synthesis from new perspectives in complex scenes under different 3D scene reconstruction / rendering methods provided in the embodiments of the present invention. Figure 4 These are visual comparison images of new perspective image synthesis in conventional scenes under different 3D scene reconstruction / rendering methods provided in the embodiments of the present invention; Figure 5 These are visual comparison images of synthesizing new perspective images in real scenes under different 3D scene reconstruction / rendering methods provided in the embodiments of the present invention; Figure 6This is a structural block diagram of a three-dimensional scene reconstruction and deblurring system provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 , Figure 1 This is a flowchart of a three-dimensional scene reconstruction and deblurring method provided in an embodiment of the present invention. The three-dimensional scene reconstruction and deblurring method includes steps S1 to S3: Step S1: Process the blurred image using a dense unconstrained stereo 3D reconstruction framework to generate an initial point cloud and the corresponding confidence score; For example, firstly, a multi-view blurred image set is obtained. The blurred image set contains N blurred images, each with width and height dimensions of H×W, and pixel values ​​belonging to the real number set R, denoted as H×W. , where i is the image sequence number; These blurred images are then processed using a dense, unconstrained stereo 3D reconstruction framework (DUSSt3R) to generate an initial coarse-grained point cloud. and its related confidence scores , where K represents the number of point clouds.

[0021] Step S2: Using a confidence-balanced sampling strategy, a preset number of points are sampled from the initial point cloud to generate a high-quality sampled point cloud; In an optional embodiment, step S2 includes steps S201 to S204: Step S201: Divide the range of confidence scores into multiple consecutive intervals; Step S202: Assign a fixed number of sampling points to each interval; Step S203: Within each interval, randomly select a specified number of points according to the probability proportional to the confidence score corresponding to the point cloud; Step S204: Summarize all selected points within the intervals to generate a high-quality sampling point cloud.

[0022] It should be noted that, in this embodiment of the invention, the goal of confidence-balanced sampling is to sample L point clouds as Gaussian units for subsequent 3D scene reconstruction.

[0023] For example, the range of confidence scores. Divided into M intervals M is defined as ,in M is set according to the different 3D scene reconstruction scales, for example, 20. For each interval... Extracting subsets The sampling process can then be designed as follows: (1) in, This represents the k-th point cloud. Point cloud The confidence level, in each Within each interval, point clouds are randomly selected with probabilities proportional to their confidence levels. To ensure balanced coverage at different confidence levels, this embodiment of the invention provides that for each interval... Assign a fixed number of points Generally, when M is set to 20, the number of points in each interval is set to L / 20. This forces sparsity in low-confidence regions (susceptible to noise) and achieves dense sampling in high-confidence regions (critical structures). The resulting subset... Provides high-quality initialization for Gaussian elements in 3D scene reconstruction.

[0024] It is worth noting that existing technologies may consider using simple spatial sampling, such as farthest-point sampling, to maintain geometric coverage. While spatial sampling can ensure uniform coverage in sharp point clouds, in blurry point clouds, it risks selecting outlier clusters of points, such as duplicate or drifting points, which can further distort the reconstruction results. In contrast, the embodiments of the present invention treat the confidence map of DUSSt3R as a relative metric under blurry conditions, achieving a balance between high-quality point extraction and overall scene fidelity.

[0025] Step S3: Use the high-quality sampled point cloud as the initialization input of the three-dimensional Gaussian primitives, perform alignment optimization on the potential sharp image after decoupling the blurred image and the camera trajectory, and output the deblurred three-dimensional scene model.

[0026] In an optional embodiment, step S3 includes steps S301 to S305: Step S301: Use the high-quality sampled point cloud as the initialization input for the three-dimensional Gaussian primitives; Step S302: Decompose each blurred image into multiple potentially sharp images arranged in time sequence within the exposure time; In one optional embodiment, step S302 includes: Based on the event stream data captured synchronously with the blurred image, an event-based dual integration method is used to inversely solve multiple potentially sharp images arranged in a time sequence within the exposure time from each blurred image.

[0027] It should be noted that, in this embodiment of the invention, the event stream is introduced into the deblurring process as a high dynamic range semantic prior for guidance, and event-based double integration (EDI) is used to decouple motion-blurred images using event data.

[0028] Specifically, the initial acquisition and input of the blurred image In the exposure range Internal and corresponding event boxes Acquired simultaneously. We indicate that at time... Potentially sharp images are Based on the event imaging process, the time within each event box... Potentially sharp images It can be represented as: (2) in, It is the event camera response threshold, assuming The blur in the image is a time-weighted combination of multiple images, and the exposure time is evenly divided into u segments. It can be approximated as the average of these potentially sharp images.

[0029] (3) Therefore, we can further deduce that at any time k and : (4) (5) Therefore, each blurred image All can be decomposed into a set of potentially sharp images Furthermore, each potentially sharp image contains richer texture information.

[0030] Step S303: Perform parametric modeling on the camera motion trajectory that causes blurring based on the time series to obtain the camera motion trajectory model; In one optional embodiment, step S303 includes: The camera motion trajectory that causes blurring is discretized into a pose sequence corresponding to the time series; Within the bundle adjustment framework, the initial and final adjustment poses of the pose sequence are parameterized, and interpolation calculations are performed on the intermediate poses between the initial and final adjustment poses to establish a camera motion trajectory model.

[0031] Step S304: Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through a joint optimization process. In an optional embodiment, step S304 includes: A potentially sharp image is rendered along the camera motion trajectory that causes blurring. The rendered potentially sharp image and the image are then converted to grayscale space and aligned. The alignment loss is then calculated. The rendered potentially sharp images are synthesized into a simulated blurred image. The simulated blurred image is then compared with the blurred image to calculate the reconstruction loss. Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through the alignment loss and the reconstruction loss.

[0032] Step S305: Output the deblurred 3D scene model using the optimized 3D Gaussian primitives.

[0033] Specifically, during exposure, a motion-blurred image can be viewed as a sequence of potential images captured along the camera's motion trajectory, which can be determined by pose. Discretization. Therefore, embodiments of the present invention, within the framework of bundle adjustment, [discretize]. Parameterization and optimization are performed to jointly recover the camera motion trajectory that caused the blurring.

[0034] Furthermore, parameterized blurred image The pose adjustment amount at the start of the corresponding camera motion trajectory and blurry images The pose adjustment amount at the termination time in the corresponding camera motion trajectory And linear interpolation of the intermediate pose is performed in the Lie algebra of SE(3) (a three-dimensional special Euclidean group used to represent rigid body transformations in three-dimensional space), where, This represents pose adjustment. Therefore, at any time k, the corresponding potential sharp image... Actual pose for: (6) in, , Represents a blurred image The initial pose.

[0035] The optimization process can combine the learnable parameters of Gaussian units. and Optimization is required. Note that since the pose generated by DUSt3R may not be optimal, the original pose of the blurred image can be refined during implementation.

[0036] In each optimization iteration, embodiments of the present invention simultaneously render from the 3D-GS (3D-Gaussian Splatting) rendering framework along the path corresponding to each blurred view. Camera trajectory Rendering a potentially sharp image Then these rendering results are compared with each blurred image. The true potential sharpness of the image after decoupling Alignment.

[0037] However, although these rendered images are potentially sharper than blurry images While containing more texture details, it still cannot fully recover the ideal latent frame and often suffers from low visual fidelity (especially color information), thus affecting the optimization effect. To address this, embodiments of the present invention convert the rendered latent sharp image and the real latent sharp image into grayscale space before aligning them to eliminate color interference while preserving sufficient texture and shape cues.

[0038] Specifically, the alignment loss is calculated using the following formula. : (7) in, This represents the grayscale conversion function.

[0039] Subsequently, to simulate the formation process of a blurred image, embodiments of the present invention perform... The average of the potentially sharpened images rendered in the image is used to approximate equation (3), which simulates the original blurred image. The average of the potential sharp images rendered in the image is: And calculate the reconstruction loss: (8) in, This represents the loss due to dual structural similarity. The weighting coefficients represent the loss of dual structural similarity.

[0040] Then, the total loss is constructed by combining the alignment loss and the reconstruction loss: + (9) in, These are the weighting coefficients for alignment damage.

[0041] It is worth noting that formula (7) promotes convergence to the correct pose by combining EDI-based event priors. In addition, it helps the model avoid suboptimal solutions caused by homogeneous pose update formula (8) under blurred image gradients. The embodiments of the present invention jointly enhance the geometric, texture and color fidelity of multi-view images, and push the deblurring performance of the model to a new level through the total loss formula.

[0042] In an optional embodiment, after step S305, the method further includes: Based on the deblurred 3D scene model, a corresponding image is generated under any given new camera pose through 3D Gaussian sputtering rendering, resulting in a new perspective synthetic image.

[0043] Understandably, the deblurred 3D scene model already contains a set of 3D Gaussian primitives optimized by event guidance and calibrated by bundle adjustment. Each Gaussian primitive has precise spatial position, scale parameters, rotational attitude, and color attributes, and completely eliminates spatial structure distortion caused by camera motion in the original blurred image. Subsequently, it can receive any given new camera pose parameters, convert the new camera pose into an imaging viewpoint in the scene coordinate system based on the 3D-GS rendering framework, and finally generate a new perspective synthetic image that perfectly matches the new camera pose by pixel-level integration and color calibration of the sputtering results of all effective Gaussian primitives. This image inherits the sharp texture and precise spatial structure of the deblurred 3D scene model and can achieve a realistic presentation from any viewpoint, supporting application scenarios such as multi-view scene roaming and virtual viewpoint switching.

[0044] In summary, see Figure 2 , Figure 2 This is another flowchart of a three-dimensional scene reconstruction and deblurring method provided in an embodiment of the present invention. For example... Figure 2As shown, firstly, the blurred image is subjected to dense unconstrained stereo 3D reconstruction and restoration processing through the synergistic effect of gradient flow and forward propagation flow, outputting a point cloud and corresponding confidence score. Next, balanced sampling is performed, and the original point cloud is balanced based on the confidence score of the point cloud to generate a good point cloud (a set of high-quality sampled point clouds). At the same time, the event-based dual integration technique EDI is used to combine the event flow with the blurred image, decoupling it to obtain a sequence of potentially sharp images. Then, the camera pose is optimized, and the potentially sharp images are aggregated and loss is calculated to complete the semantic alignment. Finally, the high-quality sampled point cloud is converted into Gaussian Primitives. Through adaptive density control, the distribution density of Gaussian Primitives is adjusted to adapt to the density of scene details, projection, and differentiable rasterization, so as to realize the generation of new perspective synthetic images under any new camera pose.

[0045] This invention provides a 3D scene reconstruction deblurring method. The confidence-balanced sampling strategy employs more precise sampling weight adjustment to ensure effective extraction of useful geometric information in blurred image scenes, improving point cloud quality and solving the problems of point cloud sparsity and displacement deviation in traditional methods. It utilizes event flow to decouple blurred images and restores clear semantics through alignment methods, further improving reconstruction results. By optimizing the camera pose and exposure time model during scene reconstruction, it can more accurately restore details and structural information in blurred images. Simultaneously, optimizing the information weight balancing and event prior extraction of multi-source data during alignment effectively fuses event cameras and RGB images. This invention can improve the point cloud quality of 3D scene reconstruction in low-light and dynamic environments, efficiently handle motion blur, achieve full fusion of multi-source data, significantly improve the accuracy, structural consistency, and perceptual realism of new perspective rendering, and enhance the robustness and practicality of the technology in complex real-world scenarios.

[0046] To further verify the technical effects of the embodiments of the present invention, the embodiments of the present invention used different 3D scene reconstruction / rendering methods to perform new perspective task synthesis in three types of scenes: complex, conventional, and realistic, for performance comparison.

[0047] It should be noted that in the embodiments of the present invention, the normal scenario refers to a simple task scenario under ideal conditions, which is used to verify the basic rendering capabilities of the method; the complex scenario is an artificially constructed extreme scenario, which is used to test the robustness of the method under extreme conditions; and the real scenario is a real-world environment scenario that closely resembles actual applications, which is used to measure the practical value of the method in real-world scenarios.

[0048] It should be noted that the existing methods used in the embodiments of the present invention include: B-NeRF: A 3D scene reconstruction and new perspective synthesis method based on Neural Radiation Field (NeRF). It has basic rendering capabilities in complex scenes and is one of the classic baseline methods in this field.

[0049] B-3DGS: A method that combines three-dimensional Gaussian sputtering (3DGS) technology to achieve scene reconstruction and new perspective rendering by optimizing the spatial distribution and properties of three-dimensional Gaussians.

[0050] UFP-GS: Based on 3D Gaussian sputtering, it may be optimized in terms of uncertainty perception or feature projection to improve the robustness of scene representation.

[0051] EFN-GS: A method that combines Enhanced Feature Network (EFN) with 3D Gaussian sputtering to enhance feature extraction and improve texture and structure quality in new perspective rendering.

[0052] E²GS: An improved 3D Gaussian sputtering method, specifically designed for scene detail preservation and structural consistency, for novel perspective synthesis tasks.

[0053] BAD-NeRF: Based on neural radiation fields, it uses a balanced sampling (BAD) strategy to optimize the training process, thereby improving the accuracy and efficiency of scene reconstruction.

[0054] BAD-GS: Combining 3D Gaussian sputtering with a balanced sampling strategy, it adjusts the sampling density of the Gaussian distribution to adapt to the detail density requirements of different scenarios.

[0055] EDNeRF: Based on neural radiation fields, it may introduce event-driven (ED) mechanisms or dynamic optimization strategies to enhance the rendering performance of new perspectives in dynamic and complex scenes.

[0056] See Table 1. Table 1 mainly shows the performance of various 3D scene reconstruction / rendering methods in three types of scenes: Hard, Normal, and Real, based on PSNR (Image Precision, higher is better), SSIM (Structural Consistency, closer to 1 is better), and LPIPS (Perceptual Consistency, lower is better). The best result is shown in bold, and the second best result is indicated by an underline. As can be seen from the table, our invention achieves the best values ​​in all three metrics across all scenarios. For example, in complex scenarios, PSNR reaches 26.94, SSIM is 0.8235, and LPIPS is 0.1355; in regular scenarios, PSNR is 32.09, SSIM is 0.9453, and LPIPS is 0.0406; and in realistic scenarios, PSNR is 25.93, SSIM is 0.7995, and LPIPS is 0.0840. This comprehensively surpasses other comparative methods such as B-NeRF and B-3DGS, fully demonstrating its significant superiority in new perspective rendering tasks from the three dimensions of image quality accuracy, structural consistency, and perceptual realism.

[0057] Table 1. Quantitative comparison results of various 3D scene reconstruction / rendering methods for synthesizing new perspectives in different scenes. Furthermore, this embodiment of the invention also selects methods with better comparison results to synthesize new perspectives in real scenes under different levels of blur. See Table 2, which mainly shows the performance of B-3DGS, E²GS, BAD-GS, EDNeRF, and our invention (Ours) in PSNR, SSIM, and LPIPS metrics under different difficulty levels (Levels 1-4). Our invention achieves optimal results in all three metrics for each difficulty level: highest PSNR and SSIM values, and lowest LPIPS values. For example, in Level 1, PSNR reaches 26.84, SSIM is 0.8272, and LPIPS is 0.0648; in Level 4, PSNR is 24.72, SSIM is 0.8274, and LPIPS is 0.0862, comprehensively surpassing other comparison methods and fully demonstrating its significant superiority in image quality accuracy, structural consistency, and perceptual realism in new perspective rendering under different difficulty levels.

[0058] Table 2. Quantitative comparison results of various 3D scene reconstruction / rendering methods for synthesizing new perspectives of real scenes under different levels of blur. In addition, see Figures 3-5 , Figures 3-5These are visual comparison images of image synthesis from new perspectives in complex, conventional, and realistic scenes using different 3D scene reconstruction / rendering methods provided in the embodiments of the present invention. As can be seen from the enlarged red boxes in each image, the methods (ours) provided in the embodiments of the present invention highly match the ground truth on the far right in terms of texture detail, structural morphology, and color reproduction.

[0059] Based on the above method items, the present invention provides corresponding system items embodiments.

[0060] See Figure 6 , Figure 6 This is a structural block diagram of a 3D scene reconstruction and deblurring system provided in an embodiment of the present invention. The 3D scene reconstruction and deblurring system includes: Point cloud and confidence score generation module 21 is used to process blurred images using a dense unconstrained stereo 3D reconstruction framework to generate initial point clouds and corresponding confidence scores. The confidence-balanced sampling module 22 is used to sample a preset number of points from the initial point cloud using a confidence-balanced sampling strategy to generate a high-quality sampled point cloud. The deblurred 3D reconstruction module 23 is used to take the high-quality sampled point cloud as the initialization input of 3D Gaussian primitives, perform alignment optimization on the potential sharp image after decoupling the blurred image and the camera trajectory, and output a deblurred 3D scene model.

[0061] In one optional embodiment, the confidence balancing sampling module 22 is specifically used for: The range of confidence scores is divided into multiple consecutive intervals; A fixed number of sampling points are assigned to each interval; Within each interval, a specified number of points are randomly selected with a probability proportional to the confidence score corresponding to the point cloud. Summarize all selected points within all intervals to generate a high-quality point cloud subset.

[0062] In one optional embodiment, the deblurred 3D reconstruction module 23 includes: A Gaussian primitive initialization unit is used to take the high-quality sampled point cloud as the initialization input for the three-dimensional Gaussian primitive; The blurred image decoupling unit is used to decompose each blurred image into multiple potentially sharp images arranged in a time sequence within the exposure time; A latent motion modeling unit is used to parametrically model the camera motion trajectory that causes blurring based on the time series, so as to obtain a camera motion trajectory model; An event-guided optimization unit is used to simultaneously update the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model through a joint optimization process based on the initialized 3D Gaussian primitives. The 3D reconstruction unit is used to output a deblurred 3D scene model through optimized 3D Gaussian units.

[0063] In one optional embodiment, the blurred image decoupling unit is specifically used for: Based on the event stream data captured synchronously with the blurred image, an event-based dual integration method is used to inversely solve multiple potentially sharp images arranged in a time sequence within the exposure time from each blurred image.

[0064] In one alternative embodiment, the potential motion modeling unit is specifically used for: The camera motion trajectory that causes blurring is discretized into a pose sequence corresponding to the time series; Within the bundle adjustment framework, the initial and final adjustment poses of the pose sequence are parameterized, and interpolation calculations are performed on the intermediate poses between the initial and final adjustment poses to establish a camera motion trajectory model.

[0065] In one optional embodiment, the event-guided optimization unit is specifically used for: A potentially sharp image is rendered along the camera motion trajectory that causes blurring. The rendered potentially sharp image and the image are then converted to grayscale space and aligned. The alignment loss is then calculated. The rendered potentially sharp images are synthesized into a simulated blurred image. The simulated blurred image is then compared with the blurred image to calculate the reconstruction loss. Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through the alignment loss and the reconstruction loss.

[0066] In an alternative embodiment, the system further includes a new perspective synthesis module for: Based on the deblurred 3D scene model, a corresponding image is generated under any given new camera pose through 3D Gaussian sputtering rendering, resulting in a new perspective synthetic image.

[0067] It should be noted that the three-dimensional scene reconstruction and deblurring system provided in this embodiment of the invention is used to execute all the process steps of the three-dimensional scene reconstruction and deblurring method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0068] This invention also provides a terminal device, such as... Figure 7The diagram shown is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, it implements the three-dimensional scene reconstruction and deblurring method as described in any of the above embodiments.

[0069] In addition, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the three-dimensional scene reconstruction and deblurring method as described in any of the above embodiments.

[0070] When the processor 31 executes the computer program, it implements the steps in the above-described embodiments of the 3D scene reconstruction and deblurring method, for example... Figure 1 The illustrated 3D scene reconstruction and deblurring method comprises all steps. Alternatively, when the processor 31 executes the computer program, it implements the functions of each module in the above-described 3D scene reconstruction and deblurring system embodiment, for example... Figure 6 The functions of each module in the 3D scene reconstruction and deblurring system are shown.

[0071] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.

[0072] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 31 can be any conventional processor. The processor 31 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0073] The memory 32 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 32 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card, etc., or the memory 32 can also be other volatile solid-state storage devices.

[0074] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 7 The structural block diagram shown is merely a structural example of the terminal device described above and does not constitute a limitation on the structure of the terminal device. The terminal device may include more or fewer components than shown, or combine certain components, or use different components.

[0075] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for deblurring 3D scene reconstruction, characterized in that, include: A dense, unconstrained stereo 3D reconstruction framework is used to process blurred images and generate initial point clouds and corresponding confidence scores. A confidence-balanced sampling strategy is adopted to sample a preset number of points from the initial point cloud to generate a high-quality sampled point cloud. The high-quality sampled point cloud is used as the initialization input of the three-dimensional Gaussian primitives. Alignment optimization is performed on the potential sharp image after decoupling the blurred image and the camera trajectory to output a deblurred three-dimensional scene model.

2. The three-dimensional scene reconstruction and deblurring method as described in claim 1, characterized in that, The confidence-balanced sampling strategy is adopted to sample a preset number of points from the initial point cloud to form a high-quality point cloud subset, including: The range of confidence scores is divided into multiple consecutive intervals; A fixed number of sampling points are assigned to each interval; Within each interval, a specified number of points are randomly selected with a probability proportional to the confidence score corresponding to the point cloud. Summarize all selected points within all intervals to generate a high-quality point cloud subset.

3. The three-dimensional scene reconstruction and deblurring method as described in claim 1, characterized in that, The process of using the high-quality sampled point cloud as initialization input for 3D Gaussian primitives, aligning and optimizing the decoupled latent sharp image and camera trajectory of the blurred image, and outputting a deblurred 3D scene model includes: The high-quality sampled point cloud is used as the initialization input for the three-dimensional Gaussian primitives; Each blurred image is decomposed into multiple potentially sharp images arranged in a time sequence within the exposure time; Based on the time series, the camera motion trajectory causing blur is parametrically modeled to obtain the camera motion trajectory model; Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through a joint optimization process. The optimized 3D Gaussian primitives output a deblurred 3D scene model.

4. The three-dimensional scene reconstruction and deblurring method as described in claim 3, characterized in that, The step of decomposing each blurred image into multiple potentially sharp images arranged in a time sequence within the exposure time includes: Based on the event stream data captured synchronously with the blurred image, an event-based dual integration method is used to inversely solve multiple potentially sharp images arranged in a time sequence within the exposure time from each blurred image.

5. The three-dimensional scene reconstruction and deblurring method as described in claim 4, characterized in that, The step of parametrically modeling the camera motion trajectory causing blurring based on the time series to obtain a camera motion trajectory model includes: The camera motion trajectory that causes blurring is discretized into a pose sequence corresponding to the time series; Within the bundle adjustment framework, the initial and final adjustment poses of the pose sequence are parameterized, and interpolation calculations are performed on the intermediate poses between the initial and final adjustment poses to establish a camera motion trajectory model.

6. The three-dimensional scene reconstruction and deblurring method as described in claim 5, characterized in that, The initialized 3D Gaussian primitives are simultaneously updated with their attribute parameters and the parameters of the camera motion trajectory model through a joint optimization process, including: A potentially sharp image is rendered along the camera motion trajectory that causes blurring. The rendered potentially sharp image and the image are then converted to grayscale space and aligned. The alignment loss is then calculated. The rendered potentially sharp images are synthesized into a simulated blurred image. The simulated blurred image is then compared with the blurred image to calculate the reconstruction loss. Based on the initialized 3D Gaussian primitives, the attribute parameters of the 3D Gaussian primitives and the parameters of the camera motion trajectory model are updated simultaneously through the alignment loss and the reconstruction loss.

7. The three-dimensional scene reconstruction and deblurring method as described in claim 1, characterized in that, Following the deblurred 3D scene model obtained by outputting the optimized 3D Gaussian primitives, the following is also included: Based on the deblurred 3D scene model, a corresponding image is generated under any given new camera pose through 3D Gaussian sputtering rendering, resulting in a new perspective synthetic image.

8. A three-dimensional scene reconstruction and deblurring system, characterized in that, include: The point cloud and confidence score generation module is used to process blurred images using a dense unconstrained stereo 3D reconstruction framework to generate initial point clouds and corresponding confidence scores. The confidence-balanced sampling module is used to sample a preset number of points from the initial point cloud using a confidence-balanced sampling strategy to generate a high-quality sampled point cloud. The deblurred 3D reconstruction module is used to take the high-quality sampled point cloud as the initialization input of 3D Gaussian primitives, align and optimize the potential sharp image after decoupling the blurred image and the camera trajectory, and output a deblurred 3D scene model.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the three-dimensional scene reconstruction deblurring method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the three-dimensional scene reconstruction and deblurring method as described in any one of claims 1 to 7.

Images