A SLAM method and system for motion-blurred images

Abstract

This paper discloses a SLAM method robust to motion blur. By leveraging the geometric information present in multi-view blurred images and explicitly modeling the physical motion blur process, the method reconstructs high-fidelity three-dimensional scenes using only blurred RGBD inputs. The method employs a motion blur-aware visual odometry to estimate local motion trajectories within the exposure time. Compared to existing methods that only estimate the instantaneous pose of the image, the method significantly enhances the robustness of the entire SLAM system to motion blur, enabling accurate estimation of camera pose, clear image rendering, and high-quality three-dimensional scene reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a SLAM method and system that is robust to motion blurred images. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology aims to enable machines to simultaneously observe, map, and determine their own position within an unknown environment. This technology is widely used in fields such as autonomous driving, robotic navigation, and virtual reality. In SLAM, the tracker and mapper are two key components, each with distinct functions. The tracker's primary task is to estimate the camera's pose (position and orientation) in real time. It determines the camera's motion in three-dimensional space by analyzing image changes between consecutive frames. The tracker typically needs to be efficient and fast to update the camera pose in real time. The mapper's task is to build and maintain a map of the three-dimensional environment. It combines the camera pose estimated by the tracker with environmental features to generate a dense or sparse 3D map. In SLAM, the tracker and mapper complement each other. The tracker focuses on estimating the camera's motion in real time, ensuring accurate pose for each frame; the mapper, on the other hand, is responsible for creating and updating the environment map, integrating information from past frames for global optimization. The close combination of the two enables SLAM to achieve efficient positioning and mapping in dynamic and unknown environments.

[0003] Computer-aided 3D reconstruction technology is a close application of computer vision and computer graphics. During the 3D reconstruction process, computer vision techniques are first used to reconstruct a computer-interpretable 3D scene. Computer graphics techniques are then used to render or reconstruct a 2D image or 3D virtual scene visible to the human eye based on the reconstructed 3D model. Currently, Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) are widely used 3D reconstruction methods. NeRF uses a neural network to train a mapping (radiance field) that takes 3D point coordinates and viewing direction as input and outputs the RGB values of the spatial position and volume density at that viewing direction. Combined with volume rendering from graphics, NeRF achieves the effect of synthesizing images from a new perspective from a set of images from known perspectives. While NeRF can achieve photorealistic image rendering, it requires time-consuming neural network training and volume rendering, resulting in high computational costs and poor real-time performance. In comparison, 3DGS offers better real-time performance. This technique uses a Gaussian function to convert discrete point cloud data into a continuous density representation, enabling faster and more efficient rendering of scenes learned from a small set of images. Since NeRF and 3DGS can achieve realistic 3D scene representation, they have been studied together with SLAM technology, demonstrating the ability to recover clear 3D scenes and camera trajectories from high-quality input images, thus forming NeRF-SLAM technology and Gaussian-SLAM technology.

[0004] However, existing NeRF-SLAM and Gaussian-SLAM techniques heavily rely on high-quality video or image inputs, and their localization and mapping performance degrades significantly when faced with motion-blurred frames often encountered in low-light or long-exposure conditions.

[0005] The difficulties that motion-blurred images pose to dense visual SLAM systems are mainly due to two factors:

[0006] (1) Inaccurate pose estimation during tracking: Current photorealistic dense visual SLAM algorithms rely on clear images to estimate camera pose by maximizing photometric consistency. However, motion-blurred images, common in real-world scenes, violate this assumption, making it difficult to accurately recover poses from blurred frames. These inaccurate tracking poses, in turn, affect the mapping process, leading to inconsistent multi-view geometry.

[0007] (2) Multi-view geometry inconsistency in mapping: The feature mismatch between multi-view blurred images introduces erroneous 3D geometric information, resulting in poor 3D map reconstruction quality. This will degrade the map reconstruction quality and thus affect the tracking process.

[0008] Combining these two factors, existing dense visual SLAM systems often exhibit performance degradation when dealing with motion blurred images. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention proposes a SLAM method that is robust to motion blurred images, which simultaneously implements NeRF-SLAM and Gaussian-SLAM, and also combines a motion blur-aware tracker and mapper. It can locate the camera trajectory corresponding to severely blurred images, and the positioning speed is very fast.

[0010] A SLAM method robust to motion blurred images comprises the following steps:

[0011] S1: Given the camera pose of the first image, the first image is mapped using the implicit neural radiance field (NeRF) or the explicit 3D Gaussian (3DGS) to obtain the initial 3D scene representation; and the first image, depth, and camera pose are added to the keyframe database;

[0012] S2: sequentially input the 2nd to nth images with motion blur after the first image into the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref ,A motion blur aware tracker is used to estimate the camera trajectory for the current frame;

[0013] S3: Calculate the motion distance of the selected pixel in the image based on the starting and ending points of the camera trajectory. When the motion distance is greater than the set threshold, the current frame is identified as a key frame and S4 is executed; otherwise, return to S2 and replace the next image with motion blur.

[0014] S4: Build a map for the key frame data.

[0015] Furthermore, the S2 specifically includes:

[0016] S2.1: Parameterize the camera pose T of the current frame at the start of exposure start ∈SE(3) and T at the end of exposure end ∈SE(3):

[0017] S2.2: To synthesize the image from the reference image I ref The re-blurred pixels, for each virtual view T uniformly sampled in the exposure interval [0, τ] of the current frame t , its pixel coordinates are converted back to the reference image I ref , and obtain image T by bilinear interpolation tThe intensity value of all virtual views T is uniformly sampled t The intensity values are averaged to calculate the re-blurred pixel intensity

[0018] S2.3: The tracker then start and T end Optimize to minimize the difference between the actual captured intensity in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between them is used to obtain the optimized camera pose

[0019] S2.4: According to the following formula, the virtual camera pose within the exposure time t∈[0,τ] of the current frame is linearly interpolated in the Lie algebra of SE(3) to obtain the camera motion trajectory;

[0020]

[0021] Furthermore, the S4 specifically includes:

[0022] S4.1: Synthesize blurred images: Based on the camera motion trajectory corresponding to the keyframe, interpolate n camera poses between the start and end points of the camera motion trajectory. From these n camera poses, render n sharp color images and depth images using implicit neural radiance fields (NeRFs) or explicit 3D Gaussian (3DGS). Finally, average the intensity values of the n color images to calculate the re-blurred pixel intensity, resulting in a synthesized motion-blurred color image B(x).

[0023] S4.2: Construct loss functions, including color loss function L color And the geometric loss function L depth ;

[0024]

[0025]

[0026] Where B(x) is the synthetic motion blurred image, B gt (x) represents the corresponding real captured blurred image; D(x) is the depth obtained by rendering the camera pose at the midpoint of the camera trajectory, and D gt (x) represents the measured depth;

[0027] S4.3: Simultaneously optimize the keyframe camera trajectory and the implicit neural radiance field (NeRF) or explicit 3D Gaussian (3DGS) via backpropagation.

[0028] S4.4: Add the color image and depth image of the current key frame and the optimized trajectory to the key frame database;

[0029] S4.5: Return to S2.

[0030] Furthermore, in S3, key frames are set according to the estimated motion state, including speed, optical flow, and motion blur degree.

[0031] Furthermore, the speed-based key frame selection specifically includes the following steps:

[0032] (1) Directly subtract the estimated poses of two adjacent frames to obtain the velocity vector;

[0033] (2) Taking the modulus of the velocity vector and its rotational and translational components, we obtain the three velocity values in the SE(3), SO(3), and R(3) spaces, respectively;

[0034] (3) Compare the three rate values with the three corresponding thresholds. If at least one of them is greater than the threshold, the current frame is considered a key frame.

[0035] A SLAM system robust to motion blurred images, comprising a motion blur aware tracker and a bundle adjustment deblurring mapper;

[0036] The motion blur perception tracker acts as the front end and uniformly samples the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view T t , and the virtual view T t The pixel coordinates are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image T is obtained by bilinear interpolation. t The intensity value of all virtual views T is uniformly sampled t The intensity values are averaged to calculate the re-blurred pixel intensity Then minimize the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose T at the beginning of the exposure of the current frame obtained according to the constant speed assumption. start ∈SE(3) and T at the end of exposure end ∈SE(3) to optimize and obtain the optimized camera pose

[0037] The bundle adjustment deblurring mapper, as a backend, obtains n camera poses between the start and end points of the camera motion trajectory corresponding to the key frame by interpolation; from these n camera poses, n clear color images and depth images are rendered using the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS; finally, the intensity values of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion-blurred color image B(x); a loss function is constructed, and the camera trajectory of the key frame and the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS are simultaneously optimized through the back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

[0038] The beneficial effects of the present invention are as follows:

[0039] The method of the present invention focuses on leveraging the geometric information present in multi-view blurred images, by explicitly modeling the physical motion blur process, and reconstructing high-fidelity 3D scenes using only blurred RGBD inputs. In addition, visual odometry, such as that used in GO-SLAM and Orbeez-SLAM, is generally difficult to track when processing frames affected by motion blur. However, the present invention uses a motion blur-aware visual odometry to estimate the local motion trajectory within the exposure time, rather than the instantaneous pose. This significantly enhances the robustness of the entire SLAM system of the present invention to motion blur, enabling accurate estimation of camera pose, rendering of clear images, and reconstruction of high-quality 3D scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 4 is a flow chart of a SLAM method robust to motion blurred images according to an embodiment of the present invention.

[0041] Figure 2 The trajectory tracking results of Ours-GS on the synthetic ArchViz dataset according to an embodiment of the present invention are shown in Figure 1. (a) shows the trajectory tracking result of Ours-GS on ArchViz-1, (b) shows the trajectory tracking result of Ours-GS on ArchViz-2, and (c) shows the trajectory tracking result of Ours-GS on ArchViz-3.

[0042] Figure 3Figure 2 shows the rendering results of an embodiment of the present invention and several existing methods on the ArchViz-1 dataset, where (a) is the input blurred image, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, (d) is the rendering result of an embodiment of the present invention using implicit neural radiation field NeRF for mapping (Ours-NeRF), (e) is the rendering result of an embodiment of the present invention using explicit three-dimensional Gaussian 3DGS for mapping (Ours-GS), and (f) is a clear image for reference.

[0043] Figure 4 The mesh reconstruction results of the visualized three-dimensional scene on the ArchViz-1 dataset, where (a) is the mesh reconstruction result of CoSLAM, (b) is the mesh reconstruction result of SplaTAM, (c) is the mesh reconstruction result when the implicit neural radiation field NeRF is used for mapping in an embodiment of the present invention, (d) is the mesh reconstruction result when the explicit three-dimensional Gaussian 3DGS is used for mapping in an embodiment of the present invention, and (e) is a mesh for reference.

[0044] Figure 5 The figures are the rendering results of images on the real public ScanNet and TUM RGB-D datasets, where (a) is the real blurred input image, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, and (d) is the rendering result when the explicit three-dimensional Gaussian 3DGS is used for mapping (Ours-GS) in an embodiment of the present invention.

[0045] Figure 6 These are the qualitative rendering results of different methods on the real-capture Realsense dataset, where (a) is the real blurred input image, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, and (d) is the rendering result when the explicit three-dimensional Gaussian 3DGS is used for mapping (Ours-GS) in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Our method integrates an efficient motion blur-aware tracker with a mapper based on neural radiance fields or Gaussian subdivisions. By accurately modeling the physical image formation process of motion-blurred images, we simultaneously learn a 3D scene representation and estimate the local trajectory of the camera within the exposure time, thereby actively compensating for motion blur caused by camera motion.

[0048] On the one hand, if Figure 1 As shown, this embodiment provides a SLAM method that is robust to motion blurred images, comprising the following steps:

[0049] Step 1: Given the camera pose of the first image, use the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS to map the first image to obtain the initial three-dimensional scene representation; and add the first image, depth and camera pose to the key frame database.

[0050] The specific mapping process in step 1 is as follows:

[0051] 1) If NeRF is used for mapping:

[0052] Given a pixel position x and its depth λ, the ray along the camera pose to a specific 3D point X w Sampling is performed and the corresponding appearance feature f can be retrieved a and geometric features f g :

[0053] f a =F a-xy (X w )+F a-xz (X w )+F a-yz (X w )

[0054] f g =F g-xy (X w )+F g-xz (X w )+F g-yz (X w )

[0055] Among them, {F a-xy , F a-xz , F a-yz} and {F g-xy , F g-xz , F g-yz} are the appearance and geometric triplanes respectively. Then, the original color c, signed distance function s and volume density σ can be calculated as follows:

[0056] c=h a (f a ), s=h g (f g ), σ=β·sigmoid(-β·s)

[0057] Among them, h a and h gThey are appearance and geometry decoders, respectively, and β is a learnable parameter used to adjust the clarity of surface boundaries. According to the principle of volume rendering, pixel color and depth are calculated by sampling 3D points along the light as follows:

[0058]

[0059]

[0060] Where n is the number of 3D points sampled along the ray, c i and σ i are the color and volume density of the i-th sampled 3D point predicted by equation (52), d i is the depth of the i-th sampling point.

[0061] 2) If 3DGS is used for mapping:

[0062] 3D Gaussian rendering uses a series of Gaussians to represent the scene. The parameters of each Gaussian G include its mean position. 3D covariance Opacity and color The distribution of each scaled Gaussian is defined as:

[0063]

[0064] In order to ensure the semi-positive definite physical property of 3D covariance ∑ and achieve differentiable rasterization, 3DGS combines 3D covariance ∑ and 2D covariance Expressed as:

[0065] ∑=RSS T R T ,

[0066] in, is zoom, It is composed of quaternions The stored rotation matrix, is the Jacobian matrix of the affine approximation of the projective transformation, R c Is the rotation part of the rendered camera pose

[0067] The color of each pixel is then rendered by rasterizing these N sorted 2D Gaussians according to depth, as follows:

[0068]

[0069] Among them, c i and d i are the color and depth of each Gaussian and

[0070] Step 2: Input the 2nd to nth images with motion blur after the first image into the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS in sequence, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref ,A motion blur aware tracker is used to estimate the camera trajectory for the current frame.

[0071] 1. Motion blur map formation model

[0072] The physical image formation process involves the digital camera collecting photons during the exposure time and converting them into a measurable charge. Mathematically modeling this process involves integrating a set of virtual sharp images: The physical motion-blurred image formation process involves the digital camera acquiring photons during the exposure duration and converting them into a quantifiable charge. Mathematically, this complex process requires integrating a series of virtual sharp images:

[0073] B(x)=φ∫0 τ I t (x)dt,

[0074] in Represents the captured image, W and H represent the width and height of the image respectively, is the pixel position, φ is the normalization factor, τ is the camera exposure time, Is the virtual clear image captured at time stamp t during the exposure time. The blurred image B(x) caused by the movement of the camera during the exposure time is the virtual image I at each time t. t (x). The discrete approximation of this model is as follows:

[0075]

[0076] Where n is the number of discrete samples.

[0077] 2. Modeling the camera motion trajectory

[0078] Accurately compensating for motion blur requires modeling the local camera trajectory during exposure. To enhance robustness, this embodiment chooses to parameterize the local camera trajectory independently of the previous frame. Specifically, two camera poses are parameterized: one is T at the start of exposure and the other is T. start ∈SE(3), the other is T at the end of exposure end ∈SE(3). Between these two poses, this embodiment linearly interpolates the poses in the Lie algebra of SE(3).

[0079] The motion blur-aware tracker of this embodiment operates by directly aligning a keyframe assumed to be sharp with the current frame, which may be affected by motion blur. To exploit photometric consistency during the alignment process, either the current frame must be deblurred or the keyframe must be reblurred. This embodiment chooses the latter because reblurring is generally simpler and more robust than motion deblurring, especially for images severely affected by motion blur.

[0080] To synthesize the image from the reference image I ref The re-blurred pixels (to facilitate comparison with the actual captured pixel intensity) are start and T end That is, for each virtual view T uniformly sampled within the exposure interval [0, τ] of the current frame t , its pixel coordinates are converted back to the reference image and image T is obtained by bilinear interpolation t The reblurred pixel intensity is then calculated by averaging the intensity values

[0081]

[0082] in Corresponds to the time in the clear reference frame The transition point of n is the number of virtual frames used to synthesize the blurred image.

[0083] The tracker then start and T end Optimize to minimize the difference between the actual captured intensity in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between them is used to obtain the optimized camera pose

[0084]

[0085] get Then, according to the following formula, the virtual camera posture within the exposure time t∈[0,τ] of the current frame is linearly interpolated in the Lie algebra of SE(3) to obtain the motion trajectory of the camera;

[0086]

[0087] Step 3: Based on the start and end points of the camera trajectory, calculate the motion distance of the selected pixel in the image. When the motion distance is greater than the set threshold, the current frame is identified as a key frame and step 4 is executed; otherwise, return to step 2 and replace it with the next image with motion blur.

[0088] 1. Keyframe selection strategy

[0089] Keyframes can be set based on the estimated motion state, such as speed, optical flow, motion blur, etc. In different application scenarios, keyframe selection strategies can be cascaded or parallel based on the sensitivity requirements.

[0090] First, for the speed-based keyframe selection strategy, the specific steps are:

[0091] (1.1) The velocity vector is obtained by directly subtracting the estimated poses of two adjacent frames. Preferably, when a continuous smooth curve such as a spline curve or a Bezier curve is used to represent the camera trajectory, the velocity can be calculated analytically (a closed-form solution) from the estimated poses of several adjacent frames.

[0092] (1.2) Taking the modulus of the velocity vector and its rotational and translational components, we obtain the three velocity values in SE(3), SO(3), and R(3) spaces, respectively.

[0093] (1.3) The three rate values are compared with the three corresponding thresholds. Different sensitivity levels can be set according to different application scenarios and strategies: if all or at least one of the rate values is greater than the threshold, it is considered a key frame.

[0094] In some embodiments, the three speed values are set to 1.0 (meters and radians per second), 0.25 (radians per second), and 1.0 (meters per second), respectively.

[0095] Second, for the key frame selection strategy based on optical flow, the specific steps are:

[0096] (2.1) First, according to the estimated relative poses of the two adjacent frames, the feature points of the nearest key frame are transformed to the new perspective based on the relative pose, the rotation component of the relative pose, and the translation component of the relative pose.

[0097] (2.2) Calculate the pixel distance of the feature point before and after the transformation and take the modulus length.

[0098] (2.3) Compare the three obtained moduli with the set threshold. Depending on the application scenario and strategy, different sensitivity levels can be set: if all or only one of the moduli is greater than the threshold, it is considered a key frame.

[0099] The selection of the threshold value is related to the resolution. In some embodiments, the input video stream resolution is 640*480, and the three threshold values are 70 (pixels), 30 (pixels) and 15 (pixels) respectively.

[0100] Because the method proposed in this patent can estimate the physical characteristics of the motion blur of the current frame, we can determine whether the current frame is a key frame based on the degree of motion blur of the current frame.

[0101] Third, the key frame selection strategy based on the degree of motion blur includes:

[0102] (3.1) First, based on the estimated relative poses of the two adjacent frames, the feature points of the nearest key frame are transformed to the perspective of the current frame.

[0103] (3.2) According to the estimated camera pose change during the exposure time, the feature points are transformed to the start and end times of the exposure.

[0104] (3) Calculate the pixel distance moved by all feature points from the start to the end of the exposure time and take the modulus length.

[0105] (4) Compare the modulus with a set threshold. If the modulus is greater than the threshold, the current frame is considered a key frame. In some embodiments, the input video stream resolution is 640*480, and the threshold is 64 (pixels).

[0106] Step 4: Build a map of the keyframe data. Step 4 specifically includes the following sub-steps:

[0107] (4.1) Synthesizing blurred images: Based on the camera motion trajectory corresponding to the keyframe, interpolate n camera poses between the start and end points of the camera motion trajectory; render n sharp color images and depth images from these n camera poses using implicit neural radiance fields (NeRFs) or explicit 3D Gaussian 3DGS; finally, average the intensity values of the n color images to calculate the re-blurred pixel intensity, resulting in a synthesized motion-blurred color image B(x);

[0108] (4.2) Construct loss function, including color loss function L color And the geometric loss function L depth ;

[0109]

[0110]

[0111] Where B(x) is the synthetic motion blurred image, B gt (x) represents the corresponding real captured blurred image; D(x) is the depth obtained by rendering the camera pose at the midpoint of the camera trajectory, and D gt (x) represents the measured depth;

[0112] (4.3) The camera trajectory of the key frame and the implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS are optimized simultaneously through the back-propagation algorithm;

[0113] (4.4) Add the color image and depth image of the current key frame and the optimized trajectory to the key frame database;

[0114] (4.5) Return to step 2.

[0115] On the other hand, this embodiment also provides a SLAM system that is robust to motion blurred images, which includes a motion blur-aware tracker and a bundle adjustment deblurring mapper.

[0116] The motion blur aware tracker is used as the front end, which uniformly samples the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view T t , and the virtual view T t The pixel coordinates of the bundle adjustment deblur mapper are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image T is obtained by bilinear interpolation. t Then, by uniformly sampling all virtual views T t The intensity values are averaged to calculate the re-blurred pixel intensity Then minimize the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose T at the beginning of the exposure of the current frame obtained according to the constant speed assumption. start ∈SE(3) and T at the end of exposure end ∈SE(3) to optimize and obtain the optimized camera pose

[0117] The bundle adjustment deblurring mapper, as the backend, interpolates n camera poses between the start and end points of the camera motion trajectory corresponding to the keyframe; from these n camera poses, it renders n clear color images and depth images using the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS; finally, the intensity values of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion-blurred color image B(x); a loss function is constructed to simultaneously optimize the camera trajectory of the keyframe and the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS through the back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

[0118] In order to verify the effectiveness of the method and system of this embodiment, the method of this embodiment is compared with the following most advanced methods based on NeRF and 3DGS.

[0119] This example first compares with NeRF-based visual SLAM, namely CoSLAM (H. Wang, J. Wang, and L. Agapito, “Co-SLAM: Joint coordinate and sparse parametric encodings for neural real-time SLAM,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13293–13302.);

[0120] In addition, this example also provides the results of a 3DGS-based visual SLAM, namely SplaTAM (N. Keetha, J. Karhade, KM Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, "SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 21357–21366.). For Gaussian sputtering SLAM, a mesh is extracted from the estimated camera pose and the rendered image and depth.

[0121] 2. Dataset

[0122] In order to evaluate the robustness and performance of the method of this embodiment, the public synthetic dataset ArchViz with camera motion blur (P.Liu, X.Zuo, V.Larsson, and M.Pollefeys, "MBA-VO: Motion Blur Aware Visual Odometry," in ICCV, 2021, pp.5550–5559.), as well as the public real dataset ScanNet with camera motion blur (A.Dai, AXChang, M.Savva, M.Halber, T.Funkhouser, and M.Nieβner, "ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes," in CVPR, 2017, pp.5828–5839.) and TUM RGBD (J.Sturm, N.Engelhard, F.Endres, W.Burgard, and D.Cremers, "ABenchmark for the Evaluation of RGB-D SLAM Systems," in IROS, 2012, pp. 573–580.), and a comprehensive evaluation was conducted on real datasets with motion blur captured by the inventors using a Realsense camera.

[0123] 3. Indicators

[0124] The widely used absolute trajectory error is used to visualize and evaluate the tracking performance. In terms of geometric accuracy evaluation, mesh visualization is used to judge the reconstruction performance.

[0125] 4. Implementation details of the tracker and mapper of this embodiment

[0126] Tracker: To improve efficiency, this embodiment downsamples high-gradient pixels to obtain sparse keypoints, ensuring uniform distribution across the image. Furthermore, nine highly efficient local patches are defined around each sampled sparse keypoint to promote better convergence. The energy function is optimized in a coarse-to-fine manner, supplemented by a robust Huber loss function for enhanced robustness. The CUDA-based backpropagation process implemented in the tracker is well-suited for real-time applications.

[0127] Mapper: Implement the mapper using Radiance Fields or 3DGS. If the mapper is Radiance Fields: We start and T end7 virtual images are interpolated between them. If the mapper is Gaussian subdivision: The GS version of MBA-SLAM is implemented using PyTorch within the gsplat framework. The number of virtual images is set to 13. L reg The coefficient and regularization R are set to 1e1 and 1.0, respectively. All other configurations are the same as the default values in gsplat. For Gaussian management, Gaussians are added to new frames in regions with alpha values less than 0.5, and the default segmentation, cloning, and pruning strategies in gsplat are used. For Gaussian color initialization, the newly added Gaussian primitives are generated directly at these pixels using the blurred color and depth information of the input. Furthermore, an interface is created in the tracker to facilitate data transfer from the mapper to the CUDA-based tracker.

[0128] 5. Evaluation on Fuzzy Datasets

[0129] The evaluation results on the synthetic fuzzy dataset ArchViz are shown in Table 1.

[0130] Table 1 Comparison of the method in this embodiment with the most advanced method on the ArchViz dataset

[0131]

[0132] As can be seen from Table 1, the tracking performance of the method in this embodiment is better than other state-of-the-art NeRF-SLAM and 3DGS-SLAM systems on the motion blurred ArchViz dataset. Figure 2 We show the estimated trajectory of Ours-GS on the motion blurred sequence of the ArchViz dataset. Figure 2 It can be seen that although the camera motion is very challenging, the method of this embodiment can still estimate the accurate trajectory. Figure 2 Both the quantitative and qualitative results highlight the effectiveness of the method proposed in this example in processing motion blurred image sequences.

[0133] Figure 3 The qualitative rendering results of different methods on the ArchViz-1 dataset are shown. Figure 3 It can be seen from the figure that both Ours-NeRF and Ours-GS in this embodiment can recover and render clear images from fuzzy inputs, and are better than the rendering results of CoSLAM and SplaTAM.

[0134] Figure 4 The mesh reconstruction results of different methods on the ArchViz-1 dataset are shown. Figure 4It can be seen from the figure that both Ours-NeRF and Ours-GS in this embodiment can reconstruct clear high-quality meshes from fuzzy inputs, and the clarity of the reconstructed meshes is better than the results of CoSLAM and SplaTAM.

[0135] At the same time, the method of this embodiment and CoSLAM in NeRF-SLAM and SplaTAM in 3DGS-SLAM are evaluated on real data sets, and the quantitative results are shown in Table 2.

[0136] Table 2 Comparison of the method in this embodiment with the most advanced method on a real dataset

[0137]

[0138] In Table 2, pub1, pub2, and pub3 are from the publicly available ScanNet and TUM RGB-D datasets, respectively. Seq1, seq2, and seq3 are from the Realsense dataset, captured by the inventors. As can be seen from Table 2, the tracking performance of the method in this embodiment outperforms other state-of-the-art NeRF-SLAM and 3DGS-SLAM systems on a real motion blur dataset.

[0139] Figure 5 The qualitative rendering results of different methods on the real public ScanNet and TUM RGB-D datasets are shown. Figure 6 The qualitative rendering results of different methods on the Realsense dataset captured by the inventor are shown. Figure 5 and Figure 6 It can be seen from the figure that the clarity of the image rendered by Ours-GS in this embodiment is better than the results of CoSLAM and SplaTAM.

[0140] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A SLAM method that is robust to motion blurred images, characterized in that: The steps include: S1: Given the camera pose of the first image, the first image is mapped using the implicit neural radiance field (NeRF) or the explicit 3D Gaussian (3DGS) to obtain the initial 3D scene representation; and the first image, depth, and camera pose are added to the keyframe database; S2: sequentially input the 2nd to nth images with motion blur after the first image into the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref ,A motion blur aware tracker is used to estimate the camera trajectory for the current frame; S3: Calculate the movement distance of the selected pixel of the image according to the starting point and end point of the camera trajectory. When the movement distance is greater than the set threshold, the current frame is identified as a key frame and S4 is executed. Otherwise, return to S2 and replace the next image with motion blur; S4: Mapping the key frame data, specifically including: S4.1: Synthesize blurred images: Based on the camera motion trajectory corresponding to the keyframe, interpolate n camera poses between the start and end points of the camera motion trajectory. From these n camera poses, render n sharp color images and depth images using implicit neural radiance fields (NeRFs) or explicit 3D Gaussian (3DGS). Finally, average the intensity values of the n color images to calculate the re-blurred pixel intensity, resulting in a synthesized motion-blurred color image B(x). S4.2: Construct loss functions, including color loss function L color And the geometric loss function L depth ; ； in, is the synthetic motion blurred image, represents the corresponding real captured blurred image; The depth obtained by rendering the camera pose at the midpoint of the camera trajectory is selected, and Indicates the measured depth; S4.3: Simultaneously optimize the keyframe camera trajectory and the implicit neural radiance field (NeRF) or explicit 3D Gaussian (3DGS) via backpropagation. S4.4: Add the color image and depth image of the current key frame and the optimized trajectory to the key frame database; S4.5: Return to S2.

2. The SLAM method for motion-blurred images according to claim 1, wherein The S2 specifically includes: S2.1: Parameterize the camera pose of the current frame at the start of exposure and at the end of exposure ; S2.2: To synthesize the image from the reference image I ref For each virtual view uniformly sampled within the exposure interval [0,τ] of the current frame, , its pixel coordinates are converted back to the reference image I ref , and obtain the image by bilinear interpolation The intensity value of all virtual views is uniformly sampled The intensity values are averaged to calculate the re-blurred pixel intensity ; S2.3: The tracker then and Optimize to minimize the difference between the actual captured intensity in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between them is used to obtain the optimized camera pose ; S2.4: According to the following formula, the exposure time of the current frame The virtual camera pose in Linear interpolation is performed in the Lie algebra to obtain the motion trajectory of the camera; 。 3. The SLAM method for motion blurred images according to claim 1, wherein In S3, key frames are set according to the estimated motion state, including speed, optical flow, and motion blur degree.

4. The SLAM method for motion-blurred images according to claim 1, wherein For speed-based keyframe selection, the specific steps include: (1) Directly subtract the estimated poses of two adjacent frames to obtain the velocity vector; (2) Taking the modulus of the velocity vector and its rotational and translational components, we can obtain the three velocity values in the SE(3), SO(3), and R(3) spaces respectively; (3) Compare the three rate values with the three corresponding thresholds. If at least one of them is greater than the threshold, the current frame is considered a key frame.

5. A SLAM system that is robust to motion blurred images, characterized in that: The system is used to implement the SLAM method for motion-blurred images as claimed in claim 1, the system comprising a motion-blur-aware tracker and a bundle-adjusted deblurring mapper; The motion blur aware tracker acts as the front end and uniformly samples the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view , and the virtual view The pixel coordinates are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image is obtained by bilinear interpolation. The intensity value of all virtual views is uniformly sampled The intensity values are averaged to calculate the re-blurred pixel intensity ; then minimize the actual captured intensity in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose at the beginning of the exposure of the current frame obtained according to the constant speed assumption. and at the end of exposure Optimize and obtain the optimized camera posture ; The bundle adjustment deblurring mapper, as a backend, obtains n camera poses between the start and end points of the camera motion trajectory corresponding to the key frame by interpolation; from these n camera poses, n clear color images and depth images are rendered using the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS; finally, the intensity values of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion-blurred color image B(x); a loss function is constructed, and the camera trajectory of the key frame and the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS are simultaneously optimized through the back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

Images