SLAM method and system for motion blurred image robustness

Abstract

The invention discloses a robust SLAM (simultaneous localization and mapping) method for a motion blurred image, which reconstructs a high-fidelity three-dimensional scene only by using blurred RGBD (red, green, blue and blue) input by utilizing geometric information in a multi-view blurred image and through an explicit modeling physical motion blurring process. According to the method, the motion blur perception visual odometer is adopted, the local motion trail is estimated in the exposure time, compared with an existing method that only the instantaneous pose of the image is estimated, the robustness of the whole SLAM system to the motion blur can be remarkably enhanced, the pose of the camera can be accurately estimated, a clear image can be rendered, and the method is suitable for large-scale popularization and application. And a high-quality three-dimensional scene is reconstructed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a SLAM method and system that is robust to motion blurred images. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology aims to enable machines to simultaneously observe the environment, build maps, and determine their own positions in unknown environments. This technology is widely used in fields such as autonomous driving, robot navigation, and virtual reality. In SLAM, "tracker" and "mapper" are two important components, each with different functions. The main task of the tracker is to estimate the pose (position and orientation) of the camera in real time. It determines the movement of the camera in three-dimensional space by analyzing the image changes between consecutive frames. The tracker usually needs to be efficient and fast so that the camera pose can be updated in real time. The task of the mapper is to build and maintain a map of the three-dimensional environment. It combines the camera pose estimated by the tracker with the environmental features to generate a dense or sparse three-dimensional map. In SLAM, the tracker and the mapper complement each other. The tracker focuses on estimating the movement of the camera in real time to ensure the accuracy of the pose of each frame; while the mapper is responsible for creating and updating the environment map and combining the information of the historical frames for global optimization. The close combination of the two enables SLAM to achieve efficient positioning and map construction in dynamic and unknown environments.

[0003] Computer 3D reconstruction technology is a close combination of computer vision and computer graphics in application. In the 3D reconstruction process, computer vision technology is first used to reconstruct a 3D scene that can be understood by the computer, and then computer graphics technology is used to render or reconstruct a 2D image or 3D virtual scene visible to the human eye based on the reconstructed 3D model. Among the 3D reconstruction methods, Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) are currently used more. NeRF uses a neural network to train a mapping (radiance field) that takes 3D point coordinates and viewing direction as input, and takes the RGB value and volume density of the spatial position under the viewing direction as output, and then combines it with volume rendering in graphics to achieve the effect of synthesizing images from a set of known viewing angles into images from a new viewing angle. NeRF can achieve photo-level image rendering, but it requires time-consuming neural network training and volume rendering, which is computationally expensive and has poor real-time performance. In comparison, 3DGS has better real-time performance. This technology uses Gaussian functions to convert discrete point cloud data into continuous density representations, which can render scenes learned from a small group of images faster and more efficiently. Since NeRF and 3DGS can achieve realistic 3D scene representation, it has been studied together with SLAM technology, demonstrating the ability to recover clear 3D scenes and camera trajectories from high-quality input images, thus forming NeRF-SLAM technology and Gaussian-SLAM technology.

[0004] However, existing NeRF-SLAM and Gaussian-SLAM techniques heavily rely on high-quality video or image inputs, and the localization and mapping performance of these methods will degrade significantly when faced with motion blurred frames often encountered in low-light or long exposure conditions.

[0005] The difficulties that motion blurred images pose to dense visual SLAM systems mainly stem from two factors:

[0006] (1) Inaccurate pose estimation during tracking: Current photorealistic dense visual SLAM algorithms rely on clear images to estimate the camera pose by maximizing photometric consistency. However, motion-blurred images commonly seen in real-world scenes violate this assumption, making it difficult to accurately recover the pose from blurred frames. These inaccurate tracking poses in turn affect the mapping process, leading to inconsistent multi-view geometry.

[0007] (2) Multi-view geometry inconsistency in mapping: The feature mismatch between multi-view blurred images introduces erroneous 3D geometric information, resulting in poor 3D map reconstruction quality. This will degrade the map reconstruction quality and thus affect the tracking process.

[0008] Combining these two factors, existing dense visual SLAM systems often show performance degradation when dealing with motion blurred images. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention proposes a SLAM method that is robust for motion blurred images, realizes NeRF-SLAM and Gaussian-SLAM at the same time, and also combines a motion blur-aware tracker and mapper, which can locate the camera trajectory corresponding to the severely blurred image, and the positioning speed is very fast.

[0010] A SLAM method for motion blurred images, comprising the following steps:

[0011] S1: Given the camera pose of the first image, the first image is mapped using the implicit neural radiance field NeRF or the explicit 3D Gaussian 3DGS to obtain the initial 3D scene representation; and the first image, depth and camera pose are added to the key frame database;

[0012] S2: sequentially input the second to nth images with motion blur after the first image into the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref , a motion blur-aware tracker is used to estimate the camera trajectory for the current frame;

[0013] S3: Calculate the motion distance of the selected pixel of the image according to the starting point and the end point of the camera trajectory. When the motion distance is greater than the set threshold, the current frame is identified as a key frame and S4 is executed; otherwise, return to S2 and replace the next image with motion blur;

[0014] S4: Build a map for the key frame data.

[0015] Furthermore, the S2 specifically includes:

[0016] S2.1: Parameterize the camera pose T of the current frame at the beginning of exposure start ∈SE(3) and T at the end of exposure end ∈SE(3):

[0017] S2.2: To synthesize the image from the reference image I ref The re-blurred pixels are, for each virtual view T uniformly sampled in the exposure interval [0, τ] of the current frame t , its pixel coordinates are converted back to the reference image I ref , and obtain image T by bilinear interpolation tThe intensity value of all virtual views T is uniformly sampled t The intensity values ​​are averaged to calculate the re-blurred pixel intensity

[0018] S2.3: The tracker then start and T end The optimization is performed to minimize the difference between the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring. The photometric consistency loss between them is used to obtain the optimized camera pose

[0019] S2.4: According to the following formula, the virtual camera posture within the exposure time t∈[0,τ] of the current frame is linearly interpolated in the Lie algebra of SE(3) to obtain the motion trajectory of the camera;

[0020]

[0021] Furthermore, the S4 specifically includes:

[0022] S4.1: Synthesize blurred images: Based on the camera motion trajectory corresponding to the key frame, interpolate n camera poses between the start and end of the camera motion trajectory; render n clear color images and depth images from these n camera poses using implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS; finally, average the intensity values ​​of the n color images to calculate the re-blurred pixel intensity and obtain the synthesized motion blurred color image B(x);

[0023] S4.2: Construct loss functions, including color loss function L color And the geometric loss function L depth ;

[0024]

[0025]

[0026] Where B(x) is the synthetic motion blurred image, B gt (x) represents the corresponding real captured blurred image; D(x) is the depth obtained by rendering the camera pose at the midpoint of the camera trajectory, and D gt (x) indicates the measured depth;

[0027] S4.3: Simultaneously optimize the camera trajectory of the keyframes and the implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS through the back-propagation algorithm;

[0028] S4.4: adding the color image and depth image of the current key frame and the optimized trajectory to the key frame database;

[0029] S4.5: Return to S2.

[0030] Furthermore, in S3, key frames are set according to the estimated motion state, including speed, optical flow, and motion blur degree.

[0031] Furthermore, the key frame selection based on speed specifically includes the following steps:

[0032] (1) Directly subtract the estimated poses of two adjacent frames to obtain the velocity vector;

[0033] (2) Taking the modulus of the velocity vector and its rotational and translational components, we obtain three velocity values ​​in SE(3), SO(3), and R(3) spaces respectively;

[0034] (3) The three rate values ​​are compared with the three corresponding thresholds. If at least one of them is greater than the threshold, the current frame is identified as a key frame.

[0035] A SLAM system robust to motion blurred images, including a motion blur aware tracker and a bundle adjustment deblur mapper;

[0036] The motion blur perception tracker is used as the front end to uniformly sample the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view T t , and the virtual view T t The pixel coordinates of are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image T is obtained by bilinear interpolation. t The intensity value of all virtual views T is uniformly sampled t The intensity values ​​are averaged to calculate the re-blurred pixel intensity Then minimize the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose T at the beginning of exposure of the current frame obtained according to the constant speed assumption. start ∈SE(3) and T at the end of exposure end ∈SE(3) to optimize, and then get the optimized camera posture

[0037] The bundle adjustment deblur mapper, as a backend, obtains n camera poses between the start point and the end point of the camera motion trajectory by interpolation based on the camera motion trajectory corresponding to the key frame; from these n camera poses, n clear color images and depth images are rendered using implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS; finally, the intensity values ​​of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion blurred color image B(x); a loss function is constructed, and the camera trajectory of the key frame and the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS are simultaneously optimized through a back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

[0038] The beneficial effects of the present invention are as follows:

[0039] The method of the present invention focuses on utilizing the geometric information present in multi-view blurred images, and reconstructs high-fidelity three-dimensional scenes using only blurred RGBD inputs by explicitly modeling the physical motion blur process. In addition, visual odometers such as those used in GO-SLAM and Orbeez-SLAM are usually difficult to track when processing frames affected by motion blur; however, the present invention uses a motion blur-aware visual odometer to estimate local motion trajectories during the exposure time rather than instantaneous poses, thereby significantly enhancing the robustness of the entire SLAM system of the present invention to motion blur, and can accurately estimate the camera pose, render clear images, and reconstruct high-quality three-dimensional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flowchart of a SLAM method robust to motion blurred images according to an embodiment of the present invention.

[0041] Figure 2 The trajectory tracking results of Ours-GS on the synthetic ArchViz dataset according to the embodiment of the present invention are shown in Figure 1. (a) shows the trajectory tracking results of Ours-GS on ArchViz-1, (b) shows the trajectory tracking results of Ours-GS on ArchViz-2, and (c) shows the trajectory tracking results of Ours-GS on ArchViz-3.

[0042] Figure 33 and 4. The figures are the rendering results of the images on the ArchViz-1 dataset according to the embodiments of the present invention and several existing methods, wherein (a) is the input blurred image, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, (d) is the rendering result when the implicit neural radiation field NeRF is used for mapping (Ours-NeRF) according to the embodiments of the present invention, (e) is the rendering result when the explicit three-dimensional Gaussian 3DGS is used for mapping (Ours-GS) according to the embodiments of the present invention, and (f) is a clear image for reference.

[0043] Figure 4 The mesh reconstruction results of the visualized three-dimensional scene on the ArchViz-1 dataset, where (a) is the mesh reconstruction result of CoSLAM, (b) is the mesh reconstruction result of SplaTAM, (c) is the mesh reconstruction result when the implicit neural radiation field NeRF is used for mapping in an embodiment of the present invention, (d) is the mesh reconstruction result when the explicit three-dimensional Gaussian 3DGS is used for mapping in an embodiment of the present invention, and (e) is a mesh for reference.

[0044] Figure 5 The figures are the rendering results of images on the real public ScanNet and TUM RGB-D datasets, where (a) is the real input blurred image, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, and (d) is the rendering result when the explicit three-dimensional Gaussian 3DGS is used for mapping (Ours-GS) in an embodiment of the present invention.

[0045] Figure 6 The figure shows the qualitative rendering results of different methods on the real captured Realsense dataset, where (a) is the real blurred image of the input, (b) is the rendering result of CoSLAM, (c) is the rendering result of SplaTAM, and (d) is the rendering result when the explicit three-dimensional Gaussian 3DGS is used for mapping (Ours-GS) in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The method of the present invention integrates an efficient motion blur-aware tracker with a mapper based on neural radiance field or Gaussian subdivision. By accurately modeling the physical image formation process of motion-blurred images, the 3D scene representation is learned and the local trajectory of the camera is estimated during the exposure time, thereby achieving active compensation for motion blur caused by camera motion.

[0048] On the one hand, if Figure 1 As shown, this embodiment provides a SLAM method that is robust to motion blurred images, comprising the following steps:

[0049] Step 1: Given the camera pose of the first image, use the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS to map the first image to obtain the initial three-dimensional scene representation; and add the first image, depth and camera pose to the key frame database.

[0050] The specific mapping process in step 1 is as follows:

[0051] 1) If NeRF is used for mapping:

[0052] Given a pixel position x and its depth λ, the ray along the camera pose to a specific 3D point X w Sampling is performed and its corresponding appearance feature f can be retrieved a and geometric features f g :

[0053] f a =F a-xy (X w )+F a-xz (X w )+F a-yz (X w )

[0054] f g =F g-xy (X w )+F g-xz (X w )+F g-yz (X w )

[0055] Among them, {F a-xy , F a-xz , F a-yz} and {F g-xy , F g-xz , F g-yz} are the appearance and geometric triplanes, respectively. Then, the original color c, the signed distance function s, and the volume density σ can be calculated as follows:

[0056] c=h a (f a ), s = h g (f g ), σ=β·sigmoid(-β·s)

[0057] Among them, h a and h gThey are appearance and geometry decoders, respectively, and β is a learnable parameter used to adjust the clarity of surface boundaries. According to the principle of volume rendering, pixel color and depth are calculated by sampling 3D points along the light as follows:

[0058]

[0059]

[0060] Where n is the number of 3D points sampled along the ray, c i and σ i are the color and volume density of the i-th sampled 3D point predicted by equation (52), d i is the depth of the ith sampling point.

[0061] 2) If 3DGS is used for mapping:

[0062] Three-dimensional Gaussian rendering uses a series of Gaussians to represent the scene. The parameters of each Gaussian G include its mean position 3D Covariance Opacity and color The distribution of each scaled Gaussian is defined as:

[0063]

[0064] In order to ensure the semi-positive definite physical property of the 3D covariance ∑ and achieve differentiable rasterization, 3DGS transforms the 3D covariance ∑ and the 2D covariance It is expressed as:

[0065] ∑=RSS T R T ,

[0066] in, is zoom, It is composed of quaternions The stored rotation matrix, is the Jacobian matrix of the affine approximation of the projective transformation, R c is the rotational part of the rendered camera pose

[0067] The color of each pixel is then rendered by rasterizing these N sorted 2D Gaussians according to depth, as follows:

[0068]

[0069] Among them, c i and d i are the color and depth of each Gaussian and

[0070] Step 2: Input the 2nd to nth images with motion blur after the first image into the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS in sequence, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref ,A motion blur-aware tracker is used to estimate the camera trajectory for the current frame.

[0071] 1. Motion blur map formation model

[0072] The physical image formation process involves the digital camera collecting photons during the exposure time and converting them into a measurable charge. Mathematical modeling of this process involves the integration of a set of virtual sharp images: The process of physical motion blurred image formation involves the digital camera acquiring photons during the exposure duration and converting them into a quantifiable charge. Mathematically, this complex process requires the integration of a series of virtual sharp images:

[0073] B(x)=φ∫0 τ I t (x)dt,

[0074] in Represents the captured image, W and H represent the width and height of the image respectively, is the pixel position, φ is the normalization factor, τ is the camera exposure time, is the virtual clear image captured at timestamp t during the exposure time. The blurred image B(x) caused by the movement of the camera during the exposure time is the virtual image I at each time t. t (x). The discrete approximation of this model is as follows:

[0075]

[0076] Where n is the number of discrete samples.

[0077] 2. Modeling the camera motion trajectory

[0078] Accurate compensation of motion blur requires modeling of the local camera trajectory during exposure. To enhance robustness, this embodiment chooses to parameterize the local camera trajectory independently of the previous frame. Specifically, two camera poses are parameterized: one is T at the beginning of exposure and the other is T. start ∈SE(3), and the other is T at the end of exposure end ∈SE(3). Between these two poses, the present embodiment linearly interpolates the poses in the Lie algebra of SE(3).

[0079] The motion blur aware tracker of this embodiment operates by directly aligning the keyframe assumed to be sharp with the current frame that may be affected by motion blur. In order to exploit photometric consistency during the alignment process, either the current frame must be deblurred or the keyframe must be reblurred. This embodiment chooses the latter because reblurring is generally simpler and more robust than motion deblurring, especially for images that are severely affected by motion blur.

[0080] To synthesize the image from the reference image I ref The re-blurred pixels (to facilitate comparison with the actual captured pixel intensity) are start and T end That is, for each virtual view T uniformly sampled within the exposure interval [0, τ] of the current frame, t , its pixel coordinates are converted back to the reference image and image T is obtained by bilinear interpolation t The reblurred pixel intensity is then calculated by averaging the intensity values

[0081]

[0082] in Corresponds to the time in the clear reference frame The transition point of n is the number of virtual frames used to synthesize the blurred image.

[0083] The tracker then start and T end The optimization is performed to minimize the difference between the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring. The photometric consistency loss between them is used to obtain the optimized camera pose

[0084]

[0085] get Then, according to the following formula, the virtual camera posture within the exposure time t∈[0,τ] of the current frame is linearly interpolated in the Lie algebra of SE(3) to obtain the motion trajectory of the camera;

[0086]

[0087] Step 3: According to the starting point and end point of the camera trajectory, calculate the movement distance of the selected pixel of the image. When the movement distance is greater than the set threshold, the current frame is identified as a key frame and step 4 is executed; otherwise, return to step 2 and replace the next image with motion blur.

[0088] 1. Key frame selection strategy

[0089] Keyframes can be set based on the estimated motion state, such as speed, optical flow, motion blur, etc. In different application scenarios, keyframe selection strategies can be cascaded or parallel according to the sensitivity requirements.

[0090] First, for the speed-based key frame selection strategy, the specific steps are:

[0091] (1.1) The velocity vector is obtained by directly subtracting the estimated postures of two adjacent frames. Preferably, when a continuous smooth curve such as a spline curve or a Bezier curve is used to represent the camera trajectory, the velocity can be calculated analytically (a closed-form solution) from the estimated postures of several adjacent frames.

[0092] (1.2) Taking the modulus of the velocity vector and its rotational and translational components, we obtain the three velocity values ​​in SE(3), SO(3), and R(3) spaces, respectively.

[0093] (1.3) The three rate values ​​are compared with the corresponding three thresholds. Different sensitivity levels can be set according to different application scenarios and strategies: if all or at least one of them is greater than the threshold, it is considered to be a key frame.

[0094] In some embodiments, the three speed values ​​are set to 1.0 (meters and radians per second), 0.25 (radians per second), and 1.0 (meters per second), respectively.

[0095] Second, for the key frame selection strategy based on optical flow, the specific steps are:

[0096] (2.1) First, according to the estimated relative poses of two adjacent frames, the feature points of the nearest key frame are transformed to the new perspective based on the relative pose, the rotation component of the relative pose, and the translation component of the relative pose.

[0097] (2.2) Calculate the pixel distance moved by the feature point before and after the transformation and take the modulus length.

[0098] (2.3) Compare the three obtained modulus lengths with the set threshold. According to different application scenarios and strategies, different sensitivity levels can be set: if all are greater than the threshold, or if only one is greater than the threshold, it is considered a key frame.

[0099] The selection of the threshold value is related to the resolution. In some embodiments, the input video stream resolution is 640*480, and the three threshold values ​​are 70 (pixels), 30 (pixels) and 15 (pixels) respectively.

[0100] Because the method proposed in this patent can estimate the physical characteristics of the motion blur of the current frame, we can determine whether the current frame is a key frame based on the degree of motion blur of the current frame.

[0101] Third, the key frame selection strategy based on the degree of motion blur includes:

[0102] (3.1) First, according to the estimated relative pose of two adjacent frames, the feature points of the nearest key frame are transformed to the current frame perspective.

[0103] (3.2) According to the estimated change in the camera’s posture during the exposure time, the feature points are transformed to the start and end times of the exposure.

[0104] (3) Calculate the pixel distance moved by all feature points from the start to the end of the exposure time and take the modulus length.

[0105] (4) Compare the module length with a set threshold value. When the module length is greater than the threshold value, the current frame is considered to be a key frame. In some embodiments, the input video stream resolution is 640*480, and the threshold value is 64 (pixels).

[0106] Step 4: Build a map of the key frame data. Step 4 specifically includes the following sub-steps:

[0107] (4.1) Synthesize blurred images: Based on the camera motion trajectory corresponding to the key frame, obtain n camera poses between the start and end of the camera motion trajectory by interpolation; from these n camera poses, render n clear color images and depth images using implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS; finally, average the intensity values ​​of the n color images to calculate the re-blurred pixel intensity and obtain the synthesized motion blurred color image B(x);

[0108] (4.2) Construct loss function, including color loss function L color And the geometric loss function L depth ;

[0109]

[0110]

[0111] Where B(x) is the synthetic motion blurred image, B gt (x) represents the corresponding real captured blurred image; D(x) is the depth obtained by rendering the camera pose at the midpoint of the camera trajectory, and D gt (x) indicates the measured depth;

[0112] (4.3) The camera trajectory of the key frame and the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS are simultaneously optimized through the back-propagation algorithm;

[0113] (4.4) Add the color image and depth image of the current key frame and the optimized trajectory to the key frame database;

[0114] (4.5) Return to step 2.

[0115] On the other hand, the present embodiment also provides a SLAM system that is robust to motion blurred images, the system comprising a motion blur aware tracker and a bundle adjustment deblurring mapper.

[0116] The motion blur aware tracker is used as the front end to uniformly sample the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view T t , and the virtual view T t The pixel coordinates of the image are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image T is obtained by bilinear interpolation. t Then, by uniformly sampling all virtual views T t The intensity values ​​are averaged to calculate the re-blurred pixel intensity Then minimize the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose T at the beginning of exposure of the current frame obtained according to the constant speed assumption. start ∈SE(3) and T at the end of exposure end ∈SE(3) to optimize, and then get the optimized camera posture

[0117] The bundle adjustment deblur mapper, as the backend, obtains n camera poses between the start and end points of the camera motion trajectory corresponding to the key frame by interpolation; from these n camera poses, n clear color images and depth images are rendered using the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS; finally, the intensity values ​​of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion blurred color image B(x); a loss function is constructed to simultaneously optimize the camera trajectory of the key frame and the implicit neural radiance field NeRF or the explicit three-dimensional Gaussian 3DGS through the back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

[0118] In order to verify the effectiveness of the method and system of this embodiment, the method of this embodiment is compared with the following most advanced methods based on NeRF and 3DGS.

[0119] This embodiment is first compared with NeRF-based visual SLAM, namely CoSLAM (H. Wang, J. Wang, and L. Agapito, "Co-SLAM: Joint coordinate and sparse parametric encodings for neural real-time SLAM," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13293–13302.);

[0120] In addition, this embodiment also provides the results of 3DGS-based visual SLAM, namely SplaTAM (N. Keetha, J. Karhade, KM Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, "SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 21357–21366.). For Gaussian sputtering SLAM, a mesh is extracted from the estimated camera pose and rendered image and depth.

[0121] 2. Dataset

[0122] In order to evaluate the robustness and performance of the method of this embodiment, the public synthetic dataset ArchViz with camera motion blur (P. Liu, X. Zuo, V. Larsson, and M. Pollefeys, "MBA-VO: Motion Blur Aware Visual Odometry," in ICCV, 2021, pp. 5550–5559.), and the public real dataset ScanNet with camera motion blur (A. Dai, AX Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nieβner, "ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes," in CVPR, 2017, pp. 5828–5839.) and TUM RGBD (J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, "ABenchmark for the Evaluation of RGB-D SLAM Systems," in IROS, 2012, pp.573–580.), as well as real datasets with motion blur taken by the inventors using a Realsense camera.

[0123] 3. Indicators

[0124] The widely used absolute trajectory error is adopted to visualize and evaluate the tracking performance. In terms of geometric accuracy evaluation, mesh visualization is used to judge the reconstruction performance.

[0125] 4. Implementation details of the tracker and mapper of this embodiment

[0126] Tracker: To improve efficiency, this embodiment downsamples high gradient pixels to obtain sparse key points to ensure uniform distribution in the image. In addition, 9 efficient local patches are defined around each sampled sparse key point to promote better convergence. The energy function is optimized in a coarse-to-fine manner and supplemented by a robust Huber loss function to improve robustness. The CUDA-based backpropagation process implemented in the tracker is well suited for real-time applications.

[0127] Mapper: Implement the mapper using radiance fields or 3DGS. If the mapper is a radiance field: We start and T end7 virtual images are interpolated between them. If the mapper is Gaussian: The GS version of MBA-SLAM is implemented using PyTorch within the gsplat framework. The number of virtual images is set to 13. L reg The coefficient and regularization R are 1e1 and 1.0 respectively. The other configurations are the same as the default values ​​of gsplat. In terms of Gaussian management, Gaussians are added for new frames in areas where the alpha value is less than 0.5, and the default segmentation, cloning and pruning strategies in gsplat are adopted. For Gaussian color initialization, the newly added Gaussian primitives are generated on these pixels directly using the input blurred color and depth information. In addition, interfaces are created in the tracker to facilitate data transfer from the mapper to the tracker implemented based on CUDA.

[0128] 5. Evaluation on Fuzzy Datasets

[0129] Evaluation on the synthetic fuzzy dataset ArchViz,the evaluation results are shown in Table 1.

[0130] Table 1 Comparison of the method in this embodiment with the most advanced method on the ArchViz dataset

[0131]

[0132] As can be seen from Table 1, the tracking performance of the method in this embodiment is better than other state-of-the-art NeRF-SLAM and 3DGS-SLAM systems on the motion blurred ArchViz dataset. Figure 2 We show the estimated trajectory of Ours-GS on the motion blurred sequence of the ArchViz dataset. Figure 2 It can be seen that although the camera motion is very challenging, the method of this embodiment can still estimate accurate trajectories. Figure 2 Both the quantitative and qualitative results highlight the effectiveness of the method proposed in this embodiment in processing motion blurred image sequences.

[0133] Figure 3 The qualitative rendering results of different methods on the ArchViz-1 dataset are shown. Figure 3 It can be seen that both Ours-NeRF and Ours-GS in this embodiment can recover and render clear images from fuzzy inputs, and are better than the rendering results of CoSLAM and SplaTAM.

[0134] Figure 4 The mesh reconstruction results of different methods on the ArchViz-1 dataset are shown. Figure 4It can be seen that Ours-NeRF and Ours-GS in this embodiment can both reconstruct clear high-quality meshes from fuzzy inputs, and the clarity of the reconstructed meshes is better than the results of CoSLAM and SplaTAM.

[0135] At the same time, the method of this embodiment and CoSLAM in NeRF-SLAM and SplaTAM in 3DGS-SLAM are also evaluated on real data sets, and the quantitative results are shown in Table 2.

[0136] Table 2 Comparison of the method in this embodiment with the most advanced method on a real dataset

[0137]

[0138] Pub1, pub2, and pub3 in Table 2 are from the real public ScanNet and TUM RGB-D datasets, respectively. Seq1, seq2, and seq3 are from the Realsense dataset captured by the inventors. As can be seen from Table 2, the tracking performance of the method in this embodiment on the real motion blur dataset is better than other state-of-the-art NeRF-SLAM and 3DGS-SLAM systems.

[0139] Figure 5 The qualitative rendering results of different methods on the real public ScanNet and TUM RGB-D datasets are shown. Figure 6 The qualitative rendering results of different methods on the Realsense dataset captured by the inventor are shown. Figure 5 and Figure 6 It can be seen from the figure that the clarity of the image rendered by Ours-GS in this embodiment is better than the results of CoSLAM and SplaTAM.

[0140] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A SLAM method robust to motion blurred images, characterized in that: The steps include: S1: Given the camera pose of the first image, the first image is mapped using the implicit neural radiance field NeRF or the explicit 3D Gaussian 3DGS to obtain the initial 3D scene representation; and the first image, depth and camera pose are added to the key frame database; S2: sequentially input the second to nth images with motion blur after the first image into the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS, take the current input image as the current frame, and render a clear image from the implicit neural radiation field NeRF or the explicit three-dimensional Gaussian 3DGS as the reference image I according to the camera pose of the previous key frame. ref , a motion blur-aware tracker is used to estimate the camera trajectory for the current frame; S3: Calculate the motion distance of the selected pixel of the image according to the starting point and the end point of the camera trajectory. When the motion distance is greater than the set threshold, the current frame is identified as a key frame and S4 is executed; otherwise, return to S2 and replace the next image with motion blur; S4: Build a map for the key frame data.

2. The SLAM method for motion blurred images according to claim 1, characterized in that: The S2 specifically includes: S2.1: Parameterize the camera pose T of the current frame at the beginning of exposure start ∈SE(3) and T at the end of exposure end ∈SE(3); S2.2: To synthesize the image from the reference image I ref The re-blurred pixels, for each virtual view T uniformly sampled in the exposure interval [0,τ] of the current frame t , its pixel coordinates are converted back to the reference image I ref , and obtain image T by bilinear interpolation t The intensity value of all virtual views T is uniformly sampled t The intensity values ​​are averaged to calculate the re-blurred pixel intensity S2.3: The tracker then start and T end The optimization is performed to minimize the difference between the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring. The photometric consistency loss between them is used to obtain the optimized camera pose S2.4: According to the following formula, the virtual camera posture within the exposure time t∈[0,τ] of the current frame is linearly interpolated in the Lie algebra of SE(3) to obtain the motion trajectory of the camera; 3. The SLAM method for motion blurred images according to claim 1, characterized in that: The S4 specifically includes: S4.1: Synthesize blurred images: Based on the camera motion trajectory corresponding to the key frame, interpolate n camera poses between the start and end of the camera motion trajectory; render n clear color images and depth images from these n camera poses using implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS; finally, average the intensity values ​​of the n color images to calculate the re-blurred pixel intensity and obtain the synthesized motion blurred color image B(x); S4.2: Construct loss functions, including color loss function L color And the geometric loss function L depth ; Where B(x) is the synthetic motion blurred image, B gt (x) represents the corresponding real captured blurred image; D(x) selects the depth rendered by the camera pose at the midpoint of the camera trajectory, and D gt (x) indicates the measured depth; S4.3: Simultaneously optimize the camera trajectory of the keyframes and the implicit neural radiance field NeRF or explicit three-dimensional Gaussian 3DGS through the back-propagation algorithm; S4.4: adding the color image and depth image of the current key frame and the optimized trajectory to the key frame database; S4.5: Return to S2.

4. The SLAM method for motion blurred images according to claim 1, characterized in that: In S3, key frames are set according to the estimated motion state, including speed, optical flow, and motion blur degree.

5. The SLAM method for motion blurred images according to claim 1, characterized in that: For speed-based keyframe selection, the specific steps include: (1) Directly subtract the estimated poses of two adjacent frames to obtain the velocity vector; (2) Taking the modulus of the velocity vector and its rotational and translational components, we obtain three velocity values ​​in SE(3), SO(3), and R(3) spaces respectively; (3) The three rate values ​​are compared with the three corresponding thresholds. If at least one of them is greater than the threshold, the current frame is identified as a key frame.

6. A SLAM system that is robust to motion blurred images, characterized in that: Includes a motion blur-aware tracker and a bundle-adjusted deblur mapper; The motion blur perception tracker is used as the front end to uniformly sample the exposure interval [0, τ] of the blurred image of the current frame to obtain the virtual view T t , and the virtual view T t The pixel coordinates of are converted back to the virtual clear key frame image recently rendered by the bundle adjustment deblur mapper, and the image T is obtained by bilinear interpolation. t The intensity value of all virtual views T is uniformly sampled t The intensity values ​​are averaged to calculate the re-blurred pixel intensity Then minimize the intensity actually captured in the current frame and the pixel intensity synthesized from the reference image by re-blurring The photometric consistency loss between the two is the camera pose T at the beginning of exposure of the current frame obtained according to the constant speed assumption. start ∈SE(3) and T at the end of exposure end ∈SE(3) to optimize, and then get the optimized camera posture The bundle adjustment deblur mapper, as a backend, obtains n camera poses between the start point and the end point of the camera motion trajectory by interpolation based on the camera motion trajectory corresponding to the key frame; from these n camera poses, n clear color images and depth images are rendered using implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS; finally, the intensity values ​​of the n color images are averaged to calculate the re-blurred pixel intensity to obtain a synthetic motion blurred color image B(x); a loss function is constructed, and the camera trajectory of the key frame and the implicit neural radiation field NeRF or explicit three-dimensional Gaussian 3DGS are simultaneously optimized through a back-propagation algorithm, so that the mapper can render a virtual clear image within the exposure time.

Images