Vision-guided multimodal SLAM method for indoor office environment based on NeRF

Abstract

The present invention proposes a vision-dominated multimodal SLAM method for indoor office environments based on NeRF, which belongs to the field of simultaneous positioning and map construction. The method retains the advantages of existing SLAM methods in pose estimation accuracy, effectively improves map texture and material continuity, and thus achieves substantial improvement in pose estimation accuracy, real-time performance and reconstruction quality. It includes the following steps: Step 1), LIO is used to register the point cloud measurement values ​​of each input LiDAR scan to construct the geometric structure of the radiation map; Step 2), VIO is used to restore the brightness information of the map; Step 3), LIO and VIO are coupled in the non-manifold error state iterative Kalman filter framework ESIKF to update the radiation point cloud map; Step 4), the radiation point cloud map is reconstructed through NeRF volume rendering, and the radiation map is divided into several dense point cloud sub-maps; Step 5), an octree voxel grid is dynamically allocated to each sub-map to guide NeRF to sample light and render the entire scene.

Description

Technical Field

[0001] The present invention relates to a vision-dominated multimodal SLAM (Simultaneous localization and mapping) method based on Neural Radiance Fields (NeRF) applicable to complex and changeable indoor office environments, belonging to the field of simultaneous positioning and map construction. Background Art

[0002] The simultaneous localization and mapping SLAM method is a technology that uses data from multiple sensors (such as cameras, lidar, IMU, etc.) to estimate the sensor posture and simultaneously reconstruct a 3D map of the surrounding environment. SLAM technology can solve the problem of autonomous positioning and navigation of robots or devices in unknown environments, and is one of the key technologies for realizing fully autonomous mobile robots. Existing multimodal SLAM technology performs poorly when dealing with low-texture areas and limited viewing constraints in complex and changeable indoor office environments. In particular, visual SLAM systems that generate dense textures based on motion structure (SfM) and multi-view stereo (MVS) often face degradation problems in indoor office scenarios. If semantic segmentation priors, optical flow motion, or resampling and residual optimization strategies are used to remove outliers in dynamic environments, they can only effectively handle small-scale motion and often fail when encountering continuous object motion.

[0003] Neural Radiance Field (NeRF) is a computer vision technology used to generate high-quality 3D reconstruction models. NeRF can be briefly summarized as using a multilayer perceptron (MLP) neural network to implicitly represent a static 3D scene and realize arbitrary new perspective synthesis (rendering) of complex scenes, but it cannot provide good assistance for the above-mentioned shortcomings of existing multimodal SLAM systems.

[0004] As shown in the following previously applied and published Chinese patent application, application number 202410906208.3, entitled A monocular dense SLAM map construction method based on image enhancement and NeRF, which inputs low-light RGB images into a low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism, which is used to enhance the brightness and contrast of low-light images and retain image detail information, and then uses the quadtree method to homogenize feature points. However, in the above patent application, the Retinex theory cannot be effectively processed in real time under dynamic scenes or rapidly changing lighting conditions. Moreover, the use of quadtrees in processing three-dimensional spatial data is not as effective as octrees, because it can only provide two-dimensional segmentation, and there are still obvious deficiencies in the overall processing effect.

[0005] In view of this, this application is hereby filed. Summary of the invention

[0006] The NeRF-based vision-dominated multimodal SLAM method for indoor office environments described in this application aims to solve the problems existing in the above-mentioned prior art and proposes an improved solution that further combines NeRF on the basis of the multi-sensor fusion SLAM system, in order to retain the advantages of the existing SLAM method in pose estimation accuracy, effectively improve the continuity of the map's texture and material, and thus achieve substantial improvements in pose estimation accuracy, real-time and reconstruction quality.

[0007] To achieve the above design objectives, the NeRF-based vision-dominated multimodal SLAM method for indoor office environments includes the following steps:

[0008] Step 1) Register the point cloud measurements of each input LiDAR scan through LIO to construct the geometry of the radiation map;

[0009] Step 2), restore the brightness information of the map through VIO;

[0010] While LIO reconstructs the geometry of the environment, VIO recovers the radiometric information from the input color image. VIO projects from the global map to the current image and then iteratively estimates the camera pose using the projected radiometric error. Inter-frame VIO updates estimate the system state by minimizing the inter-frame PnP reprojection error, while frame-to-map VIO updates minimize the brightness error between map points and the current image.

[0011] Step 3), LIO and VIO are coupled in the non-manifold error state iterative Kalman filter framework ESIKF to update the radiation point cloud map of the entire indoor working scene;

[0012] Step 4), the radiation point cloud map of the entire indoor working scene is reconstructed through NeRF volume rendering, and the radiation map is divided into several dense point cloud sub-maps;

[0013] Step 5) As the submaps are continuously updated, an octree voxel grid is dynamically allocated for each submap to guide NeRF to sample the light and render the entire scene.

[0014] Furthermore, the step 1) includes the following steps:

[0015] Step 1.1), for the input LiDAR scan point cloud data, use IMU back propagation to compensate for intra-frame motion, and use motion compensation and dedistortion to pre-process the point cloud data input by the LiDAR;

[0016] Step 1.2) Use the preprocessed point cloud data to perform a point-to-surface LIO update, use the generalized iterative closed point GICP method to minimize the distance from each point cloud in the scan to the plane fitted to the corresponding point in the map, iteratively estimate the LiDAR pose, and then use the estimated state estimate to append new points to the map;

[0017] Step 1.3), use lidar sensing to perform geometric measurements of textureless objects, and apply lidar-based depth regularization by adding a depth loss; the depth loss added to the lidar is defined as the normal distribution around the lidar depth measurement D and the light distribution rendered in the NeRF model h ( t ), the expression is as follows:

[0018] (1)

[0019] in, To add depth loss to the LiDAR; is the normal distribution around the lidar depth measurement D; is the standard deviation of the distribution; h ( t ) is the light distribution;

[0020] Step 1.4), as the radiation map is continuously updated, the state estimation value of the point cloud data is retrieved in reverse order to continuously optimize the point cloud input.

[0021] Furthermore, the step 2) includes the following steps:

[0022] Step 2.1), perform photometric calibration on the continuous images input by the camera sensor;

[0023] For each incoming image I, we first correct the image nonlinearity CRF and extinction factor , to obtain the photometrically corrected image Γ, which is at the pixel position ρ Place i The expression of the channel is as follows:

[0024] (2)

[0025] The photometrically corrected image Γ is then used for frame-to-frame VIO update, frame-to-map VIO update, and brightness restoration;

[0026] Step 2.2), in frame-to-frame VIO update, use frame-to-frame VIO to update the map points observed in the last frame of optical flow tracking, and obtain a rough estimate of the system state by minimizing the PnP reprojection error of the tracked points;

[0027] Step 2.3), for each input keyframe, calculate the dense optical flow warp set ;

[0028] Use from matching set S The basic matrix of internal line estimation sampled in , setting the matching set S Any matching point within ( O ji , O ki ), using the matrix F Calculate the distance between the corresponding point and its epipolar line, and set the threshold e th , the warp mask calculation expression corresponding to the dynamic object is as follows:

[0029] (3)

[0030] in, is the polar coefficient, is the warp mask size aligned with the current frame image size; j and k is the key frame ID, indicating the Keyframes to The optical flow mask warping process for key frames;

[0031] Step 2.4), in the frame-to-map VIO update, further refine the state estimate by minimizing the difference between the brightness of the map point and the pixel intensity of its projected position in the current image;

[0032] Step 2.5), after the frame-to-map VIO update, use the SLAM pose as the pose prior and use COLMAP to feed back to the frame-to-map VIO update to optimize the camera pose;

[0033] According to the volume rendering method in NeRF, the prediction values ​​are accumulated along the observation ray of the current estimated pose, and the color and depth values ​​are rendered as:

[0034] (4)

[0035] (5)

[0036] in is the weight calculated along the ray, and For sampling point The color and depth values ​​of

[0037] The depth and color in the independent view are rendered and compared with the real image. The expressions are as follows:

[0038] (6)

[0039] (7)

[0040] in, C ( r1 )and D ( r1 ) represent the real image and depth map corresponding to a given posture respectively; M represents the number of sampled pixels in the current image; D(^) var is the depth variance along this ray;

[0041] (8)

[0042] Furthermore, in step 2.5), add the depth variance To reduce the weight of depth outliers, the radiosity of map points is updated according to the current image measurement through the converged state estimate and the original input image;

[0043] The following steps are included:

[0044] Step 2.5.1), for i The 2D pixel p in the frame is reprojected to the j Frame, the expression is as follows:

[0045] (9)

[0046] in, K and Denote the intrinsic matrix and coordinate system respectively j and coordinate system i The transformation matrix between yes The homogeneous coordinates of ; Represents a transformation function, which is used to transform a pixel from one perspective to another; Indicates that from j Frame to i Camera motion parameters of the frame; Represents pixel Depth information;

[0047] The Canny edge detection algorithm is used to extract edges to calculate the distance map DT to describe the projection error of the nearest edge, effectively identifying the edges in the image, which is defined as the edge set ;

[0048] Step 2.5.2), for the edge set in frame i , and marginal loss Define it, the expression is as follows:

[0049] (10)

[0050] in, Representation frame j DT mapping in, is a Huber weight function to reduce the impact of large residuals; if the projection distance error is greater than, One potential outlier will be removed;

[0051] Step 2.5.3), the pose optimization problem is expressed as follows:

[0052] (11)

[0053] in, Indicates that from j Frame to i Optimal camera motion parameters for the frame; represents the weight of the regularization term.

[0054] Furthermore, the step 4) specifically includes the following steps:

[0055] Step 4.1), using the sub-mapping strategy to divide the point cloud map of the entire indoor work scene into several sub-maps;

[0056] Step 4.1.1), trajectory segmentation;

[0057] First, the SLAM trajectory optimized by COLMAP is segmented into multiple clusters using the spectral clustering method, and the eigenvectors of the graph Laplacian matrix are used to represent the data points; then, clustering is performed in this new feature space;

[0058] Step 4.1.2), local coordinate system training;

[0059] Each sub-image is trained in its own local coordinate system, and the NeRF model of each sub-image is optimized independently, only for the camera and lidar data within the sub-image area;

[0060] Step 4.1.3), transform to the world coordinate system;

[0061] After training, the final reconstruction of each sub-image needs to be transformed into the world coordinate system; this is achieved by using the metric pose of the sub-image and applying the Sim(3) transformation;

[0062] Step 4.1.4), process the boundary area;

[0063] Regions with low surface density are identified in the boundary regions of the sub-images and removed when merging the sub-image clouds to obtain the final reconstruction result.

[0064] Furthermore, the step 5) includes the following steps:

[0065] Step 5.1), select the point cloud located on the path connecting the pixel and the camera center as the sampling point of the neural radiation field NeRF;

[0066] Step 5.2), dynamically construct an octree voxel grid based on the above point cloud;

[0067] Step 5.3), perform point cloud sampling in the area covered by the voxel grid;

[0068] A continuous scene can be represented as a multi-layer perceptron F , shows the position of the camera in three-dimensional space, Indicates the camera direction The three-dimensional Cartesian unit vector of represents the network input (R, G, B), Represents the volume density, then the neural radiation field expression is as follows:

[0069] (12)

[0070] Step 5.4), by giving a key frame and a pixel coordinate [ ,in represents the internal parameter matrix, Represents the camera pose, and the following formula is used to find the distance from the camera optical center to any pixel [ u, v ]The light emitted by

[0071] (13)

[0072] Step 5.5), according to the above definition, for any ray in the world coordinate system r ( t ) = o + td ;in, o is the position of the camera in world coordinates; in the near and far intervals [ t n ,t f ] The radiance of the internal image is dependent on the viewing angle. C ( r ) can be obtained by integrating the volume rendering formula (14), which includes the transmittance on the light T ( t ) , volume density σ t and RGB values:

[0073] (14)

[0074] (15)

[0075] in Indicates from arrive cumulative transparency of

[0076] It is impractical to sample an infinite number of points on a ray, so the NeRF process based on a neural network needs to be implemented in the interval Sampling between N Discrete points ( i ∈[1 ，N ]), and sum the rendering results of each sampling point to replace the integral operation on the light. The summation operation is shown in formula (16):

[0077] (16)

[0078] (17)

[0079] Step 5.6), the rendering process is optimized by gradient descent, using the mean square error between the predicted color and the true color of the corresponding pixel of the sampled point cloud to fit the multilayer perceptron MLP (Multilayer Perceptron). The role of MLP in NeRF is to predict the color and density of each point in the scene by learning the input 3D coordinates and line of sight direction, thereby achieving high-quality 3D scene reconstruction and novel view synthesis. The fitting expression is as follows:

[0080] (19)

[0081] in, represents MLP, is the fully sampled radiance, Sample radiance for discrete points;

[0082] Step 5.7) The above sampling results are used to train NeRF using the Instant-ngp algorithm to obtain the final reconstructed map of the indoor office scene.

[0083] Furthermore, the step 5.2) includes the following steps:

[0084] Step 5.2.1), in the world coordinate system, dynamically allocate voxel grids for each sub-map according to the three-dimensional information provided by the point cloud, dynamically determine the grid size according to the spatial range of the point cloud data, and construct a three-dimensional grid framework composed of evenly distributed cubic units;

[0085] Step 5.2.2), by traversing each point in the point cloud, mapping them to the nearest voxel unit according to their spatial position;

[0086] Step 5.2.3), as the multimodal SLAM front-end pose estimation module continues to run, the point cloud will continue to expand and the voxel grid will also expand dynamically; during the dynamic expansion process, the leaf nodes of the unobserved scene area are set to empty, and when new point cloud data is added, new voxels are allocated for points that do not fall into existing voxels; in subsequent sampling, only the empty voxels need to be skipped, and the occupied voxels need to be queried and sampled in them.

[0087] Furthermore, the step 5.5) includes the following steps:

[0088] Step 5.5.1), transform the point cloud from the world coordinate system to the camera coordinate system;

[0089] Step 5.5.2) For rays that do not intersect the point cloud within the threshold range, the coarse-to-fine two-stage sampling strategy in NeRF is adopted; in the coarse sampling stage, uniform sampling is performed along the rays; in the fine sampling stage, the weights obtained in the coarse sampling stage are used. To guide sampling, it tends to choose a position closer to the potential surface; the sampling in the coarse sampling stage is done by dividing the light into small segments of equal length and uniformly sampling in each small segment; the fine sampling stage uses the same network as the coarse stage and combines the weights accumulated in the coarse sampling stage to guide sampling; the expression is:

[0090] (18)

[0091] in, Indicates i The weight of each sampling point; Indicates i The cumulative value of the weight of each sampling point; exp represents an exponential function, which is used to calculate the growth of the weight; Indicates i The volume density of the sampling points; Represents the transmittance attenuation due to volume density.

[0092] In summary, the NeRF-based vision-dominated multimodal SLAM method for indoor office environments proposed in this application has the following advantages and beneficial effects:

[0093] 1. Based on the existing multi-sensor fusion SLAM system, this application integrates the neural radiation field based on dense point cloud, while combining the advantages of NeRF volume rendering and the fast convergence and easy editing of explicit scene expression, improving the texture and material continuity of the map.

[0094] 2. Use SLAM pose as pose prior and COLMAP to improve the trajectory to jointly optimize the scene representation and camera pose. The depth and color in independent views can be rendered and compared with the real image to optimize the accuracy of camera pose estimation.

[0095] 3. Incorporating lidar measurements into NeRF optimization, that is, applying lidar-based depth regularization by adding a depth loss, can provide accurate measurements in low-texture areas and with limited viewing constraints, and improve the effect of NeRF 3D reconstruction when the surface has uniform texture and limited multi-view constraints.

[0096] 4. This application introduces a motion mask generation method to filter out invalid sampling light, thereby effectively integrating the optical flow mask and the semantic mask, and correspondingly improving the accuracy of the motion mask and the robustness of the SLAM system in a dynamic environment.

[0097] 5. The sampling strategy of sub-mapping and dynamic octree voxel grid is used to accelerate the convergence of NeRF. Combined with the multi-resolution hash coding of Instant-NGP and the NeRF implementation of CUDA framework, the training speed of NeRF can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The present application is now further described in conjunction with the following drawings;

[0099] Figure 1 is a flowchart of the NeRF-based vision-dominated multimodal SLAM method for indoor office environments;

[0100] Figure 2 This is a schematic diagram of the NeRF principle;

[0101] Figure 3 is a schematic diagram of an octree voxel grid;

[0102] Figure 4 It is the network structure diagram of this application; DETAILED DESCRIPTION

[0103] In order to further explain the technical means adopted by the present application to achieve the predetermined design objectives, the following preferred implementation scheme is proposed in conjunction with the accompanying drawings.

[0104] In the following description, specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0105] like Figures 1 to 4The NeRF-based vision-dominated multimodal SLAM method for indoor office environments is shown in the figure. The neural radiance field NeRF is integrated into the multi-sensor SLAM system, the sub-map is combined with the octree voxel grid, and the dense point cloud is used to provide a three-dimensional geometric prior for NeRF training. The voxel grid is dynamically allocated according to the three-dimensional information provided by the point cloud to guide NeRF sampling, and a depth loss is added to apply a lidar-based depth regularization and obtain the warp mask corresponding to the dynamic object through the optical flow warp set to filter out invalid sampling rays and remove the sampling pixels in the dynamic target area.

[0106] Specifically, the method includes the following implementation steps:

[0107] Step 1) The point cloud measurements of each input LiDAR scan are registered by the laser-inertial odometry (LIO) subsystem to construct the geometry of the radiation map;

[0108] Step 1.1), for the input LiDAR scan point cloud data, use IMU back propagation to compensate for intra-frame motion, and use motion compensation and dedistortion to pre-process the point cloud data input by the LiDAR;

[0109] Step 1.2) Use the preprocessed point cloud data to perform a point-to-surface LIO update, use the generalized-iterative closet point method to minimize the distance from each point cloud in the scan to the plane fitted to the corresponding point in the map, iteratively estimate the LiDAR pose, and then use the estimated state estimate to append the new point to the map;

[0110] Step 1.3), use lidar sensing to perform geometric measurements on textureless objects, and apply lidar-based depth regularization by adding a depth loss to provide sufficient depth information for the reconstruction of the radiation map; the depth loss added to the lidar is defined as the lidar depth measurement D Normal distribution of surroundings and light distribution rendered in NeRF model h(t) The difference between them is expressed as follows:

[0111] (1)

[0112] in, To add depth loss to the LiDAR; Depth measurement for LiDAR D Normal distribution around; is the standard deviation of the distribution; h(t) For light distribution.

[0113] Step 1.4), as the radiation map is continuously updated, the state estimation value of the point cloud data is retrieved in reverse order to continuously optimize the point cloud input;

[0114] Step 2) Restore the brightness information of the map through the camera-inertial navigation (VIO) subsystem;

[0115] While the LIO subsystem reconstructs the geometry of the environment, the VIO subsystem recovers the radiometric information from the input color image;

[0116] The VIO subsystem projects a certain number of points (i.e., tracking points) from the global map to the current image, and then iteratively estimates the camera pose by minimizing the radiometric error of these points; the inter-frame VIO update estimates the system state by minimizing the inter-frame PnP reprojection error, and the frame-to-map VIO update minimizes the brightness error between the map points and the current image;

[0117] Step 2.1), perform photometric calibration on the continuous images input by the camera sensor;

[0118] For each incoming image I, we first correct the image nonlinearity CRF and extinction factor , to obtain the photometrically corrected image Γ , which is at pixel position ρ Place i The expression of the channel is as follows:

[0119] (2)

[0120] The photometrically corrected image Γ is then used for frame-to-frame VIO update, frame-to-map VIO update, and brightness restoration;

[0121] Step 2.2), in frame-to-frame VIO update, use frame-to-frame VIO to update the map points observed in the last frame of optical flow tracking, and obtain a rough estimate of the system state by minimizing the PnP reprojection error of the tracked points;

[0122] Step 2.3), for each input keyframe, calculate the dense optical flow warp set ;

[0123] Use from matching set S The basic matrix of internal line estimation sampled in , setting the matching set S Any matching point within ( O ji , O ki ), using the matrix F Calculate the distance between the corresponding point and its epipolar line, and set the threshold e th , the warp mask calculation expression corresponding to the dynamic object is as follows:

[0124] (3)

[0125] in, is the polar coefficient, is the warp mask size aligned with the current frame image size; j and k is the key frame ID, indicating the Keyframes to The optical flow mask warping process for key frames;

[0126] Step 2.4), in the frame-to-map VIO update, further refine the state estimate by minimizing the difference between the brightness of the map point and the pixel intensity of its projected position in the current image;

[0127] Step 2.5), after the frame-to-map VIO update, use the SLAM pose as the pose prior and use COLMAP to feed back to the frame-to-map VIO update to optimize the camera pose;

[0128] According to the volume rendering method in NeRF, the prediction values ​​are accumulated along the observation ray of the current estimated pose, and the color and depth values ​​are rendered as:

[0129] (4)

[0130] (5)

[0131] in is the weight calculated along the ray, and For sampling point The color and depth values ​​of

[0132] The depth and color in the independent view are rendered and compared with the real image. The expressions are as follows:

[0133] (6)

[0134] (7)

[0135] in, C ( r1 )and D ( r1 ) represent the real image and depth map corresponding to a given posture respectively; M represents the number of sampled pixels in the current image; D(^) var is the depth variance along this ray;

[0136] (8)

[0137] Adding Depth Variance To reduce the weight of depth outliers, a motion mask is introduced to improve the robustness of the present application in dynamic environments. That is, the radiosity of the map points is updated according to the current image measurement through the convergence state estimation and the original input image.

[0138] The construction of complex, dynamic and changeable indoor office environment maps often encounters a lot of noise and often shows a lack of global consistency. To solve this problem, this application introduces edge warping loss into the multimodal SLAM system to enhance the geometric consistency of data association between adjacent frames, including:

[0139] Step 2.5.1), for i 2D pixels in a frame p , reprojecting it to j Frame, the expression is as follows:

[0140] (9)

[0141] in, K and Denote the intrinsic matrix and coordinate system respectively j and coordinate system i The transformation matrix between yes The homogeneous coordinates of ; Represents a transformation function, which is used to transform a pixel from one perspective to another; Indicates that from j Frame to i Camera motion parameters of the frame; Represents pixel Depth information;

[0142] The Canny edge detection algorithm is used to extract edges to calculate the distance map DT to describe the projection error of the nearest edge, effectively identifying the edges in the image, which is defined as the edge set ;

[0143] Step 2.5.2), frame alignment i The edge set in , and marginal loss Define it, the expression is as follows:

[0144] (10)

[0145] in, Representation frame j DT mapping in, is a Huber weight function to reduce the impact of large residuals. In addition, if the projection distance error is greater than, One potential outlier will be removed;

[0146] Step 2.5.3), the pose optimization problem is expressed as follows:

[0147] (11)

[0148] in, Indicates that from j Frame to i Optimal camera motion parameters for the frame; represents the weight of the regularization term;

[0149] Step 3), the LIO and VIO subsystems are coupled in the non-manifold error state iterative Kalman filter framework (ESIKF) to update the radiation point cloud map of the entire indoor working scene;

[0150] ESIKF allows iterative updates of the error state on the manifold, which enables the state estimate to more accurately reflect the sensor measurements; the lidar and camera visual measurements are fused to the same system state at their respective data reception times; when no new sensor data arrives, the state propagation through the IMU predicts the new state of the system;

[0151] Step 4), the radiation point cloud map of the entire indoor working scene is reconstructed through NeRF volume rendering, and the radiation map is divided into several dense point cloud sub-maps;

[0152] The specific steps include:

[0153] Step 4.1), using the sub-mapping strategy to divide the point cloud map of the entire indoor work scene into several sub-maps;

[0154] Step 4.1.1), trajectory segmentation;

[0155] First, the SLAM trajectory optimized by COLMAP is segmented into multiple clusters using the spectral clustering method, and the eigenvectors of the graph Laplacian matrix are used to represent the data points; then, clustering is performed in this new feature space;

[0156] Specifically, scene representation based on Instant-NGP is several orders of magnitude faster than multi-layer perception. This approach allows the system to divide large-scale scenes into local sub-graphs, each of which corresponds to a region in the scene.

[0157] Step 4.1.2), local coordinate system training;

[0158] Each sub-image is trained in its own local coordinate system, and the NeRF model of each sub-image is optimized independently, only for the camera and lidar data within the sub-image area;

[0159] Step 4.1.3), transform to the world coordinate system;

[0160] After training, the final reconstruction of each sub-image needs to be transformed into the world coordinate system; this is achieved by using the metric pose of the sub-image (obtained from the SLAM system) and applying the Sim(3) transformation (including rotation, translation and scaling);

[0161] The submap division is to divide the LiDAR SLAM trajectory into multiple submaps using a spectral clustering algorithm, and the submaps are trained in a local coordinate system;

[0162] The training in the local coordinate system is that each sub-map independently trains the NeRF model in its local coordinate system, that is, the NeRF model of each sub-map is optimized based on its viewpoint and sensor data in the local coordinate system;

[0163] The Sim(3) transformation is to use the metric scale and posture information obtained from the LiDAR SLAM system after training to transform the model in the local coordinate system of each submap to the world coordinate system through the Sim(3) transformation (including rotation, translation and scaling) to ensure that the position and orientation of all submaps in the global coordinate system are consistent;

[0164] The merged submaps are merged to construct a global 3D map of the entire scene by merging all the submaps transformed by Sim(3). During the merging process, low surface density areas in the boundary areas are identified and removed to reduce seams and artifacts between submaps.

[0165] Step 4.1.4), process the boundary area;

[0166] In the border areas of sub-images, reconstruction artifacts may appear due to the limited viewing angle. To solve this problem, the system identifies areas with low surface density and removes them when merging the sub-image clouds to obtain the final reconstruction result;

[0167] The identification of low surface density areas is that when merging the sub-map point clouds to generate the final reconstruction, the system will identify those areas with low surface density. These areas are usually the boundary areas between sub-maps because they are observed less frequently and have limited viewing constraints.

[0168] The purpose of removing low surface density areas is to reduce boundary artifacts, and the system will remove these low surface density areas during the merging process, thereby avoiding inaccurate or incoherent parts in the final 3D reconstruction due to insufficient viewing angle or sparse data.

[0169] The use of a globally consistent LiDAR SLAM trajectory ensures that the submaps have the correct spatial relationship and alignment when they are merged. This helps reduce boundary artifacts caused by inaccurate trajectories.

[0170] The training in the local coordinate system and the transformation in the global coordinate system, the submaps are trained independently in the local coordinate system, and then they are transformed to the global world coordinate system using the Sim (3) transformation. This approach allows fine reconstruction in local areas while ensuring overall consistency through global transformation.

[0171] The optimization of surface smoothness is that in addition to using depth loss to constrain the geometry during the optimization process, surface normal loss is also used to further constrain the smoothness of the reconstructed geometry. This helps to reduce wavy artifacts that appear in expected smooth areas (such as planes).

[0172] The reasonable division of the sub-maps is that when the SLAM trajectory is divided into sub-maps by the spectral clustering algorithm, the number and size of clusters are reasonably selected to reduce the boundaries between the sub-maps, thereby reducing the generation of boundary artifacts.

[0173] Step 5), as the submaps are continuously updated, an octree voxel grid is dynamically allocated for each submap to guide NeRF to sample the light and render the entire scene;

[0174] The specific steps include:

[0175] Step 5.1), select the point cloud located on the path connecting the pixel and the camera center as the sampling point of the neural radiance field (NeRF);

[0176] Step 5.2), dynamically construct an octree voxel grid based on the above point cloud;

[0177] The steps include:

[0178] Step 5.2.1), in the world coordinate system, dynamically allocate voxel grids for each sub-map according to the three-dimensional information provided by the point cloud, dynamically determine the grid size according to the spatial range of the point cloud data, and construct a three-dimensional grid framework composed of evenly distributed cubic units;

[0179] Step 5.2.2), by traversing each point in the point cloud, mapping them to the nearest voxel unit according to their spatial position;

[0180] Step 5.2.3), as the multimodal SLAM front-end pose estimation module continues to run, the point cloud will continue to expand, and the voxel grid will also expand dynamically; during the dynamic expansion process, the leaf nodes of the unobserved scene area are set to empty, and when new point cloud data is added, new voxels are allocated for points that do not fall into existing voxels; in subsequent sampling, it is only necessary to skip the empty voxels, query the occupied voxels, and sample them;

[0181] Step 5.3), perform point cloud sampling in the area covered by the voxel grid;

[0182] A continuous scene can be represented as a multi-layer perceptron F , shows the position of the camera in three-dimensional space,

[0183] Indicates the camera direction The three-dimensional Cartesian unit vector of represents the network input (R, G, B), Represents the volume density, then the neural radiation field expression is as follows:

[0184] (12)

[0185] Step 5.4), by giving a key frame and a pixel coordinate [ ,in represents the internal parameter matrix, Represents the camera pose, and the following formula is used to find the distance from the camera optical center to any pixel [ u, v ]The light emitted by

[0186] (13)

[0187] Step 5.5), according to the above definition, for any ray in the world coordinate system r ( t ) = o + td ;in, o is the position of the camera in world coordinates; in the near and far intervals [ t n ,t f ] The radiance of the internal image is dependent on the viewing angle. C ( r ) can be obtained by integrating the volume rendering formula (14), which includes the transmittance on the light T ( t ) , volume density σ t and RGB values:

[0188] (14)

[0189] (15)

[0190] in Indicates from arrive cumulative transparency.

[0191] It is impractical to sample an infinite number of points on a ray, so the NeRF process based on a neural network needs to be implemented in the interval Sampling between N Discrete points ( i∈[1, N ]), and sum the rendering results of each sampling point to replace the integral operation on the light. The summation operation is shown in formula (16):

[0192] (16)

[0193] (17)

[0194] The following steps are involved:

[0195] Step 5.5.1), transform the point cloud from the world coordinate system to the camera coordinate system;

[0196] Due to the error in the estimated camera pose in the visual odometry and the noise in the point cloud obtained by monocular, stereo triangulation or depth camera, these factors may cause the point cloud to be unable to accurately align with the projection light. To solve this problem, a threshold is set. For each pixel, along the ray starting from the camera optical center, at a radius of the threshold Search point cloud within the range.

[0197] During the search process, skip voxels not occupied by point clouds, and continue to sample enough points to render the ray only in the voxels where the point clouds are located. Since the scene surface is more likely to be around the point cloud, skip voxels not occupied by point clouds during the search process, and continue to sample enough points to render the ray only in the voxels where the point clouds are located;

[0198] Step 5.5.2), for rays that do not intersect the point cloud within the threshold range, the coarse-to-fine two-stage sampling strategy in NeRF is adopted; including,

[0199] In the coarse sampling stage, uniform sampling is performed along the ray;

[0200] In the fine sampling stage, the weights obtained in the coarse sampling stage are used To guide sampling, it tends to choose a position closer to the potential surface; the sampling in the coarse sampling stage is done by dividing the light into small segments of equal length and uniformly sampling in each small segment; the fine sampling stage uses the same network as the coarse stage and combines the weights accumulated in the coarse sampling stage to guide sampling; the expression is:

[0201] (18)

[0202] in, Represents the weight of the i-th sampling point; represents the cumulative value of the weight of the i-th sampling point; exp represents the exponential function, which is used to calculate the growth of the weight; represents the volume density of the i-th sampling point; represents the transmittance attenuation due to volume density;

[0203] Step 5.6), the rendering process is optimized by gradient descent, using the mean square error between the predicted color and the true color of the corresponding pixel of the sampled point cloud to fit the multilayer perceptron MLP (Multilayer Perceptron). The role of MLP in NeRF is to predict the color and density of each point in the scene by learning the input 3D coordinates and line of sight, thereby achieving high-quality 3D scene reconstruction and novel view synthesis. (Network structure as shown Figure 4 The fitting expression is as follows:

[0204] (19)

[0205] in, represents MLP, is the fully sampled radiance, Sample radiance for discrete points.

[0206] Step 5.7) These sampling results will be used in the Instant-ngp algorithm to train NeRF and obtain the final high-quality reconstruction map of the indoor office scene. Instant-ngp introduces multi-resolution hash coding and parallel computing architecture (Compute Unified Device Architecture, CUDA), which effectively improves the efficiency of NeRF training and alleviates the problem of slow NeRF training speed.

[0207] As described above, similar technical solutions can be derived from the solutions given in the drawings and descriptions. However, any solution content that does not deviate from the structure of the present invention still belongs to the scope of rights of the technical solution of this application.

Claims

1. A NeRF-based vision-led multimodal SLAM method for indoor office environments, characterized by: The following steps are included: Step 1) Register the point cloud measurements of each input LiDAR scan through LIO to construct the geometry of the radiation map; Step 1.1), for the input LiDAR scan point cloud data, use IMU back propagation to compensate for intra-frame motion, and use motion compensation and dedistortion to pre-process the point cloud data input by the LiDAR; Step 1.2) Use the preprocessed point cloud data to perform a point-to-surface LIO update, use the generalized iterative closed point GICP method to minimize the distance from each point cloud in the scan to the plane fitted to the corresponding point in the map, iteratively estimate the LiDAR pose, and then use the estimated state estimate to append new points to the map; Step 1.3), use lidar sensing to perform geometric measurements on textureless objects, and apply lidar-based depth regularization by adding a depth loss; the depth loss added to the lidar is defined as the divergence between the normal distribution around the lidar depth measurement D and the light distribution h(t) rendered in the NeRF model, expressed as follows: Among them, L Depth1 Added depth loss from LiDAR; is the normal distribution around the lidar depth measurement D; is the standard deviation of the distribution; h(t) is the light distribution; Step 1.4), as the radiation map is continuously updated, the state estimation value of the point cloud data is retrieved in reverse order to continuously optimize the point cloud input; Step 2), restore the brightness information of the map through VIO; While LIO reconstructs the geometry of the environment, VIO recovers the radiometric information from the input color image. VIO projects from the global map to the current image and then iteratively estimates the camera pose using the projected radiometric error. Inter-frame VIO updates estimate the system state by minimizing the inter-frame PnP reprojection error, while frame-to-map VIO updates minimize the brightness error between map points and the current image. Step 3), LIO and VIO are coupled in the non-manifold error state iterative Kalman filter framework ESIKF to update the radiation point cloud map of the entire indoor working scene; Step 4), reconstructing the radiation point cloud map of the entire indoor working scene through NeRF volume rendering, and dividing the radiation map into several dense point cloud sub-maps; Step 5), as the submaps are continuously updated, an octree voxel grid is dynamically allocated for each submap to guide NeRF to sample the light and render the entire scene.

2. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 1, characterized in that: The step 2) comprises the following steps, Step 2.1), performing photometric calibration on the continuous images input by the camera sensor; For each incoming image I, we first correct the image nonlinearity CRF f i (·) and extinction factor V(·) to obtain the photometrically corrected image Γ, whose expression for the i-th channel at pixel position ρ is as follows: The photometrically corrected image Γ is then used for frame-to-frame VIO update, frame-to-map VIO update, and brightness restoration; Step 2.2), in frame-to-frame VIO update, use frame-to-frame VIO to update the map points observed in the last frame of optical flow tracking, and obtain a rough estimate of the system state by minimizing the PnP reprojection error of the tracked points; Step 2.3), for each input keyframe, calculate the dense optical flow warp set s; Use the internal line estimation basic matrix F sampled from the matching set S, set any matching point (O ji , O ki ), use the matrix F to calculate the distance between the corresponding point and its epipolar line, and set the threshold e th , the warp mask calculation expression corresponding to the dynamic object is as follows: Where A and B are epipolar coefficients, m and n are the warping mask sizes aligned with the current frame image size; j and k are keyframe IDs, representing the optical flow mask warping process from the kth keyframe to the jth keyframe; Step 2.4), in the frame-to-map VIO update, further refine the state estimate by minimizing the difference between the brightness of the map point and the pixel intensity of its projected position in the current image; Step 2.5), after the frame-to-map VIO update, use the SLAM pose as the pose prior and use COLMAP to feed back to the frame-to-map VIO update to optimize the camera pose; According to the volume rendering method in NeRF, the prediction values ​​are accumulated along the observation ray of the current estimated pose, and the color and depth values ​​are rendered as: where w i is the weight calculated along the ray, c i and z i is the sampling point x i The color and depth values ​​of The depth and color in the independent view are rendered and compared with the real image. The expressions are as follows: Where C(r1) and D(r1) represent the real image and depth map corresponding to the given posture respectively; M represents the number of sampled pixels in the current image; is the depth variance along this ray; 3. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 2, characterized in that: Step 2.5) above, add depth variance To reduce the weight of depth outliers, the radiosity of map points is updated according to the current image measurement through the converged state estimate and the original input image; The following steps are included: Step 2.5.1), for the 2D pixel p in the i-th frame, reproject it to the j-th frame, the expression is as follows: Among them, K and T ji denote the intrinsic matrix and the transformation matrix between coordinate system j and coordinate system i, respectively. Yes i The homogeneous coordinates of warp represents the transformation function, which is used to transform a pixel from one perspective to another; ξ ji represents the camera motion parameters from the jth frame to the ith frame; D(p i ) represents pixel p i Depth information; The Canny edge detection algorithm is used to extract edges to calculate the distance map DT to describe the projection error of the nearest edge, effectively identifying the edges in the image, which is defined as the edge set ε i ; Step 2.5.2), for the edge set ε in frame i i , and the marginal loss L edge Define it, the expression is as follows: Among them, D j represents the DT mapping in frame j, ρ is a Huber weight function to reduce the impact of large residuals; if the projection distance error is greater than, δ e One potential outlier will be removed; Step 2.5.3) The pose optimization problem is expressed as follows: Among them, ξ ji * represents the optimal camera motion parameter from the jth frame to the ith frame; λ represents the weight of the regularization term.

4. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 1, characterized in that: The step 4) specifically comprises the following steps: Step 4.1), using the sub-mapping strategy to divide the point cloud map of the entire indoor work scene into several sub-maps; Step 4.1.1), trajectory segmentation; First, the SLAM trajectory optimized by COLMAP is segmented into multiple clusters using the spectral clustering method, and the eigenvectors of the graph Laplacian matrix are used to represent the data points; then, clustering is performed in this new feature space; Step 4.1.2), local coordinate system training; Each sub-image is trained in its own local coordinate system, and the NeRF model of each sub-image is optimized independently, only for the camera and lidar data within the sub-image area; Step 4.1.3), transform to the world coordinate system; After training, the final reconstruction of each sub-image needs to be transformed into the world coordinate system; this is achieved by using the metric pose of the sub-image and applying the Sim(3) transformation; Step 4.1.4), process the boundary area; Regions with low surface density are identified in the boundary regions of the sub-images and removed when merging the sub-image clouds to obtain the final reconstruction result.

5. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 1, characterized in that: The step 5) comprises the following steps, Step 5.1), select the point cloud located on the path connecting the pixel and the camera center as the sampling point of the neural radiation field NeRF; Step 5.2), dynamically construct an octree voxel grid based on the above point cloud; Step 5.3), perform point cloud sampling in the area covered by the voxel grid; A continuous scene can be represented as a multi-layer perceptron F, where x represents the position of the camera in three-dimensional space, d = (dx, dy, dz) represents the three-dimensional Cartesian unit vector of the camera direction (θ, φ), c represents the network input (R, G, B), and σ represents the volume density. The neural radiation field expression is as follows: F(x,d)=(c,σ) (12) Step 5.4), by giving a key frame (K, [R|t]) and a pixel coordinate [u, v], where K represents the intrinsic parameter matrix and [R|t] represents the camera pose, the following formula is used to find the light emitted from the camera optical center to any pixel [u, v]; r(d)=(-R T t+dR T K -1 [u,v,1] T (13) Step 5.5), according to the above definition, for any ray in the world coordinate system, r(t) = o + td; where o is the position of the camera in the world coordinate system; in the interval between the near end and the far end [t n ,t f The radiance C(r) related to the viewing angle in the volume rendering can be obtained by integrating the volume rendering formula (14), which includes the transmittance T(t) on the light, the volume density σ t and RGB values: in Indicates that from t n Cumulative transparency to t; It is impractical to sample an infinite number of points on a ray, so the NeRF process based on a neural network needs to be implemented by sampling in the interval [t n ,t f ] N discrete points are sampled between i (i∈[1, N]), and sum the rendering results of each sampling point to replace the integral operation on the light. The summation operation is shown in formula (16): Step 5.6), the rendering process is optimized by gradient descent, and the mean square error between the predicted color and the true color of the corresponding pixel of the sampling point cloud is minimized to fit the multi-layer perceptron MLP. The fitting expression is as follows: Among them, L c represents MLP, C(r) is the fully sampled radiance, Sample radiance for discrete points; Step 5.7) The above sampling results are used to train NeRF using the Instant-ngp algorithm to obtain the final reconstructed map of the indoor office scene.

6. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 5, characterized in that: The step 5.2) comprises the following steps, Step 5.2.1), in the world coordinate system, dynamically allocate voxel grids for each sub-map according to the three-dimensional information provided by the point cloud, dynamically determine the grid size according to the spatial range of the point cloud data, and construct a three-dimensional grid framework composed of evenly distributed cubic units; Step 5.2.2), by traversing each point in the point cloud, mapping them to the nearest voxel unit according to their spatial position; Step 5.2.3), as the multimodal SLAM front-end pose estimation module continues to run, the point cloud will continue to expand and the voxel grid will also expand dynamically; During the dynamic expansion process, the leaf nodes of the unobserved scene areas are set to empty. When new point cloud data is added, new voxels are allocated for points that do not fall into existing voxels. In subsequent sampling, only the empty voxels need to be skipped, and the occupied voxels are queried and sampled in them.

7. The NeRF-based vision-led multimodal SLAM method for indoor office environments according to claim 5, characterized in that: The step 5.5) comprises the following steps, Step 5.5.1), transform the point cloud from the world coordinate system to the camera coordinate system; Step 5.5.2), for the rays that do not intersect the point cloud within the threshold range, the coarse-to-fine two-stage sampling strategy in NeRF is adopted; This includes uniform sampling along the ray during the coarse sampling stage; In the fine sampling stage, the weight ω obtained in the coarse sampling stage is used i To guide sampling, it tends to choose a position closer to the potential surface; the sampling in the coarse sampling stage is done by dividing the light into small segments of equal length and uniformly sampling in each small segment; The fine sampling stage uses the same network as the coarse stage and combines the weights accumulated in the coarse sampling stage to guide sampling; the expression is: ω i =T i (1-exp(-σ i (t i+1 -t i )) (18) Among them, ω i Represents the weight of the i-th sampling point; T i represents the cumulative value of the weight of the i-th sampling point; exp represents the exponential function, which is used to calculate the growth of the weight; σ i represents the volume density of the i-th sampling point; exp(-σ i (t i+1 -t i )) represents the transmittance attenuation due to volume density.

Images