A high-fidelity neural implicit SLAM method based on dictionary factor representation
By decomposing the scene information into fundamental function factors and coefficient factors and rendering using feature integrals, the problem of rapid growth of model size and insufficient real-time performance in neural implicit SLAM method is solved, and efficient memory usage and real-time performance improvement is achieved, which is suitable for large-scale Three-dimensional reconstruction of scale scenarios.
Patent Information
- Application Number
- CN202410977705.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-22
AI Technical Summary
When the existing neural implicit SLAM method deals with large-scale scenarios, the model size grows rapidly, the memory usage efficiency is low, and the real-time performance is insufficient, making it difficult to meet the needs of high-real-time applications.
The high-fidelity neural implicit SLAM method based on dictionary factor representation is used to decompose the scene information into fundamental function factors and coefficient factors, encode sampling points through tensor grids, and render pixel colors using feature integrals to improve memory usage efficiency and real-time performance.
Achieve higher fidelity detail reconstruction, reduce the correlation between model size and scene map size, improve memory usage efficiency and real-time performance, solve the memory usage problems caused by the expansion of scene scale in traditional methods, and significantly improve color Rendering speed.
Smart Images

Figure CN118521728B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of indoor scene reconstruction and positioning, and in particular relates to a high-fidelity neural implicit SLAM method based on dictionary factor representation. Background Art
[0002] Dense visual SLAM systems are designed to estimate the pose of visual sensors such as RGB-D cameras and build dense three-dimensional maps of unknown environments. Traditional visual SLAM methods usually use multi-view geometry techniques to achieve robust camera tracking and use point clouds to represent scene maps, but this discrete form of scene representation usually requires a lot of memory resources and is also very sensitive to environmental conditions. In recent years, technologies such as neural radiance fields (NeRF) have demonstrated the powerful ability of neural networks in spatial continuous representation, bringing new possibilities for dense visual SLAM. NeRF implicitly represents three-dimensional scenes, making it possible to generate high-quality images from new perspectives. However, existing NeRF technology mainly focuses on static scenes, and its model training time is long, memory usage efficiency is low, and it is difficult to apply in large-scale dynamic scenes.
[0003] Although current neural implicit SLAM methods such as iMAP, NICE-SLAM, and ESLAM have made some progress in scene representation and camera positioning, they still face the problem of rapid growth of model size with the size of the scene map when dealing with large-scale scenes. In addition, these methods also have certain limitations in real-time performance and cannot meet the needs of high real-time applications.
[0004] In order to overcome the above problems, the present invention proposes a neural feature rendering method based on dictionary factor representation. By encoding the geometric and appearance information of the scene into a combination of basis functions and coefficient factors, it not only improves the ability to reconstruct scene details, but also significantly improves the memory utilization efficiency and real-time performance of the model. Summary of the invention
[0005] The purpose of the present invention is to provide a high-fidelity neural implicit SLAM method based on dictionary factor representation.
[0006] The problem to be solved by the present invention is: decomposing the scene information dictionary into basis function factors and coefficient factors, encoding the sampling points into a combination of basis function factors and coefficient factors by means of a tensor grid, and using them to query TSDF and render pixel colors.
[0007] A high-fidelity neural implicit SLAM method based on dictionary factor representation adopts the following technical solution, including the following steps:
[0008] Step S1: Obtain ordered RGB-D image data, obtain multi-view image frame sequences, camera intrinsic parameter data, and scene bounding box information from the Replica and ScanNet datasets;
[0009] Step S2: Create a tracking process and a mapping process, initialize system hyperparameters, and preprocess the data set;
[0010] Step S3: uniformly sampling and surface sampling are performed on the camera rays according to the input depth map, wherein the surface sampling is to sample the surface of the object in the scene according to the depth data of the depth map;
[0011] Step S4: Use different basis functions and coefficient factors to represent the geometry and appearance information of the sampling points respectively, and input the geometric features of the sampling points into the geometry decoder to query their TSDF values;
[0012] Step S5: converting the queried TSDF value into a rendering coefficient, and rendering the depth value of the pixel;
[0013] Step S6: perform feature integration on the appearance features to obtain the color features of the entire camera light, and then input them into the color decoder to query its final pixel rendering color;
[0014] Step S7: by comparing the difference between the rendered RGB-D image and the real image, the color loss, depth loss, free space loss and SDF loss are calculated;
[0015] Step S8: The tracking process copies the factor grid and the decoder copy from the mapping process, initializes the camera pose of the current frame, iteratively updates the camera pose and fixes the network parameters;
[0016] Step S9: The mapping process selects historical key frames that are co-viewed with the current frame according to the key frame selection strategy and participates in joint optimization together with the current frame.
[0017] Furthermore, the step S1 obtains multi-view image frame sequences, camera intrinsic parameter data and scene bounding box information from the Replica and ScanNet datasets, including:
[0018] The image resolution in the Replica dataset is 1200*680, and the image resolution in the ScanNet dataset is 640*480. The scene bounding box is the maximum and minimum three-dimensional coordinates of the scene corresponding to the dataset in the world coordinate system.
[0019] Furthermore, the step S2 initializes system hyperparameters and preprocesses the data set, including:
[0020] The hyperparameters include loss coefficient, number of samples, learning rate, key frame window size, number of iterations, edge cropping of the input image of the dataset, and normalization of all RGB-D frame data.
[0021] Furthermore, in step S3, uniform sampling and surface sampling are performed on camera rays according to the input depth map, including:
[0022] Sample evenly between near and far boundaries point, for rays with valid depth measurements, further in the range Internal uniform sampling near-surface points, where d is the effective depth obtained by depth measurement, is a small offset that, for rays without depth measurements, guides the method to sample the surface by looking up the TSDF value of uniformly sampled sample points.
[0023] Furthermore, in step S4, different basis functions and coefficient factors are used to represent the geometry and appearance information of the sampling points respectively, and the geometric features of the sampling points are input into the geometry decoder to query the TSDF value thereof, including:
[0024] Query sampling points from the geometry basis function grid and geometry coefficient grid by trilinear interpolation The geometric basis function factor and the geometric coefficient factor , multiply the geometric basis function factor of the sampling point by the coefficient factor element by element, and get Geometric features , for the sampling point Appearance characteristics , according to the geometric characteristics The same calculation method is used to obtain the geometric features of the sampling points and input them into the geometric decoder In, decode TSDF value , .
[0025] Furthermore, in step S5, the queried TSDF value is converted into a rendering coefficient, and the depth value of the pixel is rendered, including:
[0026] By formula and Convert the TSDF value to the weight for rendering along the ray, where yes The volume density, yes The volume density, is a hyperparameter, It is the sampling point The weight of the ray to render along, the depth of the ray to render for , where N is the number of sampling points on the ray, It is the sampling point Depth.
[0027] Furthermore, the step S6 inputs the pixel into the color decoder to query the final pixel rendering color, including:
[0028] After obtaining the appearance features and weights of all sampling points along the ray, the overall appearance features of the ray are approximated by weighted summing of the appearance features of the sampling points. , , Ray appearance characteristics Color Decoder Decoded into final rendered pixel color , .
[0029] Furthermore, in step S7, calculating the color loss, depth loss, free space loss and SDF loss includes:
[0030] Color loss is the rendered pixel color and ground truth pixel color Between loss, , the depth loss is the ground truth depth and the pixel rendering depth Between loss, ,in is a set of pixels, yes The camera ray in Indicated in Camera rays with effective depth in ;
[0031] The depth sensor measurement is used to approximate the SDF loss, and the free space loss is used when the sampling point is far away from the truncation area. To enforce consistency between TSDF values and cutoff distances, when the sampling point is in the cutoff region Internal time, that is , using SDF loss to learn the surface geometry within the truncated region, , ,in represents the sampling point on the ray between the camera origin and the surface cutoff region measured by the depth sensor, represents the sampling points within the cutoff region;
[0032] The global loss function used for ,in , , and is the loss coefficient. The same global loss function is used in tracking and mapping, but different loss coefficients are set, and different SDF loss coefficients are used for the sampling points at the center and tail of the truncated area.
[0033] Furthermore, in step S8, the camera pose of the current frame is initialized, the camera pose is iteratively updated, and the network parameters are fixed, including:
[0034] Get a copy of the factor grid and decoder from the mapping process and keep the parameters unchanged, and only optimize the frame Transformation matrix in world coordinates , at the beginning of the tracking process, the constant velocity assumption is applied to initialize the frame Camera pose , ,in is the camera pose of the i-1th frame, is the inverse matrix of the pose matrix of the i-2th frame. For the current frame, randomly select pixels participate in the iterative optimization, during which only the camera pose is updated while the factor grid and decoder remain unchanged.
[0035] Furthermore, in step S9, selecting a historical key frame that is co-viewed with the current frame and participating in joint optimization together with the current frame includes:
[0036] choose Frames participate in joint optimization, including the ones selected from the keyframe list that are co-viewed with the current frame key frames, as well as the last key frame and the current frame, after obtaining this After obtaining the RGB-D information of the frame, randomly select Pixels are iteratively optimized.
[0037] Beneficial effects of the present invention: The proposed high-fidelity neural implicit SLAM method based on dictionary factor representation demonstrates higher-fidelity detail reconstruction and more compact model memory usage in large-scale scenes;
[0038] Introducing an efficient rendering method using feature integration, which solves the problem of insufficient real-time performance in previous neural implicit SLAM methods due to the high MLP query cost in volume rendering;
[0039] The proposed method does not rely on a customized CUDA framework and has higher scalability.
[0040] Through dictionary factor representation, high-fidelity detail reconstruction of the scene is achieved, which can more accurately capture the subtle geometry and appearance structure in the scene, provide a more realistic 3D reconstruction effect, and make the size of the model independent of the size of the scene map. This solves the memory usage problem caused by the expansion of the scene scale in traditional methods, and shows higher memory usage efficiency in large-scale scenes.
[0041] Through feature integral rendering, the color rendering speed is greatly improved, ensuring the system's ability to operate in a real-time environment. Compared with traditional point-by-point query and volume rendering methods, it has obvious advantages in computing efficiency and rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is the overall framework diagram of the present invention;
[0043] Figure 2 This is a comparison chart of the scene reconstruction effect of the present invention under the Replica dataset;
[0044] Figure 3 This is a comparison chart of the scene reconstruction effect of the present invention under the ScanNet dataset;
[0045] Figure 4 This is a comparison diagram of the camera trajectory of the present invention under the ScanNet dataset;
[0046] Figure 5 This is a memory performance test chart of the present invention under the Replica and ScanNet datasets. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described are only used to explain the technical solutions of the present invention and are not limited to the present invention.
[0048] The present invention adopts dictionary factor representation as the scene representation of the neural implicit SLAM framework. Firstly, the scene signal dictionary is decomposed into a linear combination of basis function factors and coefficient factors. The encoder encodes the coordinates of the sampling points into basis function factors and coefficient factors respectively. Secondly, in order to reconstruct the fine scene geometry, TSDF is used instead of volume density. Finally, unlike the traditional neural implicit SLAM method, a more efficient rendering method based on feature integration is adopted to improve the real-time performance of the whole system.
[0049] Specifically, during the operation of the system, points in the canonical space are sampled along rays, and the basis function factors and coefficient factors of the sampling points are queried in the basis function factor grid and the coefficient factor grid according to the coordinates of the sampling points using trilinear interpolation. The inner product is used as the geometric features of the sampling points. The geometric MLP decodes the geometric features of the sampling points into the TSDF of the sampling points. A similar method is also used to query the color features of the sampling points. Finally, the color features of the sampling points along the rays are weighted summed and input into the color MLP to obtain the final pixel rendering color. A large number of experiments are carried out on the Replica dataset and the ScanNet dataset to verify the design choices. Compared with the most advanced methods, the present invention is competitive in running speed, reconstruction quality, memory utilization and positioning accuracy.
[0050] A high-fidelity neural implicit SLAM method based on dictionary factor representation adopts the following technical solution, including the following steps:
[0051] Step S1: Obtain ordered RGB-D image data, obtain multi-view image frame sequences, camera intrinsic parameter data, and scene bounding box information from the Replica and ScanNet datasets;
[0052] Step S2: Create a tracking process and a mapping process, initialize system hyperparameters, and preprocess the data set;
[0053] Step S3: uniformly sampling and surface sampling are performed on the camera rays according to the input depth map, wherein the surface sampling is to sample the surface of the object in the scene according to the depth data of the depth map;
[0054] Step S4: Use different basis functions and coefficient factors to represent the geometry and appearance information of the sampling points respectively, and input the geometric features of the sampling points into the geometry decoder to query their TSDF values;
[0055] Step S5: converting the queried TSDF value into a rendering coefficient, and rendering the depth value of the pixel;
[0056] Step S6: perform feature integration on the appearance features to obtain the color features of the entire camera light, and then input them into the color decoder to query its final pixel rendering color;
[0057] Step S7: by comparing the difference between the rendered RGB-D image and the real image, the color loss, depth loss, free space loss and SDF loss are calculated;
[0058] Step S8: The tracking process copies the factor grid and the decoder copy from the mapping process, initializes the camera pose of the current frame, iteratively updates the camera pose and fixes the network parameters;
[0059] Step S9: The mapping process selects historical key frames that are co-viewed with the current frame according to the key frame selection strategy and participates in joint optimization together with the current frame.
[0060] refer to Figure 1 Shown is the overall framework diagram of the present invention.
[0061] Furthermore, the step S1 obtains multi-view image frame sequences, camera intrinsic parameter data and scene bounding box information from the Replica and ScanNet datasets, including:
[0062] The image resolution in the Replica dataset is 1200*680, and the image resolution in the ScanNet dataset is 640*480. The scene bounding box is the maximum and minimum three-dimensional coordinates of the scene corresponding to the dataset in the world coordinate system.
[0063] Furthermore, the step S2 initializes system hyperparameters and preprocesses the data set, including:
[0064] The hyperparameters include loss coefficient, number of samples, learning rate, key frame window size, number of iterations, edge cropping of the input image of the dataset, and normalization of all RGB-D frame data.
[0065] Furthermore, in step S3, uniform sampling and surface sampling are performed on camera rays according to the input depth map, including:
[0066] Sample evenly between near and far boundaries point, for rays with valid depth measurements, further in the range Internal uniform sampling near-surface points, where d is the effective depth obtained by depth measurement, is a small offset that, for rays without depth measurements, guides the method to sample the surface by looking up the TSDF value of uniformly sampled sample points.
[0067] Furthermore, in step S4, different basis functions and coefficient factors are used to represent the geometry and appearance information of the sampling points respectively, and the geometric features of the sampling points are input into the geometry decoder to query the TSDF value thereof, including:
[0068] Query sampling points from the geometry basis function grid and geometry coefficient grid by trilinear interpolation The geometric basis function factor and the geometric coefficient factor , multiply the geometric basis function factor of the sampling point by the coefficient factor element by element, and get Geometric features , for the sampling point Appearance characteristics , according to the geometric characteristics The same calculation method is used to obtain the geometric features of the sampling points and input them into the geometric decoder In, decode TSDF value , .
[0069] Furthermore, in step S5, the queried TSDF value is converted into a rendering coefficient, and the depth value of the pixel is rendered, including:
[0070] By formula and Convert the TSDF value to the weight for rendering along the ray, where yes The volume density, yes The volume density, is a hyperparameter, It is the sampling point The weight of the ray to render along, the depth of the ray to render for , where N is the number of sampling points on the ray, It is the sampling point Depth.
[0071] Furthermore, the step S6 inputs the pixel into the color decoder to query the final pixel rendering color, including:
[0072] After obtaining the appearance features and weights of all sampling points along the ray, the overall appearance features of the ray are approximated by weighted summing of the appearance features of the sampling points. , , Ray appearance characteristics Color Decoder Decoded into final rendered pixel color , .
[0073] Furthermore, in step S7, calculating the color loss, depth loss, free space loss and SDF loss includes:
[0074] Color loss is the rendered pixel color and ground truth pixel color Between loss, , the depth loss is the ground truth depth and the pixel rendering depth Between loss, ,in is a set of pixels, yes The camera ray in Indicated in Camera rays with effective depth in ;
[0075] The depth sensor measurement is used to approximate the SDF loss, and the free space loss is used when the sampling point is far away from the truncation area. To enforce consistency between TSDF values and cutoff distances, when the sampling point is in the cutoff region Internal time, that is , using SDF loss to learn the surface geometry within the truncated region, , ,in represents the sampling point on the ray between the camera origin and the surface cutoff region measured by the depth sensor, represents the sampling points within the cutoff region;
[0076] The global loss function used for ,in , , and is the loss coefficient. The same global loss function is used in tracking and mapping, but different loss coefficients are set, and different SDF loss coefficients are used for the sampling points at the center and tail of the truncated area.
[0077] Furthermore, in step S8, the camera pose of the current frame is initialized, the camera pose is iteratively updated, and the network parameters are fixed, including:
[0078] Get a copy of the factor grid and decoder from the mapping process and keep the parameters unchanged, and only optimize the frame Transformation matrix in world coordinates , at the beginning of the tracking process, the constant velocity assumption is applied to initialize the frame Camera pose , ,in is the camera pose of the i-1th frame, is the inverse matrix of the pose matrix of the i-2th frame. For the current frame, randomly select pixels participate in the iterative optimization, during which only the camera pose is updated while the factor grid and decoder remain unchanged.
[0079] Furthermore, in step S9, selecting a historical key frame that is co-viewed with the current frame and participating in joint optimization together with the current frame includes:
[0080] choose Frames participate in joint optimization, including the ones selected from the keyframe list that are co-viewed with the current frame key frames, as well as the last key frame and the current frame, after obtaining this After obtaining the RGB-D information of the frame, randomly select Pixels are iteratively optimized.
[0081] method index Room 0 Room 1 Room 2 Office 0 Office 1 Office 2 Office 3 Office 4 average iMAP Accuracy(cm) 5.75 5.44 6.32 7.58 10.25 8.91 6.89 5.34 7.06 Finished(cm) 5.96 5.38 5.21 5.16 5.49 6.04 5.75 6.57 5.69 Completion rate (%) 67.13 68.91 71.69 70.14 73.47 63.94 67.68 62.3 68.16 Depth L1(cm) 5.55 5.47 6.93 7.63 8.13 10.61 9.66 8.44 7.8 NICE-SLAM Accuracy(cm) 1.47 1.21 1.66 1.28 1.02 1.54 1.95 1.6 1.47 Finished(cm) 1.51 1.27 1.65 1.74 1.04 1.62 2.57 1.67 1.63 Completion rate (%) 98.16 98.71 96.42 95.98 98.83 97.19 92.05 97.34 96.83 Depth L1(cm) 3.38 3.03 3.76 2.62 2.31 4.12 8.19 2.73 3.77 Vox-Fusion Accuracy(cm) 1.08 1.05 1.21 1.41 0.82 1.31 1.34 1.31 1.19 Finished(cm) 1.07 1.97 1.62 1.58 0.84 1.37 1.37 1.44 1.41 Completion rate (%) 99.46 94.76 96.37 95.8 99.12 98.2 97.55 97.32 97.32 Depth L1(cm) 1.62 10.43 3.06 4.12 2.05 2.85 3.11 4.22 3.93 Co-SLAM Accuracy(cm) 1.11 1.14 1.17 0.99 0.76 1.36 1.44 1.24 1.15 Finished(cm) 1.06 1.37 1.14 0.92 0.78 1.33 1.36 1.16 1.14 Completion rate (%) 99.62 97.49 97.86 99.07 99.25 98.81 98.48 98.96 98.69 Depth L1(cm) 1.54 6.41 3.05 1.66 1.68 2.71 2.55 1.82 2.68 ESLAM Accuracy(cm) 1.07 0.85 0.93 0.85 0.83 1.02 1.21 0.97 0.97 Finished(cm) 1.12 0.88 1.05 0.96 0.81 1.09 1.42 1.05 1.05 Completion rate (%) 99.06 99.64 98.84 98.34 98.85 98.6 96.8 98.6 98.6 Depth L1(cm) 0.97 1.07 1.28 0.86 1.26 1.71 1.43 1.18 1.18 The present invention Accuracy(cm) 0.98 0.76 0.84 0.76 0.59 0.9 1.06 1 0.86 Finished(cm) 0.99 0.78 0.94 0.79 0.66 0.94 1.11 1.07 0.91 Completion rate (%) 99.65 99.76 99.13 99.5 99.41 99.21 98.95 99.23 99.36 Depth L1(cm) 0.84 0.76 1.04 0.68 0.9 1.24 1.07 0.61 0.89
[0082] The table above shows the reconstruction results of 8 synthetic scenes in the Replica dataset, using accuracy, completion, completion rate, and depth L1 as 3D reconstruction evaluation indicators.
[0083] method Room 0 Room 1 Room 2 Office 0 Office 1 Office 2 Office 3 Office 4 average iMAP 3.88 3.01 2.43 2.67 1.07 4.68 4.83 2.48 3.13 NICE-SLAM 1.76 1.97 2.2 1.44 0.92 1.43 2.56 1.55 1.73 VoxFusion 0.73 1.1 1.1 7.4 1.26 1.87 0.93 1.49 1.98 CoSLAM 0.82 2.03 1.34 0.6 0.65 2.02 1.37 0.88 1.21 ESLAM 0.71 0.7 0.52 0.57 0.55 0.58 0.72 0.63 0.63 Point-SLAM 0.61 0.41 0.37 0.38 0.48 0.54 0.69 0.72 0.52 The present invention 0.48 0.37 0.35 0.32 0.24 0.43 0.38 0.35 0.37
[0084] The table above shows the camera positioning results in 8 synthetic scenes of the Replica dataset. The root mean square error (cm) of the absolute trajectory error is used as the average indicator. The lower the indicator, the higher the positioning accuracy.
[0085] Scene Name iMAP NICE-SLAM VoxFusion CoSLAM ESLAM Point-SLAM The present invention 0000 32.2 13.3 11.6 7.9 7.3 10.24 6.8 0059 17.3 12.8 26.3 12.6 8.5 7.81 7.8 0106 12.0 7.8 9.1 9.5 7.5 8.65 7.2 0169 17.4 13.2 32.3 6.6 6.5 22.16 5.6 0181 27.9 13.9 22.1 12.9 9.0 14.77 10.3 0207 12.7 6.2 7.4 7.1 5.7 9.54 4.2 average 19.42 11.2 18.13 9.43 7.4 12.19 6.98
[0086] The table above shows the camera positioning results in 6 real scenes of the ScanNet dataset. The root mean square error (cm) of the absolute trajectory error is used as the average indicator. The lower the indicator, the higher the positioning accuracy.
[0087] method Tracking process iteration time (ms) Time taken for one iteration of the map building process (ms) Frames per second (Hz) iMAP 34.63 20.15 0.18 NICE-SLAM 7.48 30.59 0.71 Vox-Fusion 11.2 52.7 1.67 Co-SLAM 6.15 14.33 10.5 ESLAM 7.11 20.32 5.6 Point-SLAM 12.23 35.21 0.29 The present invention 6.47 10.14 9.8
[0088] The table above shows the running speed analysis of the Replica dataset Room 0 scene.
[0089] The present invention provides a high-fidelity neural implicit SLAM method based on dictionary factor representation. Compared with the neural implicit SLAM method that directly encodes scene information as features, the method that uses dictionary factors for scene representation shows excellent scene detail reconstruction capability and more efficient memory usage. At the same time, the model size is insensitive to the size of the scene map and is more suitable for large-scale scenes. In addition, feature integral rendering is used to accelerate the color rendering speed, which greatly improves the color rendering speed while ensuring that the color rendering quality is not affected, which further enhances the real-time performance of the method. A large number of experiments on virtual and real-world datasets show that the method is competitive with the existing most advanced neural implicit SLAM methods in terms of real-time performance, positioning accuracy and scene reconstruction quality.
[0090] The above only expresses the preferred implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, several modifications, improvements and substitutions can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.
Claims
1. A high-fidelity neural implicit SLAM method based on dictionary factor representation, characterized in that: The following steps are involved: Step S1: Obtain ordered RGB-D image data, obtain multi-view image frame sequences, camera intrinsic parameter data, and scene bounding box information from the Replica and ScanNet datasets; Step S2: Create a tracking process and a mapping process, initialize system hyperparameters, and preprocess the data set. The hyperparameters include loss coefficient, number of samples, learning rate, key frame window size, number of iterations, edge cropping of the input image of the data set, and normalization of all RGB-D frame data; Step S3: Perform uniform sampling and surface sampling on the camera rays according to the input depth map, where surface sampling is to sample the surface of the object in the scene according to the depth data of the depth map, and uniformly sample between the near boundary and the far boundary. point, for rays with valid depth measurements, further in the range Internal uniform sampling near-surface points, where d is the effective depth obtained by depth measurement, is a small offset that, for rays without depth measurements, guides the method to sample the surface by querying the TSDF value of uniformly sampled sample points; Step S4: Use different basis functions and coefficient factors to represent the geometry and appearance information of the sampling points respectively, and input the geometric features of the sampling points into the geometry decoder to query their TSDF values; Step S5: converting the queried TSDF value into a rendering coefficient, and rendering the depth value of the pixel; Step S6: perform feature integration on the appearance features to obtain the color features of the entire camera light, and then input them into the color decoder to query its final pixel rendering color; Step S7: by comparing the difference between the rendered RGB-D image and the real image, the color loss, depth loss, free space loss and SDF loss are calculated; Step S8: The tracking process copies the factor grid and the decoder copy from the mapping process, initializes the camera pose of the current frame, iteratively updates the camera pose and fixes the network parameters; Step S9: The mapping process selects historical key frames that are co-viewed with the current frame according to the key frame selection strategy and participates in joint optimization together with the current frame.
2. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 1, characterized in that The step S1 obtains multi-view image frame sequences, camera intrinsic parameter data and scene bounding box information from the Replica and ScanNet datasets, including: The image resolution in the Replica dataset is 1200*680, and the image resolution in the ScanNet dataset is 640*480. The scene bounding box is the maximum and minimum three-dimensional coordinates of the scene corresponding to the dataset in the world coordinate system.
3. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 1, characterized in that In step S4, different basis functions and coefficient factors are used to represent the geometry and appearance information of the sampling points respectively, and the geometric features of the sampling points are input into the geometry decoder to query the TSDF value thereof, including: Query sampling points from the geometry basis function grid and geometry coefficient grid by trilinear interpolation The geometric basis function factor and the geometric coefficient factor , multiply the geometric basis function factor of the sampling point by the coefficient factor element by element, and get Geometric features , for the sampling point Appearance characteristics , according to the geometric characteristics The same calculation method is used to obtain the geometric features of the sampling points and input them into the geometric decoder In, decode TSDF value , .
4. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 3, characterized in that, The step S5 converts the queried TSDF value into a rendering coefficient and renders the depth value of the pixel, including: By formula and Convert the TSDF value to the weight for rendering along the ray, where yes The volume density, yes The volume density, is a hyperparameter, It is the sampling point The weight of the ray to render along, the depth of the ray to render for , where N is the number of sampling points on the ray, It is the sampling point Depth.
5. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 4, characterized in that, The step S6 inputs the pixel into the color decoder to query the final pixel rendering color, including: After obtaining the appearance features and weights of all sampling points along the ray, the overall appearance features of the ray are approximated by weighted summing of the appearance features of the sampling points. , , Ray appearance characteristics Color Decoder Decoded into final rendered pixel color , .
6. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 3, characterized in that, In step S7, calculating the color loss, depth loss, free space loss and SDF loss includes: Color loss is the rendered pixel color and ground truth pixel color Between loss, , the depth loss is the ground truth depth and the pixel rendering depth Between loss, ,in is a set of pixels, yes The camera ray in Indicated in Camera rays with effective depth in ; The depth sensor measurement is used to approximate the SDF loss, and the free space loss is used when the sampling point is far away from the truncation area. To enforce consistency between TSDF values and cutoff distances, when the sampling point is in the cutoff region Internal time, that is , using SDF loss to learn the surface geometry within the truncated region, , ,in represents the sampling point on the ray between the camera origin and the surface cutoff region measured by the depth sensor, represents the sampling points within the cutoff region; The global loss function used for ,in , , and is the loss coefficient. The same global loss function is used in tracking and mapping, but different loss coefficients are set, and different SDF loss coefficients are used for the sampling points at the center and tail of the truncated area.
7. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 1, characterized in that, In step S8, the camera pose of the current frame is initialized, the camera pose is iteratively updated, and the network parameters are fixed, including: Get a copy of the factor grid and decoder from the mapping process and keep the parameters unchanged, and only optimize the frame Transformation matrix in world coordinates , at the beginning of the tracking process, the constant velocity assumption is applied to initialize the frame Camera pose , ,in is the camera pose of the i-1th frame, is the inverse matrix of the pose matrix of the i-2th frame. For the current frame, randomly select pixels participate in the iterative optimization, during which only the camera pose is updated while the factor grid and decoder remain unchanged.
8. A high-fidelity neural implicit SLAM method based on dictionary factor representation as claimed in claim 1, characterized in that, The step S9 selects a historical key frame that is co-viewed with the current frame and participates in joint optimization together with the current frame, including: choose Frames participate in joint optimization, including the ones selected from the keyframe list that are co-viewed with the current frame key frames, as well as the last key frame and the current frame, after obtaining this After obtaining the RGB-D information of the frame, randomly select Pixels are iteratively optimized.
