Multi-neural-field implicit SLAM method, device and system for indoor environment

Abstract

The invention relates to the technical field of visual simultaneous localization and mapping, and particularly discloses a multi-neural-field implicit SLAM method, device and system for an indoor environment, and the method comprises the steps: obtaining an RGB-D image sequence of the indoor environment; performing multi-neural field initialization processing on the RGB-D image sequence; performing ray sampling in each neural field in the three-dimensional global map; performing feature coding on the three-dimensional map feature vector; performing deep learning and rendering on the three-dimensional feature codes according to a Manhattan self-attention network to obtain a rendered RGB image and a rendered depth image; and performing loss learning according to the RGB-D image sequence of the indoor environment, the rendered RGB image and the rendered depth image to obtain positioning and mapping results of the indoor environment. According to the multi-neural-field implicit SLAM method for the indoor environment, provided by the invention, the mapping precision of the complex indoor environment can be improved, and meanwhile, the mapping robustness of the complex indoor environment can also be improved.

Description

Technical Field

[0001] This invention relates to the field of visual simultaneous localization and mapping technology, and in particular to a multi-neural field implicit SLAM method, a multi-neural field implicit SLAM device, and a multi-neural field implicit SLAM system for indoor environments. Background Technology

[0002] Neural implicit dense visual SLAM (Simultaneous Localization and Mapping) technology enables robots to autonomously determine their position and simultaneously reconstruct 3D structures through a series of algorithms. As a typical task in robotics and computer vision, neural implicit dense visual SLAM is widely used in scenarios such as autonomous driving, AR / VR, and indoor robotics. These systems typically generate high-fidelity 3D maps of the environment, serving important functions such as navigation, object recognition, and semantic segmentation.

[0003] In recent years, SLAM systems based on Neural Radiation Fields (NeRF) have gained popularity due to their simplicity and efficiency. SLAM based on neural rendering frameworks and neural implicit representations are favored for their simplicity and hole-filling capabilities. Several neural implicit SLAM systems exist in the current technology, such as iMAP, which uses a single MLP to represent the entire scene. However, as the scene size increases, a single MLP becomes overwhelmed, and the accuracy of tracking and mapping decreases significantly. Another example is NICE-SLAM, which uses multiple MLPs with hierarchical scene representations to handle large scenes. However, NICE-SLAM's hole-filling capability still has room for improvement, and the computation time for training standard scenes is too long (approximately half an hour). Furthermore, Co-SLAM employs two different encoding methods: parametric encoding and coordinate encoding. The former is considered to produce smoother reconstructions, while the latter utilizes a multi-resolution hash grid, significantly improving training speed.

[0004] While current neural implicit SLAM systems perform exceptionally well in tracking and mapping, real-world environments often contain both simple and complex objects. Existing NeRF-based SLAM systems typically employ single or limited neural fields of fixed radius to represent the entire 3D scene map, a approach that remains limited when dealing with complex environments. This design restricts the precision of object reconstruction and the scalability of the scene. Using neural fields of fixed radius often struggles to balance overly smooth and overly detailed scene rendering, reducing mapping accuracy and wasting training resources. Furthermore, traditional volume rendering formulas are insufficient in representing object surface morphology and complex optical effects, leading to decreased robustness of the system in complex scenes.

[0005] Therefore, how to improve the accuracy and robustness of mapping complex indoor environments has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a multi-neural field implicit SLAM method, a multi-neural field implicit SLAM device, and a multi-neural field implicit SLAM system for indoor environments, solving the problems of low mapping accuracy and poor robustness in complex indoor environments caused by the fixed radius of the neural field in related technologies.

[0007] As a first aspect of the present invention, a multi-neural field implicit SLAM method for indoor environments is provided, comprising:

[0008] Acquire an RGB-D image sequence of an indoor environment, the RGB-D image sequence comprising an image sequence composed of RGB images and depth images acquired according to a preset frame rate;

[0009] The RGB-D image sequence is subjected to multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius;

[0010] Ray sampling is performed within each neural field of the three-dimensional global map to obtain the three-dimensional map feature vector;

[0011] The feature vectors of the three-dimensional map are subjected to feature encoding to obtain three-dimensional feature codes;

[0012] The three-dimensional feature encoding is processed by deep learning and rendering using a Manhattan self-attention network to obtain a rendered RGB image and a rendered depth image.

[0013] Loss learning is performed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image to obtain the localization and mapping results of the indoor environment.

[0014] Furthermore, the RGB-D image sequence undergoes multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields, including:

[0015] Iterate over the RGB-D image sequence;

[0016] When the current frame of the iteration is the first frame of the RGB-D image sequence, the grid with fixed side length is divided according to the distribution of the three-dimensional point cloud on the initial three-dimensional map, and an initial neural field with a preset fixed radius is formed at the center of the grid where the three-dimensional point cloud exists.

[0017] When the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined according to the density of the three-dimensional point cloud in the initial three-dimensional map, and a new neural field with multiple adjustable radii is formed.

[0018] Complete the iteration of all RGB-D image sequences to obtain a three-dimensional global map with multiple neural fields.

[0019] Furthermore, when the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined based on the density of the 3D point cloud in the initial 3D map, and a new neural field with multiple adjustable radii is formed, including:

[0020] Find the set of 3D point clouds that are not included in the current initial neural field;

[0021] The adjustable radius of the new neural field is determined based on the density of the three-dimensional point cloud set that is not included in the current initial neural field.

[0022] The adjustable side length of the mesh is determined based on the density of the three-dimensional point cloud set that is not included in the current initial neural field.

[0023] The location of the new neural field is determined based on the center of the grid.

[0024] Furthermore, ray sampling is performed within each neural field of the three-dimensional global map to obtain a three-dimensional map feature vector, including:

[0025] According to NeRF rays, rays are emitted within each nerve field;

[0026] The ray is sampled at intervals of a first preset distance to obtain ray sampling points;

[0027] Depth-guided sampling is performed at intervals of a second preset distance to obtain depth-guided sampling points;

[0028] The 3D map feature vector is obtained based on the ray sampling points and the depth-guided sampling points.

[0029] Further, feature encoding is performed on the 3D map feature vector to obtain 3D feature encoding, including:

[0030] The feature vectors of the three-dimensional map are encoded using hexahedral grid coding to obtain the color and geometric features of the three-dimensional feature code.

[0031] The feature vector of the three-dimensional map is encoded using Gaussian directional coding to obtain the positional features of the three-dimensional feature code.

[0032] Further, deep learning and rendering are performed on the 3D feature encoding based on the Manhattan self-attention network to obtain the rendered RGB image and the rendered depth map, including:

[0033] The three-dimensional feature stitching result is obtained by stitching together the color and geometric features and the positional features of the three-dimensional feature encoding.

[0034] The Manhattan attention network is used to render the 3D feature stitching result using depth, color, and a signed distance function to obtain the rendered RGB image and the rendered depth image.

[0035] Furthermore, the 3D feature stitching result is rendered using depth, color, and a signed distance function based on the Manhattan attention network to obtain a rendered RGB image and a rendered depth map, including:

[0036] The overlapping space attenuation is applied to the 3D feature stitching result using the Manhattan self-attention network to obtain the 3D feature stitching result after overlapping space attenuation;

[0037] Local enhancement is performed on the 3D feature stitching result after overlapping spatial attenuation based on the local enhancement network to obtain the enhanced query vector, key vector and value vector;

[0038] The Manhattan self-attention is obtained by decomposing the enhanced query vector, key vector, and value vector.

[0039] The depth, color, and signed distance are rendered based on the decomposed Manhattan self-attention, resulting in a rendered RGB image and a rendered depth image.

[0040] Furthermore, loss learning is performed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment, including:

[0041] The color loss function and the depth loss function are constructed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image.

[0042] Determine the free space loss function and the truncation loss function;

[0043] Construct the total mapping loss function based on the color loss function, depth loss function, free space loss function, and truncation loss function;

[0044] Construct a tracking loss function based on the motion model;

[0045] The indoor environment location and mapping results are obtained by minimizing the total mapping loss function and the tracking loss function.

[0046] As another aspect of the present invention, a multi-neural field implicit SLAM device for indoor environments is provided to implement the multi-neural field implicit SLAM method for indoor environments described above, wherein the device includes:

[0047] The acquisition module is used to acquire an RGB-D image sequence of an indoor environment, wherein the RGB-D image sequence includes an image sequence composed of RGB images and depth images acquired according to a preset frame rate;

[0048] An initialization module is used to perform multi-neural field initialization processing on the RGB-D image sequence to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius.

[0049] The ray sampling module is used to perform ray sampling in each neural field of the three-dimensional global map to obtain the feature vector of the three-dimensional map.

[0050] The feature encoding module is used to encode the feature vectors of the three-dimensional map to obtain three-dimensional feature codes;

[0051] The deep learning and rendering module is used to perform deep learning and rendering on the three-dimensional feature encoding based on the Manhattan self-attention network to obtain the rendered RGB image and the rendered depth image.

[0052] The localization and mapping module is used to perform loss learning based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment.

[0053] As another aspect of the present invention, a multi-neural field implicit SLAM system for indoor environments is provided, comprising: an image acquisition device and the aforementioned multi-neural field implicit SLAM device for indoor environments, wherein the image acquisition device is communicatively connected to the multi-neural field implicit SLAM device for indoor environments.

[0054] The image acquisition device is used to acquire RGB images and depth images of the indoor environment according to a preset frame rate to obtain an RGB-D image sequence of the indoor environment;

[0055] The multi-neural field implicit SLAM device for indoor environments is used to achieve tracking, localization, and mapping based on multi-neural fields and Manhattan self-attention networks according to the RGB-D image sequence of the indoor environment, thereby obtaining the localization and mapping results of the indoor environment.

[0056] This invention provides a multi-neural field implicit SLAM method for indoor environments. It acquires an RGB-D image sequence of the indoor environment, performs multi-neural field initialization processing on this sequence, performs ray sampling within each neural field, and then encodes the 3D map features after ray sampling. Based on a Manhattan self-attention network, it performs deep learning and rendering of the 3D feature encoding, and finally applies loss school to obtain the localization and mapping results of the indoor environment. This multi-neural field implicit SLAM method for indoor environments effectively balances memory resources and mapping accuracy through adaptive multi-neural fields, obtaining high-quality mapping results. The rendering method based on ray sampling and Manhattan self-attention networks enhances the robustness and scalability of mapping large indoor environments, thereby effectively improving the performance of high-fidelity reconstruction while maintaining mapping accuracy. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0058] Figure 1 The flowchart of the multi-neural field implicit SLAM method for indoor environments provided by the present invention is shown.

[0059] Figure 2 This is a flowchart for multi-neural field initialization processing of RGB-D image sequences provided by the present invention.

[0060] Figure 3 A schematic diagram illustrating the formation process of the novel neural field provided by this invention.

[0061] Figure 4 The flowchart provided by the present invention describes the formation of a new neural field with multiple adjustable radii.

[0062] Figure 5 This is a flowchart of ray sampling within each neural field provided by the present invention.

[0063] Figure 6 This is a flowchart for feature encoding of 3D map feature vectors provided by the present invention.

[0064] Figure 7 This is a flowchart of deep learning and rendering of three-dimensional feature encoding provided by the present invention.

[0065] Figure 8 The sampling point encoding and Manhattan Transformer process provided for this invention are illustrated in the diagram.

[0066] Figure 9This is a flowchart illustrating the rendering of 3D feature stitching results using depth, color, and a signed distance function, as provided by the present invention.

[0067] Figure 10 The flowchart for loss learning provided by this invention.

[0068] Figure 11 The flowchart illustrates the specific implementation of the multi-neural field implicit SLAM method for indoor environments provided by this invention.

[0069] Figure 12 The structural block diagram of the multi-neural field implicit SLAM device for indoor environments provided by the present invention.

[0070] Figure 13 The structural block diagram of the multi-neural field implicit SLAM system for indoor environments provided by the present invention. Detailed Implementation

[0071] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0072] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0073] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] This embodiment provides a multi-neural field implicit SLAM method for indoor environments. Figure 1 This is a flowchart of a multi-neural field implicit SLAM method for indoor environments provided according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0075] S100. Acquire an RGB-D image sequence of the indoor environment, wherein the RGB-D image sequence comprises an image sequence composed of RGB images and depth images acquired according to a preset frame rate;

[0076] In this embodiment of the invention, an RGB-D camera can be used to acquire RGB images and depth images at a frame rate of 30Hz in an indoor environment. The sequence of these images is called an RGB-D image sequence.

[0077] S200. Perform multi-neural field initialization processing on the RGB-D image sequence to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius.

[0078] In this embodiment of the invention, the RGB-D image sequence is iterated and subjected to multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields. It should be understood that each neural field can be understood as a spherical region with a preset initial radius.

[0079] S300. Ray sampling is performed in each neural field of the three-dimensional global map to obtain the feature vector of the three-dimensional map.

[0080] In this embodiment of the invention, ray sampling is performed on each neural field in the obtained three-dimensional global map, and the feature vector of the three-dimensional map is obtained based on the ray sampling results.

[0081] S400. Perform feature encoding on the feature vector of the three-dimensional map to obtain three-dimensional feature encoding;

[0082] Specifically, feature encoding is performed on the feature vectors of the 3D map to obtain color features, geometric features, and location features.

[0083] S500: Perform deep learning and rendering on the three-dimensional feature encoding based on the Manhattan self-attention network to obtain the rendered RGB image and the rendered depth image.

[0084] In this embodiment of the invention, the three-dimensional feature encoding is performed using a Manhattan self-attention network for deep learning, and the rendering is performed based on the learning results to obtain a rendered RGB image and a rendered depth image.

[0085] S600 performs loss learning based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment.

[0086] In this embodiment of the invention, an RGB loss function and a depth loss function can be constructed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image. In addition, a free space loss function and a truncation loss function need to be determined. Based on these loss functions, a total loss function is formed, and the indoor environment is located and mapped to obtain the location and mapping results of the indoor environment.

[0087] Therefore, the multi-neural field implicit SLAM method for indoor environments provided by this invention acquires an RGB-D image sequence of the indoor environment, performs multi-neural field initialization processing on the RGB-D image sequence, performs ray sampling within each neural field, and then encodes the 3D map features after ray sampling. Based on a Manhattan self-attention network, deep learning and rendering of the 3D feature encoding are then performed. Finally, loss schooling is applied to obtain the localization and mapping results of the indoor environment. This multi-neural field implicit SLAM method for indoor environments can effectively balance memory resources and mapping accuracy through adaptive multi-neural fields, obtaining high-quality mapping results. The rendering method based on ray sampling and Manhattan self-attention networks enhances the robustness and scalability of mapping large indoor environments, thereby effectively improving the performance of high-fidelity reconstruction while maintaining mapping accuracy.

[0088] In this embodiment of the invention, the RGB-D image sequence is subjected to multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields, such as... Figure 2 As shown, it includes:

[0089] S210, Iterate the RGB-D image sequence;

[0090] Specifically, based on RGB-D image sequences That is, the color image and the depth image are iterated to perform multi-neural field initialization, forming a... Centered on, with radius as neural field .

[0091] S220. When the current frame of the iteration is the first frame of the RGB-D image sequence, divide the grid with fixed side length according to the distribution of the three-dimensional point cloud on the initial three-dimensional map, and form an initial neural field with a preset fixed radius at the center of the grid where the three-dimensional point cloud exists.

[0092] In this embodiment of the invention, the first frame is used as a keyframe, and thereafter, every five frames, the frame iterates to is used as a keyframe. Furthermore, keyframes are used for both map optimization and new neural field generation; one keyframe can correspond to multiple neural fields.

[0093] Specifically, if the iteration reaches the first frame of the RGB-D image sequence, that frame is designated as a keyframe. Subsequently, every five frames, the frame reached during iteration is designated as a keyframe (i.e., frame 1, frame 6, frame 11, and so on). The initial 3D map is only related to the first frame. Furthermore, keyframes are used for map optimization and new neural field generation; one keyframe can correspond to multiple neural fields. Then, the side length is determined based on the 3D point cloud on the initial 3D map. The mesh is initialized with a fixed radius at the center of the mesh containing the 3D point cloud. neural field ( Initialized to 0.2m), details can be found as follows: Figure 3 As shown.

[0094] S230. When the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined according to the density of the three-dimensional point cloud in the initial three-dimensional map, and a new neural field with multiple adjustable radii is formed.

[0095] In this embodiment of the invention, if the iteration reaches a non-first frame of the RGB-D image sequence, the density of the 3D point cloud in the initial 3D map is first calculated. :

[0096] ,

[0097] in, Represents neural field The number of three-dimensional points in the data. Indicates volume.

[0098] Secondly, the radius of the neural field is dynamically adjusted based on the calculated density of the 3D point cloud. Specifically, the radius is dynamically adjusted according to the calculated density. The radius of the central neural field The dynamic radius adjustment formula is as follows:

[0099] ,

[0100] in, and All represent density thresholds. This represents the attenuation coefficient, which controls the rate at which the radius shrinks as density increases.

[0101] Specifically, when the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined based on the density of the 3D point cloud in the initial 3D map, and a new neural field with multiple adjustable radii is formed, such as... Figure 4 As shown, it includes:

[0102] S231. Find the set of three-dimensional point clouds that are not included in the current initial neural field;

[0103] Specifically, it searches for a set of 3D point clouds initialized in the current RGB-D image sequence frame that is not included in the current neural field. ,For example Figure 3 X shown unc It satisfies the constraints ,in, Indicates from keyframe pose transformation matrix to world coordinates This represents the coordinates of the center of the neural field.

[0104] S232. Determine the adjustable radius of the new neural field based on the density of the three-dimensional point cloud set that is not included in the current initial neural field.

[0105] In this embodiment of the invention, the initial 3D map is discretized into a shape with a side length of... 3D mesh, Initialize to And with the radius of the neural field And change. Design. To ensure complete spatial coverage of the voxel-centered neural field, if a non-empty voxel contains uncovered points but its center is not initialized with a neural field, a new field is initialized at the geometric center of the non-empty voxel. The 3D mesh also enforces a minimum distance between adjacent fields. To maintain sparsity, a random offset is applied to the mesh each time a keyframe is inserted, in order to mitigate the error in pose graph optimization.

[0106] S233. Determine the adjustable side length of the mesh based on the density of the three-dimensional point cloud set that is not included in the current initial neural field.

[0107] In this embodiment of the invention, as the radius of the neural field changes, the side length of the voxel mesh... This will also change accordingly to maintain the appropriate distance between the centers of the new neural fields. Specifically, a multi-resolution grid strategy is adopted, where the point density determines the grid resolution. In high-density regions, both the field radius and the grid edge length decrease. Conversely, in low-density environments, both increase simultaneously.

[0108] S234. Determine the location of the new neural field based on the center of the grid;

[0109] In this embodiment of the invention, the side length of the voxel mesh... The change strategy is as follows:

[0110] ,

[0111] in, and All represent density thresholds. This mapping was learned through offline simulation, ensuring that the new neural field covers more than 90% of the 3D point cloud in both smooth (e.g., walls and sofas) and detailed (e.g., bottles and flowers) scenes.

[0112] Based on the above process, the data structure representation of the neural field set is obtained as follows:

[0113] ,

[0114] in, Represents the set of all neural fields. Indicates the first The neural field of each keyframe K represents the radius of the initial field. Here, the adaptive neural field is keyframe-based, with each neural field associated with a keyframe, establishing a many-to-one relationship. In this embodiment of the invention, K... i This indicates the keyframe to which the neural field belongs. The generation of a neural field means that the computer instantiates a new neural field and stores it in a neural field collection for representation within the 3D map. The corresponding neural field can then be queried and updated within the neural field collection.

[0115] S240. Complete the iteration of all RGB-D image sequences to obtain a three-dimensional global map with multiple neural fields.

[0116] In this embodiment of the invention, a three-dimensional global map with multiple neural fields is obtained by iterating through all RGB-D image sequences, and ray sampling is then performed based on this three-dimensional global map.

[0117] In this embodiment of the invention, ray sampling is performed within each neural field of the three-dimensional global map to obtain a three-dimensional map feature vector, such as... Figure 5 As shown, it includes:

[0118] S310, based on NeRF rays emitting rays within each nerve field;

[0119] Specifically, NeRF rays are used to sample within the neural field, i.e., with the center of the camera... Choose the origin The neural field in the current keyframe, oriented towards the direction the camera is facing. Emit a ray.

[0120] It should be understood that the direction the camera is facing is the direction of the RGB-D image, i.e. the direction in which the photo is taken, and this direction has already been input into the SLAM system. Therefore, the direction the camera is facing is already determined when the image is input.

[0121] S320. Sample the ray at intervals of a first preset distance to obtain ray sampling points;

[0122] Specifically, point At the first preset distance, i.e., the fixed interval Sampling is performed at the location, such as ,in Indicates the number of points sampled by the ray.

[0123] S330. Perform depth-guided sampling at intervals of the second preset distance to obtain depth-guided sampling points;

[0124] Specifically, depth-guided sampling is used at a distance from the object surface. Within this area, a second preset distance, i.e., a fixed interval, is used. Sampling was performed, and samples were collected. One point.

[0125] It's important to note that "object" here refers to the object through which the ray passes. Since the input is a depth map, the object's depth value, i.e., its distance from the camera, is already known. Depth-guided sampling refers to uniformly sampling again at very close intervals to the object's surface, based on the original sampling, to obtain more detailed geometric features of the object; this is also performed within the 3D map.

[0126] S340. Obtain the three-dimensional map feature vector based on the ray sampling points and the depth-guided sampling points.

[0127] Specifically, the total number of sampling points is Assume the set of points sampled by the ray within the grid is Subsequently through x i This represents the input for hexahedral mesh encoding.

[0128] In this embodiment of the invention, feature encoding is performed on the three-dimensional map feature vector to obtain three-dimensional feature encoding, such as... Figure 6 As shown, it includes:

[0129] S410. Perform feature encoding on the feature vector of the three-dimensional map according to the hexahedral grid encoding to obtain the color features and geometric features of the three-dimensional feature encoding;

[0130] S420. Perform feature encoding on the feature vector of the three-dimensional map according to Gaussian direction encoding to obtain the position features of the three-dimensional feature encoding.

[0131] It should be understood that the sampling points are encoded using hexahedral mesh encoding and Gaussian directional encoding. Hexahedral mesh encoding preserves the object's color and geometric features in the mapping process. Compared to traditional hash mesh encoding, hexahedral mesh encoding uses a hexahedral lattice to accelerate the mapping and optimization process, ensuring the accuracy of object reconstruction and providing real-time performance for subsequent self-attention rendering methods. Gaussian directional encoding captures optical features such as reflectivity and object position features, overcoming the inherent reflectivity deficiency of NeRF. Finally, color features are obtained. Geometric features and location features In order to proceed with subsequent rendering.

[0132] In this embodiment of the invention, the three-dimensional feature encoding is subjected to deep learning and rendering based on a Manhattan self-attention network to obtain a rendered RGB image and a rendered depth image, as shown below. Figure 7 As shown, it includes:

[0133] S510. Based on the color and geometric features of the three-dimensional feature encoding and the positional features of the three-dimensional feature encoding, perform feature splicing to obtain the three-dimensional feature splicing result;

[0134] Specifically, the encoding results of all acquisition points in the neural field ( , , The input is processed by MLP for feature concatenation (FC) to obtain the output. ,like Figure 8 As shown.

[0135] S520. Render the 3D feature stitching result using depth, color, and signed distance functions based on the Manhattan attention network to obtain the rendered RGB image and the rendered depth image.

[0136] In this embodiment of the invention, in order to reduce the number of tokens and ensure real-time performance, features are... Overlap space reduction was implemented. Additionally, Manhattan self-attention was introduced, utilizing Manhattan distance to better handle object edges through axial spatial decay.

[0137] Specifically, the 3D feature stitching result is rendered using depth, color, and a signed distance function based on the Manhattan attention network to obtain a rendered RGB image and a rendered depth image, such as... Figure 9 As shown, it includes:

[0138] S521. The three-dimensional feature stitching result is subjected to overlap space attenuation according to the Manhattan self-attention network to obtain the three-dimensional feature stitching result after overlap space attenuation.

[0139] S522. Based on the local enhancement network, perform local enhancement on the 3D feature stitching result after the overlapping space attenuation to obtain the enhanced query vector, key vector and value vector;

[0140] S523. Decompose the enhanced query vector, key vector, and value vector to obtain the decomposed Manhattan self-attention;

[0141] S524. Render the depth, color, and signed distance based on the decomposed Manhattan self-attention to obtain the rendered RGB image and the rendered depth image.

[0142] Specifically, such as Figure 8 As shown, a Local Context Enhancement module is introduced to enhance local expressive power. Subsequently, the obtained query vector... Key vector Sum value vector The matrix is ​​decomposed into horizontal and vertical components. , , and Then, depth is rendered using decomposed Manhattan self-attention. ,color and signed distance function (SDF).

[0143] ,

[0144] ,

[0145] in, , The decomposed spatial decay matrices use Manhattan distance as their decay feature. Experiments show that this decomposition process significantly optimizes rendering time. Furthermore, overlap space reduction significantly reduces the number of self-attention tokens through downsampling without compromising the Manhattan spatial decay feature. Weights Additional output indicating rendering depth. This represents the output features. The Transformer model is applied only when the point density is below a threshold. The neural field can be used to further reduce processing time.

[0146] In this embodiment of the invention, the mapping process optimizes mapping-related parameters, including Manhattan Transformer and hexahedral mesh encoding, by minimizing an objective function. Simultaneously, the tracking process optimizes camera pose by minimizing a tracking loss function. The entire SLAM system is driven by a framework that alternately optimizes the 3D scene map and camera pose.

[0147] Specifically, loss learning is performed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment, such as... Figure 10 As shown, it includes:

[0148] S610. Construct color loss function and depth loss function based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image;

[0149] In this embodiment of the invention, similar to most neural implicit SLAM, it utilizes Three-dimensional points are collected using a sampling ray. The three-dimensional map is optimized by minimizing the color and depth loss functions, as shown in the following equation:

[0150] ,

[0151] in, Indicates the color estimate. This represents the depth estimate. Indicates the true value of the color. This represents the actual depth value.

[0152] S620. Determine the free space loss function and the truncation loss function;

[0153] In this embodiment of the invention, the free space loss function encourages the Transformer model to learn the point located between the camera optical center and the surface truncation region. cutoff distance This restricts the sampling points to areas close to the surface. The expression for the free space loss function is:

[0154] ,

[0155] For cutoff distance For sampling points within the surface, a truncation distance loss is defined to encourage the model to learn how to accurately represent the surface within the truncation region. The expression for the truncation loss function is:

[0156] ,

[0157] in, Indicates that it is located in the ray The set of points within the upper truncated region.

[0158] S630. Construct the total mapping loss function based on the color loss function, depth loss function, free space loss function, and truncation loss function;

[0159] In this embodiment of the invention, the total mapping loss function is obtained based on the color loss function, depth loss function, free space loss function, and truncation loss function. The expression for the total mapping loss function is as follows:

[0160] .

[0161] S640. Construct a tracking loss function based on the motion model;

[0162] In this embodiment of the invention, a constant velocity motion model is used. To initialize camera pose By capturing from the current frame Each pixel is used to optimize the current pose using a tracking loss function; therefore, the expression for the tracking loss function is:

[0163] .

[0164] S650. Minimize the total mapping loss function and the tracking loss function to obtain the location and mapping results of the indoor environment.

[0165] In this embodiment of the invention, the parallel minimization of the mapping loss function and the tracking loss function can optimize the camera pose and the 3D scene map, thus completing the SLAM process.

[0166] The following section combines specific application scenarios and... Figure 11 The flowchart shown illustrates the specific implementation process and experimental results of the multi-neural field implicit SLAM method for indoor environments of the present invention.

[0167] The multi-neural field implicit SLAM method of this invention for indoor environments can be specifically used for real-time localization and high-precision map building of service robots (such as hotel delivery and hospital disinfection robots) in complex indoor environments. Traditional SLAM has poor mapping accuracy in dynamic lighting, specular reflection, or high-texture areas (such as large storage rooms). The adaptive multi-neural field method of this invention dynamically adjusts the neural field radius by calculating the density of 3D points in the environment. For high-texture areas, the neural field radius is reduced to concentrate training resources, significantly improving the mapping accuracy and robustness in such scenarios. For optical effects such as specular reflection, Gaussian directional encoding is used to preserve optical information in the environment, avoiding interference from lighting and reflection in object reconstruction.

[0168] Furthermore, in applications such as virtual reality interior design and industrial equipment maintenance guidance, it is necessary to stably anchor virtual objects to physical space. Existing systems cause virtual objects to drift due to depth noise and rapid movement. This invention replaces the traditional NeRF volume rendering formula with a Manhattan Transformer based on NeRF rays, achieving efficient rendering while constraining appearance consistency across multiple viewpoints, enabling virtual objects to maintain sub-centimeter-level positioning accuracy under moving viewpoints.

[0169] Specifically, the multi-neural field implicit SLAM method for indoor environments provided by this invention aims to more effectively balance mapping accuracy and memory resources based on environmental density in indoor application scenarios. Simultaneously, the Manhattan Transformer based on NeRF rays replaces the traditional NeRF volume rendering formula, providing better scalability and robustness for mapping large indoor scenes, and effectively improving mapping accuracy and real-time performance. A summary flowchart of this invention is shown below. Figure 11 As shown, the specific architecture mainly consists of three parts: (1) Adaptive multi-neuron field: calculates the point density in the environment, and dynamically adjusts the radius of each neuron field to adapt to different environments; (2) Manhattan self-attention rendering: proposes a Manhattan ray converter to replace traditional volumetric rendering, which can render depth and color more flexibly and accurately; (3) Mapping and tracking: given an RGBD frame The camera pose and map are optimized alternately. During mapping, the map features stored in the hexahedral grid encoding and Gaussian orientation encoding are optimized by minimizing the mapping loss function. During tracking, the tracking loss function is minimized in parallel with the mapping operation.

[0170] The following is a detailed description of the specific experiments conducted in an indoor environment using the multi-neural field implicit SLAM method of this invention.

[0171] First, let's introduce the preparatory work before applying this invention. For hardware, an RGB-D camera or depth camera is needed to acquire RGB and depth images (an Astra Pro depth camera is used here, but a built-in depth camera on the robot is also acceptable); a graphics card or development board with at least 16GB of video memory is needed for SLAM algorithm calculations and 3D environment reconstruction and rendering. (An NVIDIA RTX 3090 24GB graphics card is used here; however, testing shows that an NVIDIA Jetson Orin Nano 16GB embedded development board can also run this SLAM algorithm). In terms of usage, the camera or robot should be placed in a well-lit indoor environment, facing forward with the camera's line of sight parallel to the ground. This facilitates the initialization of the SLAM algorithm and ensures optimal mapping results.

[0172] Unlike other neural implicit SLAM systems such as Co-SLAM, this invention proposes an adaptive multi-neural field in order to enable each neural field to fully adapt to its local environment and improve reconstruction efficiency. That is, by calculating the point density, the radius of each neural field can be flexibly adjusted.

[0173] Specifically, assuming there is RGB-D frames , which includes Keyframes, Keyframe Set and the corresponding keyframe pose set The representation of the adaptive multi-neuron field in the global map dictionary is shown in the following formula:

[0174] ,

[0175] in, Represents the set of all neural fields. Indicates the first The neural field of each keyframe This represents the radius of the initial field. Here, the adaptive neural field is based on keyframes, and each neural field is associated with a keyframe, establishing a many-to-one relationship.

[0176] The point density defined in the formula below is the number of 3D points within a spherical volume, with a radius twice that of the current neural field. This metric reflects the level of detail of an object; the higher the point density in the environment, the more complex the geometry of the reconstructed object, and the richer the details available for reconstruction. Therefore, in high-density regions, the radius of the neural field is reduced and the number of fields is increased to achieve finer sampling. The point density is represented as follows:

[0177] ,

[0178] in, Represents neural field The number of three-dimensional points in the data. Indicates volume, This represents the radius of the current neural field. To obtain the point density, a global IKD tree is first constructed to index the spatial distribution of all 3D points. Then, the local density is calculated by performing a range search on the IKD tree. Specifically, for the input point... Effectively found All neighboring points within the distance ,in This is the field radius stored in the global mapping dictionary. IKD trees significantly accelerate this neighborhood lookup. Finally, the point density is calculated using the formula for point density described above.

[0179] After calculating the density, a dynamic radius adjustment function is used to determine the radius of the newly instantiated neural field. Compared to the linear adjustment function of existing technologies, which can cause sudden changes in radius at the density threshold, leading to problems such as geometric breaks or artifacts (e.g., anomalous refraction at the edges of glass materials), the dynamic radius adjustment function of this embodiment effectively balances the coverage area of ​​the neural field and the accuracy of the reconstructed object. Specifically, the expression of the dynamic radius adjustment function is:

[0180] ,

[0181] in, and All represent density thresholds. Indicates the lower limit of the density threshold. This indicates the upper limit of the density threshold. This represents the attenuation coefficient, which controls the rate at which the radius shrinks as density increases.

[0182] As shown in the dynamic radius adjustment function above, the exponential function ensures smooth and gradual radius contraction in high-density regions, thereby preventing sampling discontinuities. The extreme value constraint mechanism prevents undersampling in high-density regions due to excessive radius contraction and avoids wasting computational resources in low-density regions due to radius expansion. The dynamic radius balancing mechanism of this invention enables the neural field to automatically adjust its receptive field size on complex geometric surfaces, focusing on details in high-density regions and expanding the coverage of low-density regions. Compared with traditional methods, the dynamic radius adjustment function significantly improves the spatial generalization ability of the neural field while maintaining 3D reconstruction accuracy.

[0183] In this embodiment of the invention, a two-stage strategy is used to instantiate the new neural field, such as... Figure 3 As shown. In the first stage, the set of 3D points initialized in the current RGB-D frame that are not included in the current neural field is searched. (unc is short for uncovered), which satisfies the constraint .in, Represents from keyframe pose transformation matrix to world coordinates This represents the coordinates of the neural field center. In the second stage, the initial 3D map is discretized into a shape with a side length of... 3D mesh, Initialize to And with the radius of the neural field And change. Design. To ensure complete spatial coverage of the voxel-centered neural field, if a non-empty voxel contains uncovered points but its center is not initialized with a neural field, a new field is initialized at the geometric center of the non-empty voxel. The 3D mesh also enforces a minimum distance between adjacent fields. To maintain sparsity, a random offset is applied to the mesh each time a keyframe is inserted, in order to mitigate the error in pose graph optimization.

[0184] like Figure 3 As shown, the current keyframe The set of all observed points is (Including all black and gray dots), where the lower gray dot set is... Not covered by existing neural sites. Utilizing a resolution of... The voxel grid defines the new neural field The center is chosen to maintain a suitable distance between neural fields. The radius of the new neural field is determined using the calculated density. This allows the neural field energy to completely cover New neural field Anchoring and Keyframes middle.

[0185] However, as the radius of the neural field changes, the side length of the voxel grid also changes. This will also change accordingly to maintain the appropriate distance between the centers of the new neural fields. A multi-resolution mesh strategy is employed, where the point density determines the mesh resolution. In high-density regions, both the field radius and the mesh edge length decrease. Conversely, in low-density environments, both increase simultaneously.

[0186] Side length of voxel mesh The change strategy is shown in the following formula:

[0187] ,

[0188] in, and All represent density thresholds. This mapping is learned through offline simulation, ensuring that the new neural field covers more than 90% of the 3D points in both smooth (e.g., walls and sofas) and detailed (e.g., bottles and flowers) scenes.

[0189] Sampling is performed within the neural field using NeRF rays, i.e., with the center of the camera... Choose the origin The neural field in the current keyframe, oriented towards the direction the camera is facing. Emit a ray. (Point) At fixed intervals Sampling is performed at the location, such as ,in This refers to the number of points sampled by ray tracing. Depth-guided sampling is used at distances from the object's surface. Within a fixed interval Sampling was performed, and samples were collected. The total number of sampling points is [number] points. Assume the set of points sampled by the ray within the grid is... It should be noted that, unlike traditional neural implicit SLAM systems such as Co-SLAM, the sampling in the embodiments of the present invention is performed within a selected neural field rather than on the surface of an object.

[0190] In this embodiment of the invention, hexahedral mesh encoding and Gaussian directional encoding are used to encode the sampling points. Hexahedral media mesh encoding preserves the color and geometric features of the object in the mapping. Compared to traditional hash mesh encoding, hexahedral mesh encoding uses a hexahedral lattice to accelerate the mapping and optimization process, ensuring the accuracy of object reconstruction and providing real-time performance for subsequent self-attention rendering methods. Gaussian directional encoding captures optical features such as reflectivity and object position features, overcoming the inherent reflectivity deficiency of NeRF. Finally, as... Figure 8 As shown, color features were obtained. Geometric features and location features In order to proceed with subsequent rendering.

[0191] In this embodiment of the invention, traditional NeRF volume rendering performs weighted aggregation on the output of all points, including geometric and color features. The weights are calculated using volume rendering formulas for occlusion modeling. It should be noted that this weighted aggregation process is learnable; therefore, this embodiment of the invention is based on a NeRF ray-based Manhattan Transformer. Figure 8 As shown, firstly, the encoding results of all acquisition points in the neural field ( , , The input is processed by MLP for feature concatenation (FC) to obtain the output. Secondly, in order to reduce the number of tokens and ensure real-time performance, features... Overlap space reduction was performed. Therefore, embodiments of the present invention introduce Manhattan self-attention, which utilizes Manhattan distance to better handle object edges through its axial spatial attenuation.

[0192] In addition, a Local Context Enhancement module is introduced to enhance local expressive power. Subsequently, the obtained... , and The matrix is ​​decomposed into horizontal and vertical components. , , and Then, depth is rendered using the decomposed Manhattan self-attention as shown in the following equation. ,color and signed distance function (SDF) is shown in the following formula:

[0193] ,

[0194] ,

[0195] in, , The decomposed spatial decay matrices use Manhattan distance as their decay feature. Experiments show that this decomposition process significantly optimizes rendering time. Furthermore, overlap space reduction significantly reduces the number of self-attention tokens through downsampling without compromising the Manhattan spatial decay feature. Weights It is an additional output for rendering depth. These are the output features. This is achieved by only applying the Transformer model to points with a density below a threshold. The neural field can be used to further reduce processing time.

[0196] Finally, based on the total mapping loss function and the tracking loss function, the camera pose and 3D scene map can be optimized. At this point, the SLAM process is complete. The specific methods for obtaining the total mapping loss function and the tracking loss function can be found in the previous text and will not be repeated here.

[0197] In this embodiment of the invention, the SLAM method of this embodiment, namely ATE-SLAM, was evaluated on four different indoor scene datasets. Regarding reconstruction performance, the Replica dataset [Straub J, Whelan T, Ma L, et al. The Replica dataset: A digital replica of indoor spaces[J]. arXivpreprint arXiv:1906.05797, 2019.], containing eight indoor scenes, and the NeuralRGBD synthetic dataset [Azinović D, Martin-Brualla R, Goldman DB, et al. Neural rgb-d surface reconstruction[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 6290-6301.], containing seven scenes, were used. The Replica indoor dataset is a large-scale multi-sensory 3D reconstruction database containing various scenes from the real world, such as offices, apartments, and libraries. The NeuralRGBD synthetic RGBD dataset was captured using a Structure IO camera. Regarding camera tracking performance, the ScanNet dataset [Dai A, Chang AX, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 5828-5839.] was used. This dataset contains RGB-D video data from 2.5 million viewpoints collected from various indoor environments, used for annotation of 3D camera pose detection, surface reconstruction, and instance-level semantic segmentation. Furthermore, ScanNet includes over 1500 scanned indoor environments, covering various scenes such as homes, offices, and public spaces.TUM [Sturm J, Engelhard N, Endres F, et al. A benchmark for the evaluation of RGB-D SLAM systems[C] / / 2012 IEEE / RSJinternational conference on intelligent robots and systems. IEEE, 2012: 573-580.] is a dataset for evaluating visual odometry and visual SLAM systems. Real-world trajectories were acquired from the TUM dataset using a high-precision motion capture system and a high-speed tracking camera (100 Hz).

[0198] To eliminate noise clouds outside the scene, a mesh culling operation was performed, and a virtual camera view covering the occluded areas was also predicted. After mesh culling, the performance of 3D reconstruction was evaluated by accuracy error [cm], completion error [cm], and completion rate [< 5cm%]. For camera tracking evaluation, the root mean square error of the absolute trajectory error (RMSE) was used. [cm]. First, two point clouds, P and Q, were uniformly sampled, one from the grid on the real map and the other from the grid on the reconstructed map. These point clouds have the same dimension, |P| = |Q| = 200000. The accuracy error is defined as the average distance between the nearest point on the real map grid and the nearest point on the reconstructed map grid. The mapping metrics are defined as shown in Table 1.

[0199] Table 1 Mapping Evaluation Indicators

[0200] This invention, ATE-SLAM, was implemented on a server equipped with a 2.60GHz Intel Xeon® Platinum 8358 CPU and an NVIDIA RTX 3090 GPU. For the dynamic radius adjustment function, Set to 0.4m. Set to 0.05m. Set to 20, Set to 100. For multi-resolution grid adjustment strategies, Set to 20, Set to 80. For keyframe selection, if the keyframe database contains fewer than 20 keyframes for bundle adjustment... No more than 1,000 randomly sampled neural fields For camera tracking, under the default settings... The pixel was sampled 10 times. Additionally, the truncation distance was... Set the height to 10cm. For camera pose and dense mapping, use the AdamW optimizer for optimization.

[0201] Compared to Co-SLAM, the MAR-SLAM embodiment of this invention shows an overall improvement in mapping performance on the Replica dataset. Specifically, the average completion rate of Co-SLAM reached 93.44%. As shown in Table 2, the MAR-SLAM embodiment of this invention slightly improved the completion error, reaching 93.66%. This improvement is also reflected in the accuracy error and completion error, reaching 2.00 cm and 2.07 cm, respectively. MAR-SLAM also significantly outperforms NICE-SLAM in high-fidelity reconstruction performance, improving the completion rate by 2.53%. MAR-SLAM also far surpasses iMAP in completion rate, improving it by approximately 13%. The evaluation of 3D reconstruction performance on the Replica dataset is shown in Table 2. In Table 2, bold text indicates that the current method of this invention achieved the best results in the current dataset scenario. When there are local lighting changes in the scene (such as the movement of sunlight patches caused by the swaying of curtains), existing neural implicit SLAM (such as NICE-SLAM) is prone to misinterpreting appearance changes as geometric changes. This invention achieves a reconstruction completion rate of up to 91.22% by using adaptive multi-neural fields to balance memory resources and mapping accuracy, and by combining Gaussian directional coding to preserve optical effects, even in scenarios with sudden changes in lighting in the dataset.

[0202] Table 2 Evaluation of 3D Reconstruction Performance on the Replica Dataset

[0203] Unlike the Replica dataset, the NeuralRGBD synthetic dataset contains seven scenes and includes noise from the depth sensor. In Table 3, the MAR-SLAM implementation of this invention slightly outperforms Co-SLAM across all metrics and significantly surpasses NICE-SLAM. Specifically, MAR-SLAM achieves a completion rate of 90.22%, while Co-SLAM only reaches approximately 89%. Furthermore, it shows additional improvements in accuracy error and completion error, reaching 2.64 cm and 2.59 cm, respectively.

[0204] Table 3 Evaluation of 3D Reconstruction Results on NeuralRGBD Synthetic Dataset

[0205] As one of the most challenging indoor scene datasets, the ScanNet dataset was used for camera tracking evaluation. Using absolute trajectory error (ATE RMSE) as a measure of tracking accuracy, this invention still demonstrates competitiveness compared to benchmark methods. Specifically, as shown in Table 4, the method of this embodiment achieves an average absolute trajectory error of 8.70 cm, which is 0.67 cm lower than Co-SLAM. Experimental results show that the method of this embodiment performs well in many scenes, such as scene0000, scene0059, and scene0181. It should be noted that only the average of 10 runs is used in this embodiment for fair comparison. Traditional voxel meshes face memory explosion or loss of detail in scenes such as large shopping malls. Hexahedral mesh encoding combined with Gaussian directional encoding achieves 0.8 mm geometric accuracy on the ScanNet dataset while using only 1.2 GB of GPU memory, saving 73% of memory compared to Co-SLAM. The absolute trajectory error evaluation results on the ScanNet dataset are shown in Table 4.

[0206] Table 4. Absolute trajectory error evaluation on the ScanNet dataset.

[0207] To further demonstrate the robustness of this invention, MAR-SLAM, camera tracking performance was evaluated using the TUM dataset in the same manner as the ScanNet dataset. As shown in Table 5, by averaging 10 runs, it was found that MAR-SLAM maintains the same performance as Co-SLAM, while being more accurate than NICE-SLAM in tracking trajectories (see Table 5). Specifically, the absolute trajectory error of MAR-SLAM reached 2.3 cm, significantly lower than Co-SLAM's 2.4 cm. Meanwhile, the Manhattan Transformer based on NeRF rays, employing overlap space reduction, reduced self-attention tokens, greatly optimizing tracking time. On a 30m trajectory in the TUM dataset, the average optimization time decreased from 210ms to 89ms, while maintaining an ATE error below 0.11m. The absolute trajectory error evaluation compared to the traditional TUM dataset is shown in Table 5.

[0208] Table 5. Absolute trajectory error evaluation under the TUM dataset.

[0209] Temporal performance is also a key metric for visual SLAM systems. This invention provides a comprehensive evaluation of the SLAM system, including tracking and mapping times. The average time for each tracking iteration or mapping iteration was calculated on the ScanNet and TUM datasets. In this ablation experiment, the effects of adding Manhattan self-attention decomposition and overlap space reduction on the overall time were tested. As shown in Table 6, the results indicate that piecewise overlap space reduction can improve tracking and mapping speeds to some extent. On the ScanNet dataset, MAR-SLAM achieved a single-frame execution time of 79.88 ms, an improvement of 6.24 ms. Furthermore, Manhattan self-attention decomposition had no significant impact. The entire SLAM system operated at a frequency of 5–8 Hz during the testing period. Temporal performance analysis is shown in Table 6.

[0210] Table 6 Time Performance Analysis

[0211] In summary, the multi-neural field implicit SLAM method for indoor environments provided in this invention, namely MAR-SLAM, effectively balances memory resources and mapping accuracy through its multi-neural field approach, producing high-quality reconstruction results. Furthermore, the Manhattan Transformer based on NeRF rays can replace traditional volume rendering formulas, enhancing the robustness and scalability of large-scale indoor mapping. Experimental results demonstrate that the method of this invention can effectively improve high-fidelity reconstruction performance while maintaining tracking accuracy.

[0212] As another embodiment of the present invention, a multi-neural field implicit SLAM device 100 for indoor environments is provided to implement the multi-neural field implicit SLAM method for indoor environments described above, wherein, as Figure 12 As shown, it includes:

[0213] The acquisition module 110 is used to acquire an RGB-D image sequence of an indoor environment, wherein the RGB-D image sequence includes an image sequence composed of RGB images and depth images acquired according to a preset frame rate;

[0214] The initialization module 120 is used to perform multi-neural field initialization processing on the RGB-D image sequence to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius.

[0215] The ray sampling module 130 is used to perform ray sampling in each neural field in the three-dimensional global map to obtain the three-dimensional map feature vector.

[0216] Feature encoding module 140 is used to perform feature encoding on the feature vector of the three-dimensional map to obtain three-dimensional feature encoding;

[0217] The deep learning and rendering module 150 is used to perform deep learning and rendering on the three-dimensional feature encoding based on the Manhattan self-attention network to obtain the rendered RGB image and the rendered depth image.

[0218] The localization and mapping module 160 is used to perform loss learning based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment.

[0219] This invention provides a multi-neural field implicit SLAM device for indoor environments. It acquires an RGB-D image sequence of the indoor environment, performs multi-neural field initialization processing on the RGB-D image sequence, performs ray sampling within each neural field, and then encodes the 3D map features after ray sampling. Based on a Manhattan self-attention network, it performs deep learning and rendering of the 3D feature encoding, and finally, after lossy schooling, obtains the localization and mapping results of the indoor environment. This multi-neural field implicit SLAM device for indoor environments effectively balances memory resources and mapping accuracy through adaptive multi-neural fields, obtaining high-quality mapping results. The rendering method based on ray sampling and Manhattan self-attention networks enhances the robustness and scalability of mapping large indoor environments, thereby effectively improving the performance of high-fidelity reconstruction while maintaining mapping accuracy.

[0220] The specific working principle of the multi-neural field implicit SLAM device for indoor environments of the present invention can be referred to the previous description, and will not be repeated here.

[0221] As another embodiment of the present invention, a multi-neural field implicit SLAM system 10 for indoor environments is provided, wherein, as Figure 13 As shown, it includes: an image acquisition device 200 and the aforementioned multi-neural field implicit SLAM device 100 for indoor environments, wherein the image acquisition device 200 is communicatively connected to the multi-neural field implicit SLAM device 100 for indoor environments.

[0222] The image acquisition device 200 is used to acquire RGB images and depth images of the indoor environment according to a preset frame rate to obtain an RGB-D image sequence of the indoor environment;

[0223] The multi-neural field implicit SLAM device 100 for indoor environments is used to achieve tracking, localization, and mapping based on multi-neural fields and Manhattan self-attention networks according to the RGB-D image sequence of the indoor environment, thereby obtaining the localization and mapping results of the indoor environment.

[0224] The multi-neural field implicit SLAM system for indoor environments provided by this invention adopts the aforementioned multi-neural field implicit SLAM device for indoor environments. Through adaptive multi-neural fields, it can effectively balance memory resources and mapping accuracy to obtain high-quality mapping results. Through the rendering method based on ray sampling and Manhattan self-attention network, it can enhance the robustness and scalability of mapping of large indoor environments, thereby enabling indoor environments to effectively improve the performance of high-fidelity reconstruction while maintaining mapping accuracy.

[0225] The specific working principle of the multi-neural field implicit SLAM system for indoor environments of the present invention can be referred to the previous description, and will not be repeated here.

[0226] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-neural field implicit SLAM method for indoor environments, characterized in that, include: Acquire an RGB-D image sequence of an indoor environment, the RGB-D image sequence comprising an image sequence composed of RGB images and depth images acquired according to a preset frame rate; The RGB-D image sequence is subjected to multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius; Ray sampling is performed within each neural field of the three-dimensional global map to obtain the three-dimensional map feature vector; The feature vectors of the three-dimensional map are subjected to feature encoding to obtain three-dimensional feature codes; The three-dimensional feature encoding is processed by deep learning and rendering using a Manhattan self-attention network to obtain a rendered RGB image and a rendered depth image. Loss learning is performed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image to obtain the localization and mapping results of the indoor environment.

2. The multi-neural field implicit SLAM method for indoor environments according to claim 1, characterized in that, The RGB-D image sequence is subjected to multi-neural field initialization processing to obtain a three-dimensional global map with multiple neural fields, including: Iterate over the RGB-D image sequence; When the current frame of the iteration is the first frame of the RGB-D image sequence, the grid with fixed side length is divided according to the distribution of the three-dimensional point cloud on the initial three-dimensional map, and an initial neural field with a preset fixed radius is formed at the center of the grid where the three-dimensional point cloud exists. When the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined according to the density of the three-dimensional point cloud in the initial three-dimensional map, and a new neural field with multiple adjustable radii is formed. Complete the iteration of all RGB-D image sequences to obtain a three-dimensional global map with multiple neural fields.

3. The multi-neural field implicit SLAM method for indoor environments according to claim 2, characterized in that, When the current frame of the iteration is not the first frame of the RGB-D image sequence, the radius of the new neural field is determined based on the density of the 3D point cloud in the initial 3D map, and a new neural field with multiple adjustable radii is formed, including: Find the set of 3D point clouds that are not included in the current initial neural field; The adjustable radius of the new neural field is determined based on the density of the three-dimensional point cloud set that is not included in the current initial neural field. The adjustable side length of the mesh is determined based on the density of the three-dimensional point cloud set that is not included in the current initial neural field. The location of the new neural field is determined based on the center of the grid.

4. The multi-neural field implicit SLAM method for indoor environments according to claim 1, characterized in that, Ray sampling is performed within each neural field of the three-dimensional global map to obtain a three-dimensional map feature vector, including: According to NeRF rays, rays are emitted within each nerve field; The ray is sampled at intervals of a first preset distance to obtain ray sampling points; Depth-guided sampling is performed at intervals of a second preset distance to obtain depth-guided sampling points; The 3D map feature vector is obtained based on the ray sampling points and the depth-guided sampling points.

5. The multi-neural field implicit SLAM method for indoor environments according to claim 1, characterized in that, The 3D map feature vector is subjected to feature encoding to obtain 3D feature encoding, including: The feature vectors of the three-dimensional map are encoded using hexahedral grid coding to obtain the color and geometric features of the three-dimensional feature code. The feature vector of the three-dimensional map is encoded using Gaussian directional coding to obtain the positional features of the three-dimensional feature code.

6. The multi-neural field implicit SLAM method for indoor environments according to claim 1, characterized in that, Based on the Manhattan self-attention network, deep learning and rendering are performed on the 3D feature encoding to obtain the rendered RGB image and the rendered depth map, including: The three-dimensional feature stitching result is obtained by stitching together the color and geometric features and the positional features of the three-dimensional feature encoding. The Manhattan attention network is used to render the 3D feature stitching result using depth, color, and a signed distance function to obtain the rendered RGB image and the rendered depth image.

7. The multi-neural field implicit SLAM method for indoor environments according to claim 6, characterized in that, The Manhattan attention network is used to render the 3D feature stitching result using depth, color, and a signed distance function to obtain a rendered RGB image and a rendered depth image, including: The overlapping space attenuation is applied to the 3D feature stitching result using the Manhattan self-attention network to obtain the 3D feature stitching result after overlapping space attenuation; Local enhancement is performed on the 3D feature stitching result after overlapping spatial attenuation based on the local enhancement network to obtain the enhanced query vector, key vector and value vector; The Manhattan self-attention is obtained by decomposing the enhanced query vector, key vector, and value vector. The depth, color, and signed distance are rendered based on the decomposed Manhattan self-attention, resulting in a rendered RGB image and a rendered depth image.

8. The multi-neural field implicit SLAM method for indoor environments according to claim 1, characterized in that, Loss learning is performed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment, including: The color loss function and the depth loss function are constructed based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth image. Determine the free space loss function and the truncation loss function; Construct the total mapping loss function based on the color loss function, depth loss function, free space loss function, and truncation loss function; Construct a tracking loss function based on the motion model; The indoor environment location and mapping results are obtained by minimizing the total mapping loss function and the tracking loss function.

9. A multi-neural field implicit SLAM device for indoor environments, used to implement the multi-neural field implicit SLAM method for indoor environments as described in any one of claims 1 to 8, characterized in that, include: The acquisition module is used to acquire an RGB-D image sequence of an indoor environment, wherein the RGB-D image sequence includes an image sequence composed of RGB images and depth images acquired according to a preset frame rate; An initialization module is used to perform multi-neural field initialization processing on the RGB-D image sequence to obtain a three-dimensional global map with multiple neural fields, wherein each neural field includes a spherical region with a preset initial radius. The ray sampling module is used to perform ray sampling in each neural field of the three-dimensional global map to obtain the feature vector of the three-dimensional map. The feature encoding module is used to encode the feature vectors of the three-dimensional map to obtain three-dimensional feature codes; The deep learning and rendering module is used to perform deep learning and rendering on the three-dimensional feature encoding based on the Manhattan self-attention network to obtain the rendered RGB image and the rendered depth image. The localization and mapping module is used to perform loss learning based on the RGB-D image sequence of the indoor environment, the rendered RGB image, and the rendered depth map to obtain the localization and mapping results of the indoor environment.

10. A multi-neural field implicit SLAM system for indoor environments, characterized in that, include: The image acquisition device and the multi-neural field implicit SLAM device for indoor environments as described in claim 9 are connected in communication, wherein the image acquisition device is communicatively connected to the multi-neural field implicit SLAM device for indoor environments. The image acquisition device is used to acquire RGB images and depth images of the indoor environment according to a preset frame rate to obtain an RGB-D image sequence of the indoor environment; The multi-neural field implicit SLAM device for indoor environments is used to achieve tracking, localization, and mapping based on multi-neural fields and Manhattan self-attention networks according to the RGB-D image sequence of the indoor environment, thereby obtaining the localization and mapping results of the indoor environment.

Images