A SLAM construction method and system based on RGB-D semantic segmentation

Abstract

This invention provides a SLAM construction method and system based on RGB-D semantic segmentation, belonging to the field of robot localization and navigation technology. The method includes: constructing an RGB-D-based SLAM system framework; training an RGB-D semantic segmentation model based on Transformer-based dual-branch multi-level attention fusion within the SLAM system framework to obtain an initial semantic segmentation model; and pre-training the initial semantic segmentation model using a depth noise decoder to obtain a SLAM semantic segmentation model. This invention combines RGB-D images and a depth completion algorithm in a visual SLAM system, and by introducing an RGB-D semantic segmentation model, ensures the robustness of the acquired semantic information under poor imaging conditions.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to a SLAM construction method and system based on RGB-D semantic segmentation. Background Technology

[0002] Semantic vision SLAM is a low-cost, high-efficiency Simultaneous Localization and Mapping (SLAM) system used by robots to create maps of their surroundings for tasks such as decision-making and navigation. Semantic vision SLAM extracts semantic information from images acquired by a camera.

[0003] However, visual SLAM systems that use RGB-D cameras as sensors can acquire RGB images and depth images simultaneously, but existing RGB-D SLAM systems only use semantic acquisition models based on RGB images to extract semantics, and their semantics are limited by imaging conditions, which mainly have the following two problems.

[0004] (1) In poor imaging conditions such as strong light, weak light, and transparent objects, incomplete and inaccurate semantics will have a negative impact on the visual SLAM system.

[0005] (2) The spatial information of depth was not fully utilized to extract semantics, resulting in a waste of information. Summary of the Invention

[0006] This invention provides a SLAM construction method and system based on RGB-D semantic segmentation, which addresses the numerous limitations of using RGB-D images in existing SLAM systems.

[0007] In a first aspect, the present invention provides a SLAM construction method based on RGB-D semantic segmentation, comprising:

[0008] Construct a SLAM system framework based on RGB-D;

[0009] In the SLAM system framework, an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion is used for training to obtain the initial semantic segmentation model;

[0010] The initial semantic segmentation model is pre-trained with a deep noise decoder to obtain the SLAM semantic segmentation model.

[0011] According to the SLAM construction method based on RGB-D semantic segmentation provided by the present invention, a framework for a simultaneous localization and mapping SLAM system based on color depth RGB-D is constructed, including:

[0012] Acquire sensor data from an RGB-D camera, and acquire multiple adjacent frames of the sensor data;

[0013] ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships;

[0014] By introducing a semantic segmentation thread based on the feature point matching relationship, a semantic segmentation result is obtained. The semantic segmentation result is then passed to a tracking thread, and multiple feature points are sent to a visual odometry system.

[0015] Once the visual odometry completes tracking of adjacent frames and obtains multiple key frames, the multiple key frames are determined for entry and exit loop threads;

[0016] After the loopback thread passes verification, a semantic octree map is constructed to obtain the SLAM system framework.

[0017] According to the present invention, a SLAM construction method based on RGB-D semantic segmentation acquires sensor data from an RGB-D camera and acquires multiple adjacent frames of the sensor data, including:

[0018] The parameters of the RGB-D camera are calibrated, and color and depth images are acquired.

[0019] Acquire multiple adjacent frames from the color image and the depth image.

[0020] According to the SLAM construction method based on RGB-D semantic segmentation provided by the present invention, ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships, including:

[0021] The pose of the first keyframe is determined as the reference world coordinate system, and the spatial points of the first keyframe are used as map points to obtain the initial map.

[0022] The feature point matching relationship between the multiple adjacent frames is obtained using an optical flow pyramid.

[0023] According to the present invention, a SLAM construction method based on RGB-D semantic segmentation is provided, which introduces the semantic segmentation thread by matching the feature points to obtain the semantic segmentation result, passes the semantic segmentation result to the tracking thread, and sends multiple feature points to a visual odometry system, including:

[0024] By combining semantic information, geometric outliers in the feature point matching relationship are examined, dynamic feature points are removed, and static feature points are retained to obtain the semantic segmentation result.

[0025] Multiple feature points from the semantic segmentation result are input into the visual odometry.

[0026] According to the SLAM construction method based on RGB-D semantic segmentation provided by the present invention, after the visual odometry completes tracking of adjacent frames and obtains multiple key frames, the method determines the multiple key frames for loop-in / loop-out threads, including:

[0027] Track the local map, update and optimize the local map, and determine the multiple keyframes;

[0028] Candidate keyframes are identified from the plurality of keyframes, and loop closure verification is performed on the candidate keyframes to obtain candidate loop closure frames;

[0029] The candidate loopback frames are corrected to obtain updated keyframes.

[0030] According to the SLAM construction method based on RGB-D semantic segmentation provided by the present invention, an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion is trained in the SLAM system framework to obtain an initial semantic segmentation model, including:

[0031] The RGB image and the depth image are channel aligned to obtain the aligned RGB image and HHA image;

[0032] The pair of feature maps output from each stage are input into the Transformer unit for calibration and fusion to obtain a fused feature map. The fused feature map is then added to the pair of feature maps and the average is taken to obtain the processed fused feature map.

[0033] The final processed and fused feature map is input into the Separation and Aggregation Gate (SA-Gate) unit to obtain the output feature map, which is then merged with the output of the first Transformer module to obtain the initial semantic segmentation model.

[0034] According to the SLAM construction method based on RGB-D semantic segmentation provided by the present invention, the initial semantic segmentation model is pre-trained with a deep noise decoder to obtain a SLAM semantic segmentation model, including:

[0035] The encoder was pre-trained for classification using the ImageNet dataset, and the encoder parameters were frozen.

[0036] Depth noise is pre-trained by a preset objective function;

[0037] Pre-training is performed on the DREDS dataset to minimize the error of the preset objective function;

[0038] Unfreeze the encoder parameters and fine-tune them using the cross-entropy loss function to obtain the SLAM semantic segmentation model.

[0039] Secondly, the present invention also provides a SLAM construction system based on RGB-D semantic segmentation, comprising:

[0040] The building module is used to construct the framework of a simultaneous localization and mapping (SLAM) system based on color depth RGB-D.

[0041] The training module is used to train an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion in the SLAM system framework to obtain an initial semantic segmentation model.

[0042] The adjustment module is used to pre-train the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

[0043] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the SLAM construction method based on RGB-D semantic segmentation as described above.

[0044] The SLAM construction method and system based on RGB-D semantic segmentation provided by this invention combine RGB-D images and depth completion algorithms in the visual SLAM system, and introduces an RGB-D semantic segmentation model to ensure that the acquired semantic information is robust under poor imaging conditions. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the SLAM construction method based on RGB-D semantic segmentation provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the SLAM system framework for RGB-D semantic segmentation provided by the present invention;

[0048] Figure 3 This is a flowchart of the RGB-D semantic segmentation model provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the decoding pre-training strategy provided by the present invention;

[0050] Figure 5This is a schematic diagram of the SLAM construction system based on RGB-D semantic segmentation provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Figure 1 This is a flowchart illustrating the SLAM construction method based on RGB-D semantic segmentation provided in this embodiment of the invention, as shown below. Figure 1 As shown, it includes:

[0054] Step 100: Construct a SLAM system framework based on RGB-D;

[0055] Step 200: In the SLAM system framework, an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion is used for training to obtain an initial semantic segmentation model;

[0056] Step 300: Pre-train the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

[0057] Specifically, in response to the shortcomings of existing technologies, this invention proposes a visual SLAM system based on RGB-D image semantic segmentation and depth completion algorithms, comprising:

[0058] First, we construct a SLAM system framework based on RGB-D semantic segmentation;

[0059] Then, the RGB-D semantic segmentation model is trained, and the RGB-D semantic segmentation model design based on Transformer dual-branch multi-level attention fusion is adopted.

[0060] Finally, the deep noise decoder pre-training strategy of the RGB-D semantic segmentation model is applied to obtain the SLAM semantic segmentation model.

[0061] This invention combines RGB-D images and depth completion algorithms in a visual SLAM system, and introduces an RGB-D semantic segmentation model to ensure that the acquired semantic information is robust under poor imaging conditions.

[0062] Based on the above embodiments, a framework for a simultaneous localization and mapping (SLAM) system based on color depth RGB-D is constructed, including:

[0063] Acquire sensor data from an RGB-D camera, and acquire multiple adjacent frames of the sensor data;

[0064] ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships;

[0065] By introducing a semantic segmentation thread based on the feature point matching relationship, a semantic segmentation result is obtained. The semantic segmentation result is then passed to a tracking thread, and multiple feature points are sent to a visual odometry system.

[0066] Once the visual odometry completes tracking of adjacent frames and obtains multiple key frames, the multiple key frames are determined for entry and exit loop threads;

[0067] After the loopback thread passes verification, a semantic octree map is constructed to obtain the SLAM system framework.

[0068] This includes acquiring sensor data from an RGB-D camera, and acquiring multiple adjacent frames of the sensor data, including:

[0069] The parameters of the RGB-D camera are calibrated, and color and depth images are acquired.

[0070] Acquire multiple adjacent frames from the color image and the depth image.

[0071] Specifically, ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships, including:

[0072] The pose of the first keyframe is determined as the reference world coordinate system, and the spatial points of the first keyframe are used as map points to obtain the initial map.

[0073] The feature point matching relationship between the multiple adjacent frames is obtained using an optical flow pyramid.

[0074] Specifically, a semantic segmentation thread is introduced by matching the feature points to obtain semantic segmentation results. These results are then passed to a tracking thread, and multiple feature points are fed into a visual odometry system, including:

[0075] By combining semantic information, geometric outliers in the feature point matching relationship are examined, dynamic feature points are removed, and static feature points are retained to obtain the semantic segmentation result.

[0076] Multiple feature points from the semantic segmentation result are input into the visual odometry.

[0077] The process involves the visual odometry completing the tracking of adjacent frames to obtain multiple key frames, and determining the multiple key frames for the loopback / entry thread, including:

[0078] Track the local map, update and optimize the local map, and determine the multiple keyframes;

[0079] Candidate keyframes are identified from the plurality of keyframes, and loop closure verification is performed on the candidate keyframes to obtain candidate loop closure frames;

[0080] The candidate loopback frames are corrected to obtain updated keyframes.

[0081] Specifically, see Figure 2 As shown, this embodiment of the invention requires acquiring sensor data from an RGB-D camera. The RGB-D camera provides information on color and depth images, which is typically transmitted to a computer via USB or other interfaces for camera calibration, determining the camera's internal parameters (such as focal length, principal point, etc.) and distortion coefficients.

[0082] Set the pose of the first keyframe to the reference world coordinate system, and create the spatial points corresponding to its feature points as map points to complete the map initialization.

[0083] The optical flow pyramid is used to quickly obtain the feature point matching relationship between adjacent frames. For successfully matched feature points, if they are located at the image edge, they are directly ignored. Then, for each pair of matched feature points, the total value of the gray value difference in the 3×3 region centered on each pair is calculated. If the total value is greater than a preset threshold φ, the pair of matched points is discarded. The filtered matched feature point pairs are then used to calculate the fundamental matrix using the Random Sampling Consensus (RANSAC) algorithm. Then, the epipolar distance is calculated, and it is determined whether the distance from the matched point to the corresponding epipolar line is greater than the set threshold. If it is greater than the threshold, it is determined that the matched point is moving.

[0084] Then, a semantic segmentation thread is introduced, which utilizes the semantic information of the RGB-D semantic segmentation model and passes the semantic segmentation results to the tracking thread. In the semantic segmentation thread, geometric outliers are checked by combining semantic information, dynamic feature points are eliminated, while feature points of stationary potential moving objects are retained, and the feature points are sent to the visual odometry.

[0085] Once the visual odometry successfully tracks adjacent frames, it will then track the local map to obtain a more accurate pose. If tracking adjacent frames or the local map fails, the system will use relocalization to obtain the pose. After completing the localization of the current frame, the tracking thread will decide whether to create the current frame as a keyframe.

[0086] The local mapping thread is responsible for maintaining and updating the local map and performing local optimizations, including inserting new keyframes and map points, removing redundant map points, removing old local keyframes, and performing local BA to optimize pose and map points. The loop closure thread mainly includes loop closure detection, loop closure correction, and global optimization. The loop closure detection part includes two steps: finding candidate keyframes and loop closure verification. To detect loop closures, candidate loop closure keyframes are first searched in the historical keyframe library. For the current keyframe, its bag-of-words vector is extracted and compared with the similarity of other adjacent keyframes in the co-view. The lowest similarity score is used as the loop closure judgment threshold. Based on this threshold, candidate loop closure frames are searched in the re-identification database. A candidate loop closure frame and its co-view frame need to be searched three times consecutively to be considered a credible candidate loop closure frame. After the loop closure verification is successful, the loop closure thread will perform loop closure correction. In this step, duplicate map points will be merged, new loop edges will be added to the co-view, the pose of the current keyframe will be corrected, and the correction result will be passed to other keyframes that have a co-view relationship with the current keyframe. The new co-view relationship and map point information will be updated.

[0087] Finally, the semantic octree map construction thread is entered, which uses the depth image, the segmentation results provided by the semantic segmentation thread, and the pose estimation results to construct the semantic octree map.

[0088] Based on the above embodiments, an RGB-D semantic segmentation model with dual-branch multi-level attention fusion based on Transformer is used for training in the SLAM system framework to obtain an initial semantic segmentation model, including:

[0089] The RGB image and the depth image are channel aligned to obtain the aligned RGB image and HHA image;

[0090] The pair of feature maps output from each stage are input into the Transformer unit for calibration and fusion to obtain a fused feature map. The fused feature map is then added to the pair of feature maps and the average is taken to obtain the processed fused feature map.

[0091] The final processed and fused feature map is input into the Separation and Aggregation Gate (SA-Gate) unit to obtain the output feature map, which is then merged with the output of the first Transformer module to obtain the initial semantic segmentation model.

[0092] Specifically, such as Figure 3 As shown, first align the input channels of the RGB-D image:

[0093] The model input consists of an RGB image and an HHA image. The RGB image has 3 channels, and the depth map has 1 channel. For channel alignment, the depth map can be re-encoded into a three-channel image. The three channels represent the horizontal parallax, the height above the ground, the local surface normal of the pixel, and the tilt angle between the inferred gravity direction. This encoding method is called HHA. Then, the 3-channel data obtained from the HHA encoding is linearly transformed to 0-255 to obtain the HHA image aligned with the RGB image.

[0094] The pair of feature maps output from each stage are fed into the Transformer unit for information calibration and fusion midway through, resulting in a fused feature map. This fused feature map is then added back to the input feature map, averaged, and fed into the next stage.

[0095] The last pair of feature maps are fed into the SA-Gate unit for feature separation and feature aggregation. The feature separation stage includes: (1) First, a set of RGB feature maps and HHA feature maps are concatenated, and then global average pooling is performed to obtain feature vector I. (2) Then, I is passed through two multi-layer perceptrons (MLPs) and then normalized by the sigmoid function to obtain a set of attention vectors. (3) The set of attention vectors is multiplied by the HHA feature map and the RGB feature map by channel to obtain the offset features used for calibration. (4) Finally, the offset features are added to the input features to obtain the calibrated features. The feature aggregation stage includes: (1) First, the calibrated RGB features and HHA features output from feature separation are concatenated, and then two mapping functions (i.e., 1×1 convolution) are used to map the high-dimensional features to two different spatial-wise gates. (2) Then, the weights of each position in the RGB feature map and HHA feature map are obtained by processing with Softmax. (3) Finally, the weights are multiplied by a set of feature maps input to the SA-Gate according to their spatial positions, and then the sums are output as a fused feature map.

[0096] The output of the final SA-Gate and the output of the first Transformer module are fed into the decoder (mainly DeepLab v3+) to obtain the final mask.

[0097] Based on the above embodiments, the initial semantic segmentation model is pre-trained with a deep noise decoder to obtain a SLAM semantic segmentation model, including:

[0098] The encoder was pre-trained for classification using the ImageNet dataset, and the encoder parameters were frozen.

[0099] Depth noise is pre-trained by a preset objective function;

[0100] Pre-training is performed on the DREDS dataset to minimize the error of the preset objective function;

[0101] Unfreeze the encoder parameters and fine-tune them using the cross-entropy loss function to obtain the SLAM semantic segmentation model.

[0102] Specifically, the encoder is pre-trained for classification on the ImageNet dataset, and then the encoder parameters are frozen.

[0103] Decoding pre-training with deep noise: Let the encoder be f θ The decoder is g φ Using parameters θ and φ respectively, this model converts RGB-D images into... and As input, it is transformed into a mask image. Define x d The original, noisy depth image. For the repaired depth image, in order to perform depth noise pre-training, it is necessary to minimize the reconstruction error. Therefore, the objective function of this optimization problem is defined as:

[0104]

[0105] in This indicates that for a noisy depth map x d The expected operation is used to average all possible original noisy depth maps. The square of the Euclidean distance between the output obtained by the encoder and decoder and the repaired depth map is the square of the reconstruction error. Minimizing the reconstruction error can be used to optimize the model parameters so that the model can more accurately reproduce the input.

[0106] Pre-training was performed on the DREDS dataset, whose training set contains 100k RGB-D images consisting of 1801 objects from the ShapeNetCore database, covering 7 categories. The error objective function was minimized on the dataset.

[0107] Unfreeze the encoder parameters, use the cross-entropy loss function, and fine-tune it on the semantic segmentation task to obtain the final semantic segmentation model.

[0108] The SLAM construction system based on RGB-D semantic segmentation provided by this invention will be described below. The SLAM construction system based on RGB-D semantic segmentation described below can be referred to in correspondence with the SLAM construction method based on RGB-D semantic segmentation described above.

[0109] Figure 5This is a schematic diagram of the SLAM construction system based on RGB-D semantic segmentation provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: a construction module 51, a training module 52, and a tuning module 53, wherein:

[0110] The construction module 51 is used to construct a simultaneous localization and mapping SLAM system framework based on color depth RGB-D; the training module 52 is used to train the RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion in the SLAM system framework to obtain an initial semantic segmentation model; the adjustment module 53 is used to pre-train the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

[0111] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a SLAM construction method based on RGB-D semantic segmentation. This method includes: constructing an RGB-D-based SLAM system framework; training an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion within the SLAM system framework to obtain an initial semantic segmentation model; and pre-training the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

[0112] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the SLAM construction method based on RGB-D semantic segmentation provided by the above methods. The method includes: constructing an RGB-D based SLAM system framework; training an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion in the SLAM system framework to obtain an initial semantic segmentation model; and pre-training the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

[0114] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A SLAM construction method based on RGB-D semantic segmentation, characterized in that, include: Construct a framework for a simultaneous localization and mapping (SLAM) system based on color depth RGB-D; In the SLAM system framework, an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion is used for training to obtain the initial semantic segmentation model; The initial semantic segmentation model is pre-trained with a deep noise decoder to obtain a SLAM semantic segmentation model; A framework for a simultaneous localization and mapping (SLAM) system based on color depth RGB-D is constructed, including: Acquire sensor data from an RGB-D camera, and acquire multiple adjacent frames of the sensor data; ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships; By introducing a semantic segmentation thread based on the feature point matching relationship, a semantic segmentation result is obtained. The semantic segmentation result is then passed to a tracking thread, and multiple feature points are sent to a visual odometry system. Once the visual odometry completes tracking of adjacent frames and obtains multiple key frames, the multiple key frames are determined for entry and exit loop threads; After the loopback thread is verified, a semantic octree map is constructed to obtain the SLAM system framework. By introducing a semantic segmentation thread based on the feature point matching relationship, a semantic segmentation result is obtained. This semantic segmentation result is then passed to a tracking thread, and multiple feature points are fed into a visual odometry system, including: By combining semantic information, geometric outliers in the feature point matching relationship are examined, dynamic feature points are removed, and static feature points are retained to obtain the semantic segmentation result. Multiple feature points from the semantic segmentation result are input into the visual odometry.

2. The SLAM construction method based on RGB-D semantic segmentation according to claim 1, characterized in that, Acquire sensor data from an RGB-D camera, and acquire multiple adjacent frames of the sensor data, including: The parameters of the RGB-D camera are calibrated, and color and depth images are acquired. Acquire multiple adjacent frames from the color image and the depth image.

3. The SLAM construction method based on RGB-D semantic segmentation according to claim 1, characterized in that, ORB feature extraction is performed on the multiple adjacent frames to complete motion consistency detection and obtain feature point matching relationships, including: The pose of the first keyframe is determined as the reference world coordinate system, and the spatial points of the first keyframe are used as map points to obtain the initial map. The feature point matching relationship between the multiple adjacent frames is obtained using an optical flow pyramid.

4. The SLAM construction method based on RGB-D semantic segmentation according to claim 1, characterized in that, Once the visual odometry completes tracking of adjacent frames and obtains multiple keyframes, the multiple keyframes are used to determine the entry and exit loop threads, including: Track the local map, update and optimize the local map, and determine the multiple keyframes; Candidate keyframes are identified from the plurality of keyframes, and loop closure verification is performed on the candidate keyframes to obtain candidate loop closure frames; The candidate loopback frames are corrected to obtain updated keyframes.

5. The SLAM construction method based on RGB-D semantic segmentation according to claim 1, characterized in that, In the SLAM system framework, an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion is used for training to obtain an initial semantic segmentation model, including: The RGB image and the depth image are channel aligned to obtain the aligned RGB image and HHA image; The pair of feature maps output from each stage are input into the Transformer unit for calibration and fusion to obtain a fused feature map. The fused feature map is then added to the pair of feature maps and the average is taken to obtain the processed fused feature map. The final processed and fused feature map is input into the Separation and Aggregation Gate (SA-Gate) unit to obtain the output feature map, which is then merged with the output of the first Transformer module to obtain the initial semantic segmentation model.

6. The SLAM construction method based on RGB-D semantic segmentation according to claim 1, characterized in that, The initial semantic segmentation model is pre-trained with a deep noise decoder to obtain a SLAM semantic segmentation model, including: The encoder was pre-trained for classification using the ImageNet dataset, and the encoder parameters were frozen. Depth noise is pre-trained by a preset objective function; Pre-training is performed on the DREDS dataset to minimize the error of the preset objective function; Unfreeze the encoder parameters and fine-tune them using the cross-entropy loss function to obtain the SLAM semantic segmentation model.

7. A SLAM construction system based on RGB-D semantic segmentation, based on the SLAM construction method based on RGB-D semantic segmentation as described in any one of claims 1 to 6, characterized in that, include: The building module is used to construct the framework of a simultaneous localization and mapping (SLAM) system based on color depth RGB-D. The training module is used to train an RGB-D semantic segmentation model based on Transformer dual-branch multi-level attention fusion in the SLAM system framework to obtain an initial semantic segmentation model. The adjustment module is used to pre-train the initial semantic segmentation model with a deep noise decoder to obtain a SLAM semantic segmentation model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the SLAM construction method based on RGB-D semantic segmentation as described in any one of claims 1 to 6.

Images