A local refinement mapping system and method based on slam and semantic segmentation
By combining the semantic segmentation techniques of ORB-SLAM2 and DANet models, the SLAM mapping process is made more refined and efficient. This solves the problem of environmental influence during the secondary mapping process of traditional SLAM technology, and improves the timeliness and accuracy of mapping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-03-29
- Publication Date
- 2026-07-28
AI Technical Summary
Existing SLAM technology is susceptible to environmental conditions during the secondary mapping process, resulting in long mapping time and wasted computing resources. Furthermore, traditional methods have insufficient object recognition capabilities when mapping in a blurred background.
The ORB-SLAM2 algorithm is combined with the DANet model based on the FCN network. Semantic segmentation technology is used to extract features and select scenes for local targets. ORB-SLAM2 technology is used for refined recognition and map optimization. Channel attention module and spatial attention module are used for feature recognition and fusion to narrow the feature range and improve the timeliness and accuracy of map building.
Through the initial identification and processing by the semantic segmentation module, external environmental interference is reduced, the timeliness and local accuracy of mapping are improved, the applicable scenarios are more extensive, and the waste of computing resources is reduced.
Smart Images
Figure CN116772820B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, specifically relating to a local refinement mapping system and method based on SLAM and semantic segmentation. Background Technology
[0002] SLAM (Simultaneous Localization and Mapping) describes a scenario where a robot starts moving from an unknown location in an unknown environment, performs self-localization based on its position and a map during movement, and simultaneously builds an incremental map based on its self-localization, achieving autonomous localization and navigation. SLAM technology has undergone continuous development, iteration, and updates, with ongoing improvements to its sensors, algorithms, and technical framework. It is a primary means of achieving self-pose evaluation and virtual image feedback, and constructing effective interaction between humans and virtual content. SLAM technology mainly includes visual SLAM and laser SLAM; this invention focuses on visual SLAM technology.
[0003] Traditional SLAM is easily affected by environmental conditions such as weather, lighting, and geographical location. Traditional binocular vision SLAM requires extremely high accuracy in single-shot modeling. Secondary mapping in SLAM is more aligned with user needs in many application scenarios, but secondary modeling technology based on SLAM is not yet mature in the current market. Secondary modeling technology means: first, a preliminary fuzzy identification of the target environment is performed, followed by a secondary, refined model of the target scene or object.
[0004] There are some related studies, such as patent application CN110827305A, which discloses a tightly coupled method for semantic segmentation and visual SLAM for dynamic environments. The semantic segmentation module outputs pixel-level semantic segmentation results and depth recovery results of the image, and then passes the original image, the semantic label of each pixel, and the corresponding depth map to the visual SLAM module for high-precision mapping. This method is designed to narrow down the mapping range by applying semantic segmentation technology in macro scenes, and then complete the fine mapping task for a specific scene segment. However, the noise monitoring algorithm of this method is relatively complex, and the mapping time is relatively longer.
[0005] Jingwen Wang, Martin Runz, and Lourdes Agapito proposed an object-oriented SLAM system, DSP-SLAM (Object OrientedSLAM with Deep Shape Priors). This system constructs a rich and accurate joint map of dense 3D models for foreground objects and uses sparse landmarks to represent the background environment. DSP-SLAM takes the 3D point cloud reconstructed by a feature-based SLAM system as input and enables it to enhance its sparse map by densely reconstructing detected objects. It detects objects through semantic instance segmentation and estimates the shape and pose of the target using a novel second-order optimization algorithm with depth shape embeddings of specific categories as priors. However, due to its highly global mapping requirements and extremely high precision, it easily leads to wasted computational resources during model evaluation, resulting in long computation times. Furthermore, it lacks the ability to recognize objects when mapping in a blurred background. Summary of the Invention
[0006] To address the aforementioned issues, this invention provides a local refinement mapping system and method based on SLAM and semantic segmentation. It combines the ORB-SLAM2 algorithm with the DANet model based on the FCN network. Semantic segmentation is used to extract features and select scenes from local target maps, resulting in strong recognition capabilities. For the selected target objects, ORB-SLAM2 technology is further applied for refined recognition, mapping optimization, loop closure detection, and real-time map construction to achieve the mapping task, improving both the timeliness and local accuracy of mapping.
[0007] The present invention discloses a local refinement mapping system based on SLAM and semantic segmentation, the system comprising an image acquisition module, a semantic segmentation module, a front-end visual odometry module, a back-end optimization module, and a loop closure detection module;
[0008] The image acquisition module acquires scene images using a binocular camera;
[0009] The semantic segmentation module adopts the DANet model based on the FCN network framework. It interacts with contextual information to quickly extract semantic features from images captured by the stereo camera. It also uses convolutional layers for feature fusion and convolutional mapping to perform pixel-level classification of the target scene. Pixels of the same class are predicted to be of the same class. The classification results are stored in the dataset to achieve semantic segmentation of the target scene image.
[0010] The front-end visual odometry module extracts feature points from the semantically segmented target scene image and matches them with the same feature points in other images using the FCN network framework. Based on the pose estimation algorithm, it calculates the pose of the binocular camera and the spatial position of the feature points.
[0011] The backend optimization module selects keyframes from the global camera pose based on the ORB-SLAM2 algorithm, establishes global constraints using the relationships between keyframes, and constructs a pose graph.
[0012] The closed-loop detection module optimizes the pose map based on the camera pose and the error of the pose map to obtain a globally consistent pose estimate.
[0013] Furthermore, the semantic segmentation module adopts the DANet model based on the FCN network framework. The backbone of the DANet model adopts the ResNet residual network. The ResNet residual network is used to perform preliminary convolution processing on two scene images captured consecutively from the stereo camera, that is, to perform convolution, pooling and full connection on the input image to generate the original feature matrix.
[0014] The initialized image is fed into the channel attention module and spatial attention module of the DANet model itself. The two attention modules are connected to the end of the residual network ResNet. The channel attention module adopts a self-attention mechanism, that is, it captures the channel dependency between two consecutive captured scenes through the channel attention matrix specified in the channel mechanism, and updates the generated channel map in real time using the weighted sum of all channel maps. The spatial attention module introduces a similar self-attention mechanism, using the spatial attention matrix specified in the spatial attention mechanism to capture the spatial dependency between any two positions, and updates it by aggregating features at all positions through weighted summation. The summation weight of the spatial attention module is determined by the feature similarity between two positions of an object between two consecutively captured images. The outputs of the channel attention module and the spatial attention module are fused through a convolutional layer, and then passed through another convolutional layer to generate the final prediction map, completing the feature recognition task. The prediction results are stored in the CamVid dataset, and finally semantic segmentation is achieved.
[0015] Furthermore, the front-end visual odometry module extracts the input layer image through the encoder layer in the FCN network framework, and downsamples the upper-layer feature image through convolution calculation; the decoder layer recovers the feature image based on the inverse decoding method, refines the feature image through convolution calculation, and upsamples and merges the multi-scale feature images; then, the camera rotation matrix and translation matrix are obtained by applying the projection coordinate minimization and true matching coordinate method, and the depth of field is calculated on the image. The feature data stored in the semantic segmentation module through keyframes is calculated and trained by the PnP algorithm, and mapped to the two-dimensional image by the PnP algorithm to determine the current position.
[0016] Furthermore, a keyframe is the state of the target image captured at a certain moment. Global constraints are established using the relationships between keyframes. Specifically, the global pose of the keyframe is used as the node of the graph, and the relative pose error between keyframes is used as the weight of the edge of the graph. By minimizing the sum of the weights of all edges in the entire graph, the value of each graph node is optimized, the relationships between keyframes are derived, and the pose graph is constructed.
[0017] Furthermore, in the loop closure detection process, considering the scenario where errors accumulate in the estimated pose map and 3D map points due to camera motion, the system saves detectable loops during camera motion. Based on the loop closures, iterative calculations are performed on the pose map generated from keyframes to achieve optimization, ultimately obtaining a globally consistent pose estimate. The loop closure detection module adopts the Bag-of-Words (BoW) algorithm based on the ORB-SLAM2 algorithm. The BoW algorithm measures the similarity between two consecutive scene images captured by a stereo camera. A bag-of-words model is constructed, and a set of feature descriptors is generated by extracting feature points. Feature descriptors with similar points are clustered to obtain K bags of words. Finally, the bags of words are stored for subsequent search matching and loop closure detection.
[0018] A local thinning mapping method based on SLAM and semantic segmentation, comprising the following steps:
[0019] Step 1, Image Acquisition: Acquire scene images using a stereo camera;
[0020] Step 2, Target Extraction: The acquired scene image is put into a ResNet network with a pre-trained set to obtain the original feature matrix. The target is extracted in the convolutional layer of the DANet network model and input into the corresponding channel attention module or spatial attention module.
[0021] Step 3, Feature Fusion: Perform matrix multiplication on the original feature matrix and the channel attention matrix and spatial attention matrix obtained according to the prescribed algorithm in the channel attention module and spatial attention module respectively, and then use the summation operation of row elements to realize the feature fusion of semantic information;
[0022] Step 4, Convolutional Mapping: The data stored in the semantic segmentation module is fused with the elements of the channel attention module and the spatial attention module through a new convolutional layer, and the image information obtained by semantic segmentation is saved in the CamVid dataset.
[0023] Step 5, Feature Recognition: After determining the vector direction of feature points using the gray-scale centroid method, the image information stored in the CamVid dataset by the semantic segmentation module is processed by Gaussian pyramid to finally extract the feature points.
[0024] Step 6, Feature Matching: The selected feature points are fed into the input layer. The input layer image is extracted through the encoder layer in the FCN network framework. The upper layer feature image is downsampled through convolution calculation. The decoder layer recovers the feature image based on the inverse decoding method. The feature image is thinned through convolution calculation, and the multi-scale feature images are upsampled and merged. Then, the camera rotation matrix and translation matrix are obtained by applying the projection coordinate minimization and true matching coordinate method.
[0025] Step 7, Pose Estimation: Use a binocular camera to calculate the depth of field of the selected target scene for fine recognition. Use the PnP algorithm to calculate and train the feature data stored in the keyframes in the semantic segmentation module. Map the PnP algorithm onto the generated two-dimensional image to determine the current position.
[0026] Step 8, Pose Optimization: Using BA optimization based on ORM-SLAM2, the light rays reflected from each feature point are optimally adjusted according to the camera pose and the spatial position of the feature points through keyframe recognition and extraction tasks in the back-end optimization module. Finally, the light beam converges to the camera optical center, so that the light beam satisfies the constraint plane. By applying the least squares method to the depth data acquired by the front end, the error between the original captured image and the generated 2D image is optimized, and refined map construction information is obtained.
[0027] Step 9, Loop Closure Detection: Use Bag-of-Words (BoW) to measure the similarity between two images, specifically:
[0028] First, feature points are extracted from images of different categories to generate corresponding feature descriptors. In ORB-SLAM2, the vocabulary of BoW is the Rotated BRIEF, the descriptor operator of ORB features. Then, all feature descriptor sets are clustered using a clustering algorithm, grouping semantically similar words into one class and using the mean of all points in the class to represent that class, thus constructing a bag of words containing K words. Finally, the bag of words is organized into a multidimensional tree or hash table to facilitate subsequent search and matching, completing the keyframe selection for the tracking module. After determining that a loop closure has been detected, BoW matching is performed, and then the Sim3 similarity transformation is solved to integrate the loop closure into the entire trajectory, thereby achieving the pose estimation task.
[0029] Step 10, Image Presentation: Output the map construction information from the SLAM mapping.
[0030] The beneficial effects of this invention are as follows: Based on SLAM, this invention integrates semantic segmentation algorithms from computer vision. Through a semantic segmentation module, it achieves semantic segmentation functionality on top of object detection, which helps complete object recognition tasks in sparse scene maps, making the mapping process more intelligent and faster. By introducing the DANet model, a semantic segmentation technology from deep learning, for optimized computation, preliminary scene recognition is first completed through semantic segmentation. Then, the feature recognition and extraction are prioritized using channel attention and spatial attention modules, narrowing the feature range, saving computation time and cost, and improving the timeliness and local accuracy of mapping. The secondary refined SLAM mapping technology based on semantic segmentation reduces interference from the external environment through initial image recognition and processing, making intelligent mapping applicable to a wider range of scenarios. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of the system operation of the present invention;
[0032] Figure 2 This is a schematic diagram of the system described in this invention;
[0033] Figure 3 This is a schematic diagram of the semantic segmentation module structure;
[0034] Figure 4 This is a flowchart of the method described in this invention. Detailed Implementation
[0035] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0036] like Figure 1 As shown, the local refinement mapping system based on SLAM and semantic segmentation of the present invention includes: an image acquisition module, a semantic segmentation module, a front-end visual odometry module, a back-end optimization module, and a loop closure detection module.
[0037] The image acquisition module acquires scene images using a binocular camera;
[0038] The semantic segmentation module is a DANet model based on the FCN network framework, which processes scene images to achieve semantic segmentation and target recognition;
[0039] The front-end visual odometry module will match the feature points extracted from the semantically segmented target scene with the same feature points in other images, and calculate the camera pose and the spatial position of the feature points based on the pose estimation algorithm.
[0040] The backend optimization module selects keyframes from the global camera pose, establishes global constraints using the relationships of keyframe quality inspection, and constructs a pose graph.
[0041] The closed-loop detection module optimizes the pose map based on the camera pose and the error of the pose map to obtain a globally consistent pose estimate.
[0042] (1) Image acquisition module:
[0043] Select a stereo camera for scene image acquisition. In the task scenario, since stereo cameras generally include the calibration of the intrinsic parameters and distortion coefficients of each camera and the calibration of the relative positions between the two cameras, stereo calibration can solve the problem of acquisition accuracy caused by parallax and depth, and accurately complete the image acquisition task.
[0044] (2) Semantic segmentation module, such as Figure 2 As shown:
[0045] Semantic segmentation technology is introduced based on image acquisition. In this module, the DANet model based on the FCN network framework is selected.
[0046] This paper introduces spatial attention and communication attention modules into the existing ReNet residual network. These modules interact through contextual information. Each module calculates its own attention matrix, multiplies the original matrix by the feature matrix, and performs an element-wise summation to achieve rapid semantic feature extraction. A convolutional layer transforms the outputs of the two attention modules and performs element-wise summation to complete feature fusion. Finally, another convolutional layer generates the final predicted map, realizing the semantic segmentation map task and facilitating subsequent local map feature extraction.
[0047] (3) Front-end visual mileage module:
[0048] The main task of front-end visual mileage estimation is to perform feature matching between images and targets based on information from neighboring images, estimate a rough camera pose, and provide a better initial pose value for the back-end. In this solution, the steps involved are to extract feature points from the semantically segmented target scene, match them with the same feature points in other images, and calculate the camera pose and the spatial location of the feature points using a pose estimation algorithm, thus providing the basic requirements for the back-end mapping task.
[0049] (4) Backend optimization module:
[0050] The backend optimization process involves selecting keyframes from the global (camera motion) context and using the relationships between these keyframes to establish global constraints with a larger temporal and spatial span that need to be satisfied simultaneously, thus completing a Bundle Adjustment (minimizing reprojection error). Therefore, this is solved by building and optimizing a pose graph. The pose graph uses the global poses of the keyframes as nodes and the relative pose errors between keyframes as edge weights. By minimizing the sum of the weights of all edges in the entire graph, the value of each graph node is optimized, achieving the goal of graph construction.
[0051] (5) Closed-loop detection module:
[0052] In the loop closure detection process, there is a scenario where camera motion causes error accumulation in the estimated camera pose and 3D map points. By saving the detectable loops during camera motion and optimizing all results based on the loop closures, a globally consistent pose estimate can be obtained. This module adopts the Bag-of-Words (BoW) algorithm based on the ORB-SLAM2 algorithm. The BoW algorithm measures the similarity between two images, constructs a bag-of-words model, generates a set of feature descriptors by extracting feature points, clusters feature descriptors with similar points, and finally obtains K bags of words. Finally, the bags of words are stored for subsequent search matching and loop closure detection.
[0053] A local thinning mapping method based on SLAM and semantic segmentation, the specific steps of which are as follows:
[0054] Step 1, Image Acquisition: Use a binocular camera to acquire images of a wide range of scenes, which will facilitate subsequent semantic segmentation.
[0055] Step 2, Target Extraction: The image acquired in Step 1 is fed into a ResNet network with a pre-trained set to obtain the original feature matrix. The target is extracted in the convolutional layer of the DANet network model and input into the corresponding channel attention module or spatial attention module.
[0056] Step 3, Feature Fusion: Perform matrix multiplication on the original feature matrix from Step 2 and the attention matrix calculated by training in the channel attention module and the spatial attention module respectively, that is, realize the connection operation between the layers of the neural network, and then use the sum of row elements to realize the feature fusion of semantic information.
[0057] Step 4, Convolutional Mapping: The data stored in the semantic segmentation module is fused with elements from the channel attention module and the spatial attention module through a new convolutional layer, and the image information obtained from semantic segmentation is saved in the CamVid dataset;
[0058] Step 5, Feature Recognition: ORB features consist of key points and descriptors. This step uses the information obtained in Step 4 to determine the vector direction of the feature points using the gray-scale centroid method, and then employs Gaussian pyramid processing to finally achieve the feature point extraction task.
[0059] Step 6, Feature Matching: The feature point pairs selected in Step 2 are fed into the input layer. The input layer image is extracted by encoding through the encoder layer. The upper layer feature image is downsampled by convolution calculation. The decoder layer mainly recovers the feature image based on the inverse decoding method. The feature image is thinned by convolution calculation, and the multi-scale feature images are upsampled and merged. Then, the camera rotation matrix and translation matrix are obtained by applying the method of minimizing the projected coordinates and matching the coordinates.
[0060] Step 7, Pose Estimation: Use a binocular camera to perform depth calculation on the selected target scene for fine recognition. Use the PnP algorithm to calculate and train the feature data stored in the keyframes in the semantic segmentation module. Map the PnP algorithm onto the two-dimensional image to determine the current position.
[0061] Step 8, Pose Optimization: Using BA optimization based on ORM-SLAM2, the light rays reflected from each feature point are optimally adjusted by the camera pose and spatial position of the feature points through the keyframe recognition and extraction task in the backend optimization module, and finally converge to the camera optical center so that the light beam satisfies the constraint plane. By applying the least squares method to the depth data acquired by the front end, the error between the original captured image and the generated 2D image is optimized, and refined map construction information is obtained.
[0062] Step 9, Loop Closure Detection: Bag-of-Words (BoW) is used to measure the similarity between two images. This involves three main steps: First, feature points are extracted from images of different categories, and corresponding feature descriptors are generated. In ORB-SLAM2, the BoW vocabulary is the Rotated BRIEF, the descriptor operator for ORB features. Then, all feature descriptors are clustered using a clustering algorithm, grouping semantically similar words into one class and using the mean of all points in that class to represent that class, ultimately constructing a bag-of-words containing K words. Finally, the bag-of-words is organized into a multidimensional tree or hash table to facilitate subsequent search matching and loop closure detection, completing the keyframe selection for the tracking module. After detecting a loop, BoW matching is performed, and then the Sim3 similarity transformation is solved to integrate the loop into the entire trajectory, thus achieving the pose estimation task.
[0063] Step 10, Image Presentation: Output the map construction information from the SLAM mapping.
[0064] like Figure 4As shown, assuming the equipment is functioning normally and the environment is suitable, without interference from unforeseen factors, the system first uses a binocular camera to capture images of the scene. A microcomputer then processes the images within the device to perform semantic segmentation and target recognition. If image acquisition fails, the system re-enters the command. Based on the user's instructions, the system locates the target elements, extracts the corresponding sections, converts the graphic features into data information, and stores the obtained data in a pre-established database for later use.
[0065] At this point, the information stored in the database becomes the applicable elements for sparse mapping. The stored images undergo grayscale processing and other calculations and training, enabling local map drawing for specific geographic modules. Based on the user's target instructions, i.e., the information elements stored in the database, the binocular camera uses multiple conditions such as depth of field and landmarks to achieve pose estimation, and completes the construction of a displayable, refined map in the backend pose optimization process.
[0066] To detect the accuracy of the mapping, a loop closure detection module is introduced. In loop closure detection, the original image is compared with the mapping model using a bag-of-words model. Clustering algorithm is used to cluster semantically similar words into one class, and the mean of all points in the class is used to represent the class. Finally, the bag-of-words is organized into a multidimensional tree or hash table to facilitate subsequent search matching and loop closure detection.
[0067] Under multiple conditions such as clear target recognition, accurate pose estimation, and error-free loop closure detection, the image is output, and finally, a local fine-grained map based on SLAM and semantic segmentation is completed.
[0068] The above description is merely a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made based on the description and drawings of the present invention are within the protection scope of the present invention.
Claims
1. A local refinement mapping system based on SLAM and semantic segmentation, characterized in that, The system includes an image acquisition module, a semantic segmentation module, a front-end visual odometry module, a back-end optimization module, and a loop closure detection module. The image acquisition module acquires scene images using a binocular camera; The semantic segmentation module adopts the DANet model based on the FCN network framework, which interacts with contextual information to quickly extract semantic features from images captured by the stereo camera; It utilizes convolutional layers for feature fusion and convolutional mapping, performs pixel-level classification of the target scene, predicts pixels of the same class as the same class, stores the classification results in the dataset, and achieves semantic segmentation of the target scene image. The front-end visual odometry module extracts feature points from the semantically segmented target scene image and matches them with the same feature points in other images using the FCN network framework. Based on the pose estimation algorithm, it calculates the pose of the binocular camera and the spatial position of the feature points. The backend optimization module selects keyframes from the global camera pose based on the ORB-SLAM2 algorithm, establishes global constraints using the relationships between keyframes, and constructs a pose graph. The closed-loop detection module optimizes the pose graph based on the camera pose and the error of the pose graph to obtain a globally consistent pose estimate. The semantic segmentation module adopts the DANet model based on the FCN network framework. The backbone of the DANet model adopts the ResNet residual network. The ResNet residual network is used to perform preliminary convolution processing on two scene images captured consecutively from the stereo camera, that is, to perform convolution, pooling and full connection on the input image to generate the original feature matrix. The initialized image is fed into the channel attention module and spatial attention module of the DANet model itself. The two attention modules are connected to the end of the residual network ResNet. The channel attention module adopts a self-attention mechanism, that is, it captures the channel dependency between two consecutive captured scenes through the channel attention matrix specified in the channel mechanism, and updates the generated channel map in real time using the weighted sum of all channel maps. The spatial attention module introduces a similar self-attention mechanism, using the spatial attention matrix specified in the spatial attention mechanism to capture the spatial dependency between any two positions, and updates it by aggregating features at all positions through weighted summation. The summation weight of the spatial attention module is determined by the feature similarity between two positions of an object between two consecutively captured images. The outputs of the channel attention module and the spatial attention module are fused through a convolutional layer, and then passed through another convolutional layer to generate the final prediction map, completing the feature recognition task. The prediction results are stored in the CamVid dataset, and finally semantic segmentation is achieved.
2. The local refinement mapping system based on SLAM and semantic segmentation according to claim 1, characterized in that, The front-end visual odometry module extracts the input layer image through the encoder layer in the FCN network framework and downsamples the upper layer feature image through convolution calculation; The decoder layer recovers the feature image based on the inverse decoding method, refines the feature image through convolution calculation, and upsamples and merges the multi-scale feature images. Then, it uses the method of minimizing projected coordinates and true matching coordinates to obtain the camera rotation matrix and translation matrix, performs depth calculation on the image, and uses the PnP algorithm to calculate and train the feature data stored in the semantic segmentation module through keyframes. The PnP algorithm maps the data to the two-dimensional image to determine the current position.
3. The local refinement mapping system based on SLAM and semantic segmentation according to claim 1, characterized in that, A keyframe is the state of the target image captured at a certain moment. Global constraints are established by using the relationship between keyframes. Specifically, the global pose of the keyframe is used as the node of the graph, and the relative pose error between keyframes is used as the weight of the edge of the graph. By minimizing the sum of the weights of all edges in the entire graph, the value of each graph node is optimized, the relationship between keyframes is obtained, and the pose graph is constructed.
4. The local refinement mapping system based on SLAM and semantic segmentation according to claim 1, characterized in that, In the loop closure detection process, there is a scenario where errors accumulate in the pose map and 3D map points estimated by camera motion. During camera motion, the system saves detectable loops and iteratively calculates the pose map generated based on keyframes to optimize the loop closures, ultimately obtaining a globally consistent pose estimate. The loop closure detection module adopts the Bag-of-Words (BoW) algorithm based on the ORB-SLAM2 algorithm. The BoW algorithm measures the similarity between two consecutive scene images captured by a stereo camera. A bag-of-words model is constructed, and a set of feature descriptors is generated by extracting feature points. Feature descriptors with similar points are clustered to obtain K bags of words. Finally, the bags of words are stored for subsequent search matching and loop closure detection.
5. A local refinement mapping method based on SLAM and semantic segmentation, characterized in that, The method steps are as follows: Step 1, Image Acquisition: Acquire scene images using a stereo camera; Step 2, Target Extraction: The acquired scene image is put into a ResNet network with a pre-trained set to obtain the original feature matrix. The target is extracted in the convolutional layer of the DANet network model and input into the corresponding channel attention module or spatial attention module. Step 3, Feature Fusion: Perform matrix multiplication on the original feature matrix and the channel attention matrix and spatial attention matrix obtained according to the prescribed algorithm in the channel attention module and spatial attention module respectively, and then use the summation operation of row elements to realize the feature fusion of semantic information; Step 4, Convolutional Mapping: The data stored in the semantic segmentation module is fused with elements from the channel attention module and the spatial attention module through a new convolutional layer, and the image information obtained from semantic segmentation is saved in the CamVid dataset; specifically: The channel attention module employs a self-attention mechanism, capturing the channel dependencies between two consecutive captured scenes using the channel attention matrix defined in the channel mechanism, and updating the generated channel map in real time using a weighted sum of all channel maps. The spatial attention module introduces a similar self-attention mechanism, capturing the spatial dependencies between any two positions using the spatial attention matrix defined in the spatial attention mechanism, and updating by aggregating features at all positions through weighted summation. The summation weights of the spatial attention module are determined by the feature similarity between two positions of an object in two consecutively captured images. The outputs of the channel attention module and the spatial attention module are fused through a convolutional layer, and then passed through another convolutional layer to generate the final prediction map, completing the feature recognition task. The prediction results are stored in the CamVid dataset, ultimately achieving semantic segmentation. Step 5, Feature Recognition: After determining the vector direction of feature points using the gray-scale centroid method, the image information stored in the CamVid dataset by the semantic segmentation module is processed by Gaussian pyramid to finally extract the feature points. Step 6, Feature Matching: The selected feature points are fed into the input layer. The input layer image is extracted through the encoder layer in the FCN network framework. The upper layer feature image is downsampled through convolution calculation. The decoder layer recovers the feature image based on the inverse decoding method. The feature image is thinned through convolution calculation, and the multi-scale feature images are upsampled and merged. Then, the camera rotation matrix and translation matrix are obtained by applying the projection coordinate minimization and true matching coordinate method. Step 7, Pose Estimation: Use a binocular camera to calculate the depth of field of the selected target scene for fine recognition. Use the PnP algorithm to calculate and train the feature data stored in the keyframes in the semantic segmentation module. Map the PnP algorithm onto the generated two-dimensional image to determine the current position. Step 8, Pose Optimization: Using ORB-SLAM2-based BA optimization, the light rays reflected from each feature point are optimally adjusted based on the camera pose and the spatial position of the feature points through keyframe recognition and extraction tasks in the backend optimization module. Finally, the light beam converges to the camera optical center, ensuring that the light beam satisfies the constraint plane. By applying the least squares method to the depth data acquired by the front end, the error between the original captured image and the generated 2D image is optimized, thereby obtaining refined map construction information. Step 9, Loop Closure Detection: Use Bag-of-Words (BoW) to measure the similarity between two images, specifically: First, feature points are extracted from images of different categories to generate corresponding feature descriptors. In ORB-SLAM2, the vocabulary of BoW is the Rotated BRIEF, the descriptor operator of ORB features. Then, all feature descriptor sets are clustered using a clustering algorithm, grouping semantically similar words into one class and using the mean of all points in the class to represent that class, ultimately constructing a bag of words containing K words. Finally, the bag of words is organized into a multidimensional tree or hash table to facilitate subsequent search and matching, completing the keyframe selection for the tracking module. After determining that a loop closure has been detected, BoW matching is performed, and then the Sim3 similarity transformation is solved to integrate the loop closure into the entire trajectory, thereby achieving the pose estimation task. Step 10, Image Presentation: Output the map construction information from the SLAM mapping.