A real-time simultaneous localization and mapping method based on STDC semantic segmentation network
By combining the STDC semantic segmentation network and the ORB-SLAM3 algorithm, the problems of low localization accuracy and slow processing speed of visual SLAM systems in dynamic environments are solved, achieving high-precision and real-time simultaneous localization and mapping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2022-06-17
- Publication Date
- 2026-05-15
AI Technical Summary
Existing visual SLAM systems suffer from low positioning accuracy and slow processing speed in complex dynamic environments, making it difficult to meet real-time requirements. In particular, the impact of highly dynamic targets on the system's robustness and accuracy has not been effectively addressed.
The STDC semantic segmentation network is used to remove dynamic feature points, combined with the ORB-SLAM3 algorithm for localization and navigation. The STDC2-Seg75 network is used for encoding and training loss to guide the low-level learning of spatial details. The manual annotation method is combined to improve the processing speed and accuracy.
This improves the positioning accuracy and robustness of the SLAM system in dynamic environments, meets real-time requirements, and experiments show that the positioning error is small and the time consumption is low.
Smart Images
Figure CN115187614B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous navigation for intelligent robots, and in particular, it is a real-time simultaneous localization and mapping method based on STDC semantic segmentation network. Background Technology
[0002] In recent years, the rapid development of technologies such as big data and deep learning has led to more mature technologies related to mobile robots, greatly facilitating people's lives. Mobile robots involve multiple disciplines, including kinematics, dynamics, control theory, computer science, mechanical principles, and sensor technology, and are one of the most active areas of scientific and technological research today. Simultaneous Localization and Mapping (SLAM) refers to a mobile robot simultaneously locating itself and constructing a map of its surrounding environment without any prior knowledge of the environment.
[0003] Although visual SLAM technology has made great strides, some problems still need to be addressed. For example, when mobile robots operate in complex dynamic environments, highly dynamic targets can cause significant inconsistencies between adjacent frames, severely impacting the robustness of the SLAM system. Furthermore, many existing visual SLAM algorithms assume a static external environment, neglecting the influence of dynamic objects on the SLAM system. The presence of dynamic objects in the environment affects the robustness of the SLAM system, reduces its localization accuracy, and can even lead to tracking failure.
[0004] Combining deep learning with SLAM systems can effectively reduce the impact of dynamic targets on system localization and mapping. However, high-precision semantic segmentation networks consume a lot of time when processing images, which does not meet the real-time requirements of applications.
[0005] CN113516664A discloses a visual SLAM method based on semantic segmentation of dynamic points. This patent employs a Mask R-CNN segmentation network and a multi-view geometric constraint algorithm to remove dynamic feature points, improving the system's localization accuracy and robustness in dynamic environments. However, the Mask R-CNN network used in the above method consumes a significant amount of time during segmentation, reducing the system's processing speed.
[0006] CN112435262A discloses a dynamic environment information detection method based on semantic segmentation networks and multi-view geometry. This patent employs a lightweight semantic segmentation network, FcHarDnet, and a multi-view geometric constraint algorithm to remove dynamic feature points, thereby improving system robustness. However, the segmentation speed of the semantic segmentation network used in the above method still does not meet real-time requirements, and the multi-view geometric constraint algorithm is time-consuming, resulting in the system failing to meet real-time requirements. Summary of the Invention
[0007] This invention aims to solve the problems of the prior art. It proposes a real-time simultaneous localization and mapping method based on the STDC semantic segmentation network. The technical solution of this invention is as follows:
[0008] A real-time simultaneous localization and mapping method based on STDC semantic segmentation network includes the following steps:
[0009] S1 uses an RGB-D depth camera to directly capture RGB image information and image depth information;
[0010] S2, using the ORB corner detection and feature description algorithm to extract feature points from RGB image information and image depth information;
[0011] S3 utilizes the STDC short-term dense connection network to obtain image semantic information and uses the semantic information to remove dynamic feature points;
[0012] S4. The remaining feature points are located and navigated using the ORB-SLAM3 algorithm. ORB-SLAM3 is a real-time SLAM algorithm based on feature points, which includes a tracking thread, a local mapping thread, and a loop closure detection thread.
[0013] Furthermore, step S2 utilizes the ORB algorithm to extract feature points; specifically, it includes:
[0014] First, the FAST algorithm is used to obtain FAST key points. The steps are as follows: 1. Traverse each pixel in the image to be extracted and calculate the gray value I of that pixel. p 2. Set a threshold T (T is generally taken as I). p 30%); 3. Select 16 pixels on a circle with the center of the pixel and a radius of 3; 4. If 12 consecutive pixels among these 16 pixels have a grayscale value greater than I... p +T or less than I p -T indicates that the pixel is a FAST key point.
[0015] Then, using the gray-level centroid of the image patch to add a rotation description for the feature points, the moments of image patch A are defined as:
[0016]
[0017]
[0018] In the formula: I(x,y) is the gray value at pixel (x,y) in the image, m x m y These represent the moments in the horizontal direction and the moments in the vertical direction, respectively.
[0019] The direction of the feature point is defined as:
[0020]
[0021] Finally, the BRIEF descriptor is used to describe the feature point; N pairs of points are randomly selected around feature point P, and their gray values are compared:
[0022]
[0023] In the formula: P(x) and P(y) are the gray values at points x and y, respectively;
[0024] Therefore, the BRIEF descriptor is represented as:
[0025]
[0026] In the formula: n represents the nth descriptor of the feature point;
[0027] Furthermore, the STDC semantic segmentation network in step S3 specifically involves selecting the STDC2-Seg75 network, which uses STDC modules for encoding and utilizes training loss to guide the low-level learning of spatial details.
[0028] Furthermore, the STDC module specifically involves acquiring feature maps of different receptive fields using four convolutional layers and an activation layer. These feature maps are then cascaded and fused. The four convolutional layers employ a 2D convolution algorithm with a stride of 1 and kernel sizes of {1, 3, 3, 3}. The activation layer uses the ReLU function, with the following formula:
[0029] R(x) = max(0,x) (13)
[0030] In the formula: x is the input, and R(x) is the output after passing through the ReLU unit.
[0031] Furthermore, the training loss of the STDC semantic segmentation network specifically employs a combination of cross-entropy and binary classification segmentation loss:
[0032] L d =L dice (p d ,g d )+L bce (p d ,g d (14)
[0033] In the formula: L d L represents the training detail loss. dice L represents the binary classification loss. bce p represents the cross-entropy loss.d To indicate the details of the prediction, g d The corresponding ground truth values are represented, and the model is trained using a loss function to segment the input image and obtain a semantic segmentation map.
[0034] Furthermore, the removal of dynamic feature points using semantic information specifically involves: first, marking highly dynamic targets in the semantic segmentation map; then, using the marked semantic segmentation map as a mask; and finally, using the mask and feature point map to remove dynamic feature points.
[0035] Furthermore, step S4 uses the ORB-SLAM3 algorithm to locate and navigate the remaining feature points, specifically including:
[0036] Tracking thread: Finds and matches local map feature points, uses the Bundle Adjustment (BA) algorithm to minimize reprojection error, and locates the camera pose for each frame.
[0037] Local mapping thread: Utilizes local BA algorithm to optimize camera pose and feature point cloud.
[0038] Loop closure detection thread: Detects loop closures and eliminates accumulated drift errors through pose graph optimization. After pose graph optimization, a global BA algorithm thread is started to calculate the optimal structure and motion results for the entire system.
[0039] The advantages and beneficial effects of this invention are as follows:
[0040] This invention addresses the problems of low localization accuracy and high processing time of SLAM systems integrating semantic segmentation networks in dynamic environments. It proposes a real-time simultaneous localization and mapping (SMR) method based on the STDC semantic segmentation network. On one hand, this invention employs the STDC short-term dense connection network as the system's semantic segmentation network, as shown in steps 3 and 4. Training loss guides the lower layers to learn spatial details, and the guidance step is removed during the prediction phase, improving both network accuracy and processing speed. On the other hand, due to the high time consumption of multi-view geometric constraints, this invention uses direct manual annotation, as shown in step 6, to manually label highly dynamic objects such as people, animals, and cars. This method significantly improves processing speed and accuracy.
[0041] Experiments on the public dataset TUM demonstrate that the real-time simultaneous localization and mapping (SML) method incorporating the STDC semantic segmentation network exhibits good localization accuracy and robustness, while also exhibiting lower latency in the semantic segmentation thread, meeting real-time requirements. To more clearly compare the localization accuracy and latency of SLAM systems based on different semantic segmentation networks, the proposed system is compared with other SLAM systems based on semantic segmentation networks. The proposed system demonstrates superior performance with smaller localization errors and lower latency. Attached Figure Description
[0042] Figure 1 This invention provides a preferred embodiment of a real-time simultaneous localization and mapping method framework based on the STDC semantic segmentation network;
[0043] Figure 2 This is the STDC semantic segmentation network framework. Detailed Implementation
[0044] The technical solutions of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.
[0045] The technical solution of the present invention to solve the above-mentioned technical problems is:
[0046] like Figure 1 As shown, a real-time simultaneous localization and mapping (SLAM) method based on the STDC semantic segmentation network is presented. This method utilizes the STDC semantic segmentation network to segment dynamic targets in the environment in real time, eliminating the impact of dynamic objects on the stability of the SLAM system in the real environment, and improving the localization accuracy and robustness of SLAM. The specific steps include:
[0047] S1 uses an RGB-D depth camera to directly capture RGB image information and image depth information.
[0048] S2, preprocess the input image information, extracting ORB feature points from the RGB image information and image depth information using the ORB algorithm. First, obtain FAST key points using the FAST algorithm. The steps are: 1. Traverse each pixel in the image to be extracted and calculate the gray value I of that pixel. p 2. Set a threshold T (T is generally taken as I). p 30%); 3. Select 16 pixels on a circle with the center of the pixel and a radius of 3; 4. If 12 consecutive pixels among these 16 pixels have a grayscale value greater than I... p +T or less than I p -T indicates that the pixel is a FAST key point.
[0049] Then, using the gray-level centroid of the image patch to add a rotation description for the feature points, the moments of image patch A are defined as:
[0050]
[0051]
[0052] In the formula: I(x,y) is the gray value at pixel (x,y) in the image, m x m yThese represent the moments in the horizontal direction and the moments in the vertical direction, respectively.
[0053] The direction of the feature point is defined as:
[0054]
[0055] Finally, the BRIEF descriptor is used to describe the feature point; N pairs of points are randomly selected around feature point P, and their gray values are compared:
[0056]
[0057] In the formula: P(x) and P(y) are the gray values at points x and y, respectively;
[0058] Therefore, the BRIEF descriptor is represented as:
[0059]
[0060] In the formula: n represents the nth descriptor of the feature point;
[0061] S3. The STDC semantic segmentation network is used to obtain image semantic information. The STDC2-Seg75 network is selected, which uses STDC modules for encoding. Four convolutional layers and an activation layer are used to obtain feature maps of different receptive fields. These feature maps are then cascaded and fused. The four convolutional layers use a 2D convolution algorithm with a stride of 1 and kernel sizes of {1, 3, 3, 3}. The activation layer uses the ReLU function, with the following formula:
[0062] R(x) = max(0,x) (20)
[0063] In the formula: x is the input, and R(x) is the output after passing through the ReLU unit.
[0064] A combination of cross-entropy and binary classification segmentation loss is used:
[0065] L d =L dice (p d ,g d )+L bce (p d ,g d ) (twenty one)
[0066] In the formula: L d L represents the training detail loss. dice L represents the binary classification loss. bce p represents the cross-entropy loss. d To indicate the details of the prediction, g dThis represents the ground truth value for the corresponding details. A model is trained using a loss function to segment the input image, obtaining a semantic segmentation map. Then, highly dynamic targets in the semantic segmentation map are labeled, such as people, animals, and cars. The labeled semantic segmentation map is then used as a mask. Finally, dynamic feature points are removed using the mask and the feature point map.
[0067] S4. The remaining feature points are located and navigated using the ORB-SLAM3 algorithm to obtain the SLAM system trajectory tracking map and the environmental point cloud map.
[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0069] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.
Claims
1. A real-time simultaneous localization and mapping method based on STDC semantic segmentation network, characterized in that, Includes the following steps: S1 uses an RGB-D depth camera to directly capture RGB image information and image depth information; S2, using the ORB corner detection and feature description algorithm to extract feature points from RGB image information and image depth information; S3 utilizes the STDC semantic segmentation network to obtain image semantic information and uses the semantic information to remove dynamic feature points; S4. The remaining feature points are located and navigated using the ORB-SLAM3 algorithm. ORB-SLAM3 is a real-time SLAM algorithm based on feature points. The algorithm includes a tracking thread, a local mapping thread, and a loop closure detection thread. The STDC semantic segmentation network in step S3 specifically involves selecting the STDC2-Seg75 network, which uses STDC modules for encoding and utilizes training loss to guide the low-level learning of spatial details. The STDC module specifically utilizes four convolutional layers and an activation layer to obtain feature maps of different receptive fields. These feature maps are then cascaded and fused. The four convolutional layers employ a 2D convolution algorithm with a stride of 1 and kernel sizes of {1, 3, 3, 3}. The activation layer uses the ReLU function, with the following formula: （6） In the formula: x is the input, It is the output after passing through the ReLU unit; The process of removing dynamic feature points from semantic information is as follows: First, highly dynamic targets in the semantic segmentation map are marked; then, the marked semantic segmentation map is used as a mask; finally, dynamic feature points are removed using the mask and the feature point map. Step S4 involves using the ORB-SLAM3 algorithm to locate and navigate the remaining feature points, specifically including: Tracking thread: Finds and matches local map feature points, uses bundle adjustment optimization (BA) algorithm to minimize reprojection error, and locates the camera pose for each frame; Local mapping thread: Utilizes local BA algorithm to optimize camera pose and feature point cloud; Loop closure detection thread: detects loop closures and eliminates accumulated drift errors through pose graph optimization; after pose graph optimization, it starts the global BA algorithm thread to calculate the optimal structure and motion results of the entire system.
2. The real-time simultaneous localization and mapping method based on STDC semantic segmentation network according to claim 1, characterized in that, Step S2 extracts feature points using the ORB algorithm; specifically, it includes: First, the FAST algorithm is used to obtain FAST key points. The steps are as follows:
1. Traverse each pixel in the image to be extracted and calculate the gray value I of that pixel. p 2. Set a threshold T, where T is equal to I. p 30%; 3. Select 16 pixels on a circle with the center of the pixel and a radius of 3; 4. If 12 consecutive pixels among these 16 pixels have a grayscale value greater than I... p + T or less than I p - T, determines that the pixel is a FAST key point; Then, using the gray-level centroid of the image patch to add a rotation description for the feature points, the moments of image patch A are defined as: （1） （2） In the formula: For image pixels grayscale value at that location , These represent the moments in the horizontal direction and the moments in the vertical direction, respectively. The direction of the feature point is defined as: （3） Finally, the BRIEF descriptor is used to describe the feature point; N pairs of points are randomly selected around feature point P, and their gray values are compared: （4） In the formula: P(x) and P(y) are the gray values at points x and y, respectively; Therefore, the BRIEF descriptor is represented as: （5） In the formula: n represents the nth descriptor of the feature point.
3. The real-time simultaneous localization and mapping method based on STDC semantic segmentation network according to claim 1, characterized in that, The training loss of the STDC semantic segmentation network is specifically achieved by combining cross-entropy and binary classification segmentation loss: （7） In the formula: L d L represents the training detail loss. dice L represents the binary classification loss. bce This represents cross-entropy loss. Indicates prediction details, The corresponding ground truth values are represented, and the model is trained using a loss function to segment the input image and obtain a semantic segmentation map.