A dynamic visual slam method for indoor single-thread semantics and depth estimation

Abstract

This invention discloses a single-threaded dynamic visual SLAM method for semantic and depth estimation, comprising the following steps: S1, a tracking thread; S2, a deep learning thread: the deep learning thread is responsible for providing semantic information to the tracking thread; semantic information refers to the segmented image of a specified dynamic object or organism, i.e., the prior dynamic region; the deep learning thread is also responsible for combining depth estimation and the camera pose corresponding to keyframes to select keyframes for the local mapping thread; S3, the local mapping thread; S4, the loop closure detection thread. A single bilateral convolutional neural network is used to perform multiple tasks to simultaneously output semantic and depth estimation information, saving computational parameters while providing spatial structure and positional information of the environment, as well as more accurate 3D point depth information. Furthermore, this invention also designs a semantic segmentation-guided depth estimation branch to improve the accuracy of the boundary depth information obtained by the convolutional neural network.

Description

Technical Field

[0001] This invention relates to, and more particularly to, a dynamic visual SLAM method for indoor single-threaded semantic and depth estimation. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) refers to the process by which a robot, while moving in an unknown environment and location, uses its onboard sensors to create a complete map of the environment and determine its own location. Due to its significant academic and practical value, SLAM has long been considered a key technology for achieving fully autonomous mobile robots. Based on sensor type, SLAM can be broadly categorized into laser SLAM and visual SLAM. Traditional visual SLAM systems typically use cameras such as monocular cameras, binocular cameras, monocular structured light RGB-D cameras, and binocular structured RGB-D cameras; fisheye cameras are less commonly used.

[0003] There are few mature applications of visual SLAM, partly because visual SLAM systems are highly dependent on light. Specifically, monocular visual SLAM systems often fail to extract depth in low-texture areas. Furthermore, the resulting point cloud maps lack absolute scale information, and the point cloud data lacks scale correlation with point cloud data extracted by other systems in the same space. For monocular structured light RGB-D cameras, the depth extracted from transparent indoor media often deviates significantly from the true depth value, affecting not only mapping accuracy but also camera pose estimation.

[0004] Another reason is that traditional visual SLAM systems are based on static scenes. When dynamic objects appear in the scene, unstable dynamic feature points directly affect the camera's pose estimation. Furthermore, typical SLAM systems cannot provide structural and positional information about the environment and lack an abstract understanding of the map.

[0005] Berta Bescos et al. proposed a dynamic SLAM algorithm. They employed Mask-RCNN as the semantic segmentation network, combining prior dynamic regions with traditional epipolar geometry algorithms to achieve dynamic feature point removal. However, this network cannot run in real-time, and its deployment in real-world scenarios places high demands on the equipment. Furthermore, the algorithm framework does not address the issue of inaccurate camera pose due to environmental influences in visual SLAM.

[0006] To address the aforementioned issues, CNN-SLAM systems employ two types of convolutional neural networks (CNNs) to provide semantic and depth information, respectively. Their biggest drawback is the enormous computational cost. Furthermore, a major limitation of depth estimation using CNNs is the inaccuracy of object boundary depth values ​​in the depth map. Typically, the grayscale variations between different objects are significant, causing most feature points extracted by visual SLAM systems to cluster in these areas. Because the boundaries of the depth map extracted by convolutional neural networks are inaccurate, directly using these depth values ​​for camera pose estimation often introduces even greater errors. Summary of the Invention

[0007] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a dynamic visual SLAM method for indoor single-threaded semantic and depth estimation. This invention improves the pose estimation algorithm of visual SLAM systems, enhances the accuracy of scale information, and recovers the absolute scale information missing by monocular structured light cameras. Furthermore, it improves the accuracy of pose estimation for visual SLAM systems in dynamic indoor environments.

[0008] The technical solution of the present invention is as follows:

[0009] A single-threaded dynamic visual SLAM method for semantic and depth estimation includes the following steps:

[0010] S1. Tracking Thread: The tracking thread is responsible for tracking the camera pose in real time. By tracking feature points between consecutive frames, it estimates the camera pose in real time through motion estimation methods, that is, the position and orientation of the camera in three-dimensional space, and manages the local map at the same time.

[0011] S2, Deep Learning Thread: The deep learning thread is responsible for providing semantic information to the tracking thread; semantic information refers to the segmented image of a specified dynamic object or organism, i.e., the prior dynamic region; the deep learning thread is also responsible for combining depth estimation and the camera pose corresponding to keyframes to select keyframes for the local mapping thread; depth estimation refers to generating a depth map relative to the camera for each pixel in the input image.

[0012] S3. Local mapping thread: The local mapping thread is responsible for continuously and periodically processing the poses and map points of the filtered keyframes multiple times.

[0013] S4. Loop closure detection thread:

[0014] The loop closure detection thread is responsible for identifying whether the current frame has similar visual content to a previous keyframe, thereby correcting accumulated errors and improving the consistency of the mapping.

[0015] The method, wherein step S1 includes the following steps:

[0016] S11. Receive the input image and extract the ORB feature points of each frame.

[0017] S12. Depending on the camera type, decide whether to perform initial pose estimation. Monocular cameras lack scale information and need to use ORB feature points and semantic information from the deep learning thread to perform initial pose estimation to obtain the initial camera pose and map scale. For stereo cameras and RGB-D cameras, the camera pose and map scale can be obtained directly through stereo matching and sensor data, so there is no need to perform initial pose estimation. However, regardless of the camera type, it is necessary to use the semantic information from the deep learning thread to remove dynamic feature points.

[0018] S13. Perform local map tracking in real time; calculate the camera motion of the current frame relative to the previous frame using motion estimation methods;

[0019] S14. Real-time selection of new keyframes: By analyzing the motion between the current frame and the most recent keyframe, determine whether the current frame should be selected as a new keyframe and use it as input for subsequent threads.

[0020] The method described above, in step S12, employs a dynamic feature point removal algorithm that includes the following steps:

[0021] S121. In the tracking thread, ORB feature points are first extracted from the input image; at the same time, the prior dynamic region of the input image is generated in the deep learning thread.

[0022] S122. The Lucas-Kanade optical flow method is used to predict the position of a fixed point in two-dimensional space at different times; the position difference of the two-dimensional point between two frames is used to calculate the motion velocity of the two-dimensional point; the motion velocity is compared with the velocity threshold, and those greater than the threshold are classified as dynamic feature points, and those less than the threshold are classified as static feature points.

[0023] S123. Utilize semantic information, i.e., the prior dynamic region, to further filter dynamic feature points among static feature points; if a static feature point is within the prior dynamic region, mark it as a dynamic feature point and remove it; if a static feature point is outside the prior dynamic region, retain the static feature point.

[0024] The method described in step S2, within a deep learning thread, includes the following steps:

[0025] S21, Receive the input image and use the BiSeNetV2 network for inference, output the depth estimate and semantic information of the image; the depth estimate provides the distance information between each pixel and the camera, while the semantic information is used to specify the segmented image of dynamic objects or organisms;

[0026] S22. The semantic information is passed to the tracking thread. The tracking thread uses this semantic information as a priori dynamic region to further eliminate dynamic feature points and improve the accuracy of camera pose tracking.

[0027] S23. At the same time, the deep learning thread receives keyframes created by the tracking thread; through a keyframe filtering algorithm, it decides whether to use the keyframe for initialization in subsequent threads or to insert it into an existing set of keyframes.

[0028] The method described in step S21 employs the BiSeNetV2 neural network algorithm: a semantically guided deep branch is introduced into the BiSeNetV2 network, along with a corresponding loss function; the loss function is Loss. m The expression is as follows:

[0029]

[0030] In the formula, s and d represent the semantic and depth information output by the network, respectively; x and y represent the horizontal and vertical pixels of the image, respectively. and These represent the horizontal and vertical gradients of the semantic graph predicted by the network, respectively. and These represent the horizontal and vertical gradients, respectively, indicating the network's predicted depth.

[0031] The network's deep branch and semantic branch employ Huber loss and standard softmax cross-entropy loss, respectively; and use Loss... deep and Loss seg The specific expression is as follows:

[0032]

[0033] In the formula, y d Represents the pixel depth value predicted by the network; N and N c y represents the total number of pixels and the number of pixels in the GT category c, respectively; c and p c These represent the true pixel category and the pixel category predicted by the network, respectively.

[0034] Therefore, the specific expression of the total loss function Loss used in this network is as follows:

[0035] Loss = 0.45 * Loss seg +0.45*Loss deep +0.1*Loss m .

[0036] The keyframe filtering algorithm used in step S23 of the method described is as follows:

[0037] Calculate the difference in camera pose between two key frames in the deep learning thread, set a similarity threshold, and determine whether key frame initialization work is required; the calculation formula is as follows:

[0038]

[0039] In the formula, A ij and B ij are two matrices A and B of size i*j, representing the transformation matrices of key frames A and B respectively; S represents similarity. When A ij = B ij , S = 0, and it is considered that the transformation matrices of the two key frames are the same;

[0040] Introduce the concept of camera pose similarity. Assume that at a certain moment, the poses of the camera at different times have a certain similarity;

[0041] Set a threshold (thred), and substitute the transformation matrices of the current key frame and the previous key frame into the above formula; compare the calculated similarity (S) with the threshold. When S > thred, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key frame initialization and recalculation of the camera pose. When S < thred, it is considered that the pose estimation of the current key frame is accurate, and the key frame is sent to the local mapping thread and inserted into the existing key frame set.

[0042] For the method described above, the workflow of step S3 is as follows:

[0043] S31. Receive instructions and information from the deep learning thread; if key frame initialization is to be performed, recalculate the camera pose based on the depth estimation, establish and insert a new key frame; otherwise, directly insert the key frame;

[0044] S32. Perform map point screening and reconstruction of new map points; in the local map, eliminate map points with poor quality and reconstruct map points through new observation data;

[0045] S33. When there are more than 2 key frames in the local map, perform local map BA optimization, that is, local BA optimization; by simultaneously optimizing the poses of key frames and the positions of map points, improve the consistency and accuracy of the entire map;

[0046] S34. Check and eliminate redundant key frames among the key frames adjacent to the current frame, and send the screened key frames to the loop closure detection thread, that is, local key frame screening; the screened key frames will be used for loop closure detection to find possible loop closure loops.

[0047] For the method described above, the workflow of step S4 is as follows:

[0048] First, keyframes from the local mapping thread are received, and keyframes with continuity are selected; this is called candidate frame selection.

[0049] Secondly, calculate the Sim3 transform between the current frame and the candidate closed-loop frame;

[0050] Then, perform local BA to further optimize the camera pose and map point positions in the essence graph;

[0051] Finally, perform global BA, using all keyframes and map points from the algorithm as optimization variables, and update the map.

[0052] By adopting the above solution, the present invention has the following beneficial effects:

[0053] 1. A deep branch is introduced based on the BiSeNetV2 bilateral semantic neural network to achieve simultaneous prediction of semantic information and depth map. A semantically guided deep branch is also introduced to optimize the problem of unstable depth map boundaries.

[0054] 2. Utilize a real-time semantic segmentation network to output a priori dynamic region, and combine it with the LK optical flow method to remove dynamic and static feature points.

[0055] 3. Impart absolute scale information to the monocular camera. Use neural networks to predict depth maps to solve the problem of inaccurate depth in visual SLAM caused by environmental factors. Attached Figure Description

[0056] Figure 1 This is the overall flowchart of the present invention;

[0057] Figure 2 Flowchart of the dynamic feature point removal algorithm;

[0058] Figure 3 Diagram of the improved BiSeNetV2 network structure; Detailed Implementation

[0059] The present invention will be described in detail below with reference to specific embodiments.

[0060] Example 1

[0061] A single-threaded dynamic visual SLAM method for semantic and depth estimation employs a single bilateral convolutional neural network to perform multiple tasks, simultaneously outputting semantic and depth estimation information. This saves computational parameters while incorporating spatial structure and location information of the environment, as well as more accurate 3D point depth information. Furthermore, this invention designs a semantic segmentation-guided depth estimation branch to improve the accuracy of boundary depth information obtained by the convolutional neural network.

[0062] The overall framework of this method is as follows: Figure 1 This includes the following steps:

[0063] S1. Tracing Threads:

[0064] As one of the primary tasks of the algorithm in this invention, the tracking thread is responsible for real-time tracking of the camera's pose. By tracking feature points between consecutive frames, it estimates the camera's pose (i.e., its position and orientation in 3D space) in real time using motion estimation methods, while simultaneously managing the local map. The following is a detailed description of its workflow:

[0065] S11. Receive the input image and extract the ORB feature points of each frame.

[0066] S12. Depending on the camera type, decide whether to perform initial pose estimation. This is because monocular cameras lack scale information and require ORB feature points and semantic information from the deep learning thread to perform initial pose estimation to obtain the initial camera pose and map scale. For stereo cameras and RGB-D cameras, however, the camera pose and map scale can be obtained directly through stereo matching and sensor data, thus eliminating the need for initial pose estimation. Nevertheless, regardless of the camera type, it is necessary to utilize the semantic information from the deep learning thread to remove dynamic feature points. This will be discussed in detail later.

[0067] S13. Perform local map tracking in real time. Calculate the camera motion of the current frame relative to the previous frame using motion estimation methods.

[0068] S14. Real-time selection of new keyframes. By analyzing the motion between the current frame and the most recent keyframe, determine whether the current frame should be selected as a new keyframe and use it as input for subsequent threads.

[0069] Through the above steps, the tracking thread can track the camera's pose in real time and manage the local map, providing stable and accurate input for subsequent worker threads.

[0070] Regarding the dynamic feature point removal algorithm used in step S12 of the above workflow:

[0071] like Figure 2 As shown, this invention employs a combination of semantic segmentation and the RANSAC algorithm to avoid the influence of dynamic feature points on initial pose estimation. Even in stereo cameras and RGB-D cameras where initial pose estimation is not required, the improved method of this invention still uses semantic information from the deep learning thread to remove dynamic feature points, thereby improving the accuracy of camera pose tracking.

[0072] The specific process includes:

[0073] S121. In the tracking thread, ORB feature points are first extracted from the input image. Simultaneously, the prior dynamic region of the input image is generated in the deep learning thread.

[0074] S122. The Lucas-Kanade optical flow method is used to predict the position of a fixed point in two-dimensional space at different times. The motion velocity of the two-dimensional point is calculated by the position difference between two frames of images. The motion velocity is compared with a velocity threshold; those greater than the threshold are classified as dynamic feature points, and those less than the threshold are classified as static feature points.

[0075] S123. Utilize semantic information, i.e., the prior dynamic region, to further filter dynamic feature points among static feature points. If a static feature point is within the prior dynamic region, mark it as a dynamic feature point and remove it. If a static feature point is outside the prior dynamic region, retain that static feature point.

[0076] This algorithm addresses challenges in images with missing textures, repetitive textures, and large scale variations. By leveraging information from deep learning threads, it overcomes the inaccurate optical flow estimation issues that can occur in the Lucas-Kanade optical flow method in these regions.

[0077] By combining semantic segmentation and the RANSAC algorithm, dynamic feature point removal was achieved, thereby improving the accuracy of camera pose tracking.

[0078] S2. Deep Learning Thread:

[0079] As the second key step in the method of this invention, the deep learning thread is responsible for providing semantic information to the tracking thread. Semantic information refers to the segmented image of a specified dynamic object or organism, which we call the prior dynamic region. Furthermore, the deep learning thread is also responsible for combining depth estimation and the camera pose corresponding to keyframes to select keyframes for the local mapping thread. Depth estimation refers to generating a depth map relative to the camera for each pixel in the input image.

[0080] In the deep learning thread, the input image is received and inference is performed using the BiSeNetV2 network, outputting a depth estimate and semantic information for the image. The depth estimate provides distance information between each pixel and the camera, while the semantic information is used to segment images of dynamic objects or organisms.

[0081] Semantic information is passed to the tracking thread, which uses this semantic information as a priori dynamic region to further eliminate dynamic feature points and improve the accuracy of camera pose tracking.

[0082] Meanwhile, the deep learning thread receives keyframes created by the tracking thread. A keyframe filtering algorithm determines whether to use the keyframe for initialization in a subsequent thread or to insert it into an existing set of keyframes.

[0083] Through the steps described above, the deep learning thread provides semantic information, i.e., prior dynamic regions, to the tracking thread, effectively eliminating dynamic feature points. Simultaneously, the deep learning thread also participates in keyframe selection, ensuring that appropriate keyframes are used for initialization or insertion operations in the local mapping thread. Thus, the work of the deep learning thread provides crucial support for the smooth operation of the entire algorithm.

[0084] Regarding the neural network algorithm used in S2:

[0085] This invention employs the BiSeNetV2 network, which boasts excellent real-time performance and accuracy. The network structure... Figure 3 As shown, the BiSeNetV2 network employs a semantic branch and a detail branch to extract semantic and spatial location information from the input image, respectively. It uses a pooling layer (BGA-Layer) to fuse the outputs of these two branches, and then the SegHead outputs the predicted information. To ensure single-threaded semantic segmentation and depth estimation, this invention adds a DeepHead to the BiSeNetV2 network to output the depth values ​​of pixels. The DeepHead consists of convolutional layers with kernel_size=3, stride=1, and padding=1, and ReLU activation layers. To address the issue of inaccurate boundary values ​​in the depth estimation output by the neural network, this invention designs a semantically guided depth branch, as shown by the direction between the SegHead and DeepHead in the figure.

[0086] For the semantic information-guided depth estimation part, this invention introduces a corresponding loss function. Loss Function — Loss m The expression is as follows:

[0087]

[0088] In the formula, s and d represent the semantic and depth information output by the network, respectively; x and y represent the horizontal and vertical pixels of the image, respectively. and These represent the horizontal and vertical gradients of the semantic graph predicted by the network, respectively. and These represent the horizontal and vertical gradients, respectively, representing the network's predicted depth.

[0089] The network depth branch and semantic branch employ Huber loss and standard softmax cross-entropy loss, respectively. Loss is used for each. deep and Loss seg The specific expression is as follows:

[0090]

[0091] In the formula, y d Represents the pixel depth value predicted by the network; N and N c y represents the total number of pixels and the number of pixels in the GT category c, respectively; c and p c These represent the true pixel category and the pixel category predicted by the network, respectively.

[0092] Therefore, the specific expression of the total loss function Loss used in this network is as follows:

[0093] Loss = 0.45 * Loss seg +0.45*Loss deep +0.1*Loss m

[0094] Regarding the keyframe selection algorithm used in S2:

[0095] This invention calculates the difference in camera pose between two keyframes within a deep learning thread, sets a similarity threshold, and determines whether keyframe initialization is necessary. The calculation formula is as follows:

[0096]

[0097] In the formula, A ij and B ij Let A and B be two matrices of size i*j, representing the transformation matrices of keyframes A and B, respectively; S represents the similarity, when A ij =B ij When S = 0, it is assumed that the transformation matrices of the two keyframes are consistent.

[0098] In practical deployments of visual SLAM systems, camera motion is often gradual with some translation. Based on this assumption, this algorithm introduces the concept of camera pose similarity, assuming that the camera poses at different times have a certain similarity within a certain timeframe. This timeframe can be several frames.

[0099] The specific implementation method is to set a threshold (thred), and substitute the transformation matrices of the current key frame and the previous key frame into the above formula. The calculated similarity (S) is compared with the threshold. When S > thred, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key frame initialization and recalculation of the camera pose; when S < thred, it is considered that the pose estimation of the current key frame is accurate, and the key frame is sent to the local mapping thread and inserted into the existing set of key frames.

[0100] S3. Local mapping thread:

[0101] The local mapping thread is responsible for continuously and regularly processing the poses of the filtered key frames and the map points multiple times. Its main workflow is as follows:

[0102] S31. Receive instructions and information from the deep learning thread. If it is for key frame initialization, recalculate the camera pose based on the depth estimation, establish and insert a new key frame. Otherwise, directly insert the key frame.

[0103] S32. Perform map point screening and reconstruction of new map points. In the local map, eliminate map points with poor quality and reconstruct map points through new observation data.

[0104] S33. When there are more than 2 key frames in the local map, perform local bundle adjustment (BA) optimization of the local map, that is, local BA optimization. By simultaneously optimizing the poses of the key frames and the positions of the map points, the consistency and accuracy of the entire map are improved.

[0105] S34. Check and eliminate redundant key frames among the key frames adjacent to the current frame, and send the filtered key frames to the loop closure detection thread. That is, local key frame screening. The purpose of local key frame screening is to remove redundant key frames, reduce computational overhead, and improve the efficiency of the system. The filtered key frames will be used for loop closure detection to find possible loop closure loops.

[0106] Through the above workflow, the local mapping thread continuously processes the poses of the filtered key frames and the map points, maintains the accuracy and consistency of the map, and provides appropriate key frames for loop closure detection. This can improve the robustness and performance of the SLAM system.

[0107] S4. Loop closure detection thread:

[0108] The loop closure detection thread is responsible for identifying whether there is similar visual content between the current frame and a certain previous key frame, so as to correct the cumulative error and improve the consistency of mapping. Its workflow is as follows:

[0109] First, keyframes from the local mapping thread are received, and keyframes with continuity are selected. This is known as candidate frame selection.

[0110] Next, calculate the Sim3 transform between the current frame and the candidate closed-loop frame.

[0111] Then, local BA is performed to further optimize the camera pose and map point positions in the essence graph.

[0112] Finally, a global BA is executed, using all keyframes and map points from the algorithm as optimization variables, and the map is updated.

[0113] Through the above workflow, the loop closure detection thread can identify closed frames and correct accumulated errors by optimizing camera pose and map point positions, thereby improving mapping consistency. This enhances the robustness and accuracy of the SLAM system and provides more reliable map information.

[0114] Example 2

[0115] 1. Select the working camera type. First, start the tracking thread and the deep learning thread. The tracking thread extracts ORB features, while the deep learning thread obtains the depth map and prior dynamic region semantic map of the input image. The LK optical flow method is used in conjunction with the prior dynamic region to remove dynamic feature points. For ease of calculation, pixels of a 3x3 region are substituted into the constraint equations. In the middle, the pixel movement speed threshold V is set according to the actual operation of the scenario. x V y If u > V x or v>V y If the feature point is within the prior dynamic region, it is considered a dynamic feature point; otherwise, it is considered a static feature point. The remaining static feature points are then further differentiated by determining whether they fall within the prior dynamic region.

[0116] 2. If it is a monocular camera, initialization is performed. For the monocular initialization process, the 3D point depth information d1 and d2 obtained by the deep learning thread are substituted into the epipolar geometric constraint d1p. 1 =d2p 2 +t, and the coordinates p of the target point in the image coordinate system are known. 1 p 2 Therefore, the camera's unique displacement t can be calculated. For stereo cameras, the camera pose can be directly obtained through stereo matching. For RGB-D cameras, depth data can be directly obtained from the sensor, and the camera pose can be obtained through a similar calculation process to that of a monocular camera. Finally, the camera pose corresponding to each frame is output for keyframe selection and transmission to the local mapping thread.

[0117] 3. Substitute the transformation matrices of the current keyframe and the previous keyframe into the formula for calculating the camera pose difference. In the middle, set the threshold S tes =0.7. If S>S tes If the current keyframe estimate of the camera pose is relatively accurate, then the keyframe is inserted; if S tes If the current keyframe is affected by the environment, its depth value is inaccurate, resulting in a large camera pose error. The corresponding neural network-predicted depth estimate is extracted, the camera pose of the current keyframe is recalculated, and inserted into the current keyframe. BA optimization is performed to adjust the camera pose and position information corresponding to the new map points, and redundant keyframes are filtered and removed.

[0118] 4. Perform loop closure detection and correction. Loop closure detection uses bag-of-words to detect loop closure keyframes, followed by similarity transformation using the sim3 algorithm; loop closure correction uses global BA to optimize the intrinsic image.

[0119] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.​

Claims

1. A single-threaded dynamic visual SLAM method for semantic and depth estimation, characterized in that, Includes the following steps: S1, Tracking Thread: The tracking thread is responsible for tracking the camera pose in real time. By tracking feature points between consecutive frames, it estimates the camera pose in real time through motion estimation methods, that is, the position and orientation of the camera in three-dimensional space, and manages the local map at the same time. Depending on the camera type, it is determined whether to perform initial pose estimation. Monocular cameras need to use ORB feature points and semantic information from the deep learning thread to perform initial pose estimation to obtain the initial camera pose and map scale. Binocular cameras and RGB-D cameras, on the other hand, directly obtain the camera pose and map scale through stereo matching and sensor data. Regardless of the camera type, it is necessary to utilize the semantic information of deep learning threads to remove dynamic feature points; The dynamic feature point removal algorithm used includes the following steps: S121. In the tracking thread, ORB feature points are first extracted from the input image; at the same time, the prior dynamic region of the input image is generated in the deep learning thread. S122. The Lucas-Kanade optical flow method is used to predict the position of a fixed point in two-dimensional space at different times; the position difference of the two-dimensional point between two frames is used to calculate the motion velocity of the two-dimensional point; the motion velocity is compared with the velocity threshold, and those greater than the threshold are classified as dynamic feature points, and those less than the threshold are classified as static feature points. S123. Utilize semantic information, i.e., the prior dynamic region, to further filter dynamic feature points among static feature points; if a static feature point is within the prior dynamic region, mark it as a dynamic feature point and remove it; if a static feature point is outside the prior dynamic region, retain the static feature point. S2, Deep Learning Thread: The deep learning thread is responsible for providing semantic information to the tracking thread; semantic information refers to the segmented image of a specified dynamic object or organism, i.e., the prior dynamic region; the deep learning thread is also responsible for combining depth estimation and the camera pose corresponding to keyframes to select keyframes for the local mapping thread; depth estimation refers to generating a depth map relative to the camera for each pixel in the input image. The deep learning thread includes the following steps: S21, Receive the input image and use the BiSeNetV2 network for inference, output the depth estimate and semantic information of the image; the depth estimate provides the distance information between each pixel and the camera, while the semantic information is used to specify the segmented image of dynamic objects or organisms; The BiSeNetV2 neural network algorithm used in step S21: A semantically guided deep branch is introduced into the BiSeNetV2 network, along with a corresponding loss function; the loss function— The expression is as follows: ； In the formula, These represent the semantic and depth information output by the network, respectively. These represent the horizontal and vertical pixels of the image, respectively. and These represent the horizontal and vertical gradients of the semantic graph predicted by the network, respectively. and These represent the horizontal and vertical gradients, respectively, indicating the network's predicted depth. The network's deep branch and semantic branch employ Huber loss and standard softmax cross-entropy loss, respectively; and respectively use and The specific expression is as follows: ； ； In the formula, This represents the pixel depth value predicted by the network. and These represent the total number of pixels and the number of pixels with GT category c, respectively. and These represent the true pixel category and the pixel category predicted by the network, respectively. Therefore, the total loss function adopted by this network The specific expression is as follows: ； S22. The semantic information is passed to the tracking thread. The tracking thread uses this semantic information as a priori dynamic region to further eliminate dynamic feature points and improve the accuracy of camera pose tracking. S23. At the same time, the deep learning thread receives keyframes created by the tracking thread; through a keyframe filtering algorithm, it decides whether to use the keyframe for initialization in subsequent threads or to insert it into an existing set of keyframes. S3, Local Mapping Thread: The local mapping thread is responsible for continuously and periodically processing the poses and map points of the filtered keyframes multiple times. S4, Loopback Detection Thread: The loop closure detection thread is responsible for identifying whether the current frame has similar visual content to a previous keyframe, thereby correcting accumulated errors and improving the consistency of the mapping.

2. The method according to claim 1, characterized in that, Step S1 includes the following steps: S11. Receive the input image and extract the ORB feature points of each frame of the image; S12. Depending on the camera type, decide whether to perform initial pose estimation. Monocular cameras need to use ORB feature points and semantic information from the deep learning thread to perform initial pose estimation to obtain the initial camera pose and map scale. Binocular cameras and RGB-D cameras directly obtain the camera pose and map scale through stereo matching and sensor data. Regardless of the camera type, it is necessary to use the semantic information from the deep learning thread to remove dynamic feature points. S13. Perform local map tracking in real time; calculate the camera motion of the current frame relative to the previous frame using motion estimation methods; S14. Real-time selection of new keyframes: By analyzing the motion between the current frame and the most recent keyframe, determine whether the current frame should be selected as a new keyframe and use it as input for subsequent threads.

3. The method according to claim 1, characterized in that, The keyframe filtering algorithm used in step S23 is as follows: In the deep learning thread, the difference in camera pose between two keyframes is calculated, a similarity threshold is set, and it is determined whether keyframe initialization is needed; the calculation formula is as follows: ； In the formula, and For two sizes Matrices A and B represent the transformation matrices of keyframes A and B, respectively. Represents similarity, when = hour, It is assumed that the transformation matrices of the two keyframes are identical; The concept of camera pose similarity is introduced, assuming that at a certain moment, the camera poses at different moments have a certain similarity. Set a threshold thresh and substitute the transformation matrices of the current keyframe and the previous keyframe into the above formula; Calculated similarity Compare with the threshold; when S > thresh, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key frame initialization and recalculation of the camera pose; when S < thresh, it is considered that the pose estimation of the current key frame is accurate, and the key frame is sent to the local mapping thread and inserted into the existing key frame set.

4. The method according to claim 1, characterized in that, The workflow for step S3 is as follows: S31. Receive deep learning thread instructions and information; if keyframe initialization is to be performed, recalculate the camera pose based on depth estimation, establish and insert new keyframes; otherwise, insert keyframes directly. S32. Perform map point filtering and new map point reconstruction; in the local map, remove map points of poor quality and reconstruct map points using new observation data; S33. When there are more than two keyframes in a local map, perform local map BA optimization, i.e., local BA optimization. By simultaneously optimizing the pose of keyframes and the position of map points, the consistency and accuracy of the entire map can be improved. S34. Check and remove redundant keyframes in the adjacent keyframes of the current frame, and send the filtered keyframes to the loop closure detection thread, i.e., local keyframe filtering; the filtered keyframes will be used for loop closure detection to find possible closed loops.

5. The method according to claim 1, characterized in that, The workflow for step S4 is as follows: First, keyframes from the local mapping thread are received, and keyframes with continuity are selected; this is called candidate frame selection. Secondly, calculate the Sim3 transform between the current frame and the candidate closed-loop frame; Then, perform local BA to further optimize the camera pose and map point positions in the essence graph; Finally, global BA is executed, using all keyframes and map points from the algorithm as optimization variables; and the map is updated.

Images