Dynamic vision SLAM (Simultaneous Localization and Mapping) method for indoor single-thread semantics and depth estimation

Abstract

The invention discloses a dynamic vision SLAM (Simultaneous Localization and Mapping) method for single-thread semantic and depth estimation. The dynamic vision SLAM method comprises the following steps: S1, tracking a thread; s2, a deep learning thread, wherein the deep learning thread is responsible for providing semantic information for the tracking thread; the semantic information refers to a segmented image of a specified dynamic object or organism, namely a prior dynamic region; the deep learning thread is also responsible for screening key frames for the local mapping thread in combination with depth estimation and camera poses corresponding to the key frames; s3, a local mapping thread; and S4, carrying out loopback detection on the thread performance. A single bilateral convolutional neural network is adopted to execute multiple tasks so as to output semantic and depth estimation information at the same time, calculation parameters are saved, and meanwhile space structure information and position information of the environment and more accurate 3D point depth information are given. In addition, semantic segmentation is further designed to guide depth estimation branches, so that the accuracy of boundary depth information obtained by the convolutional neural network is improved.

Description

Technical Field

[0001] The present invention relates to, in particular to, a dynamic visual SLAM method for indoor single-threaded semantics and depth estimation. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) refers to a robot moving in an unknown environment and unknown location while using its own sensors to map out a complete map of the environment and determine its own location. Due to the important academic and application value of SLAM, it has always been considered a key technology for realizing fully autonomous mobile robots. According to the classification of sensors, it can be divided into two categories: laser SLAM and visual SLAM. In traditional visual SLAM systems, the cameras used are usually divided into: monocular cameras, binocular cameras, monocular structured light RGB-D cameras, and binocular structured RGB-D cameras. Fisheye cameras are usually less used.

[0003] However, there are not many mature application cases of visual SLAM. One of the reasons is that the visual SLAM system is highly dependent on light. Specifically, the monocular visual SLAM system often fails to extract depth in low-texture areas. In addition, the obtained point cloud map lacks absolute scale information, and the obtained point cloud data lacks scale correlation with the point cloud data extracted by other systems in the same space. Monocular structured light RGB-D cameras are often affected by indoor transparent media, and the depth extracted from such objects often has a large error with the true value, which not only affects the mapping effect, but also has a certain impact on the camera posture estimation.

[0004] Another reason is that traditional visual SLAM systems are used in static scenes. When dynamic objects appear in the scene, unstable dynamic feature points directly affect the camera's attitude estimation. In addition, typical SLAM systems cannot provide structural and location information of the environment and lack an abstract understanding of the map.

[0005] Berta Bescos et al. proposed a dynamic SLAM. Mask-RCNN was used as the semantic segmentation network, and the dynamic region prior was combined with the traditional epipolar geometry algorithm to eliminate dynamic feature points. However, the network cannot run in real time, and the requirements for equipment deployment in actual scenarios are very high. At the same time, the algorithm framework does not solve the problem that visual SLAM is affected by the environment and shows inaccurate camera pose.

[0006] To solve the above problems, the CNN-SLAM system uses two types of convolutional neural networks to provide semantic information and depth information respectively. Its biggest disadvantage is the huge computational overhead. At the same time, the biggest disadvantage of convolutional neural networks for depth estimation is the inaccurate depth value of the object boundary in the depth map. Usually, the grayscale changes between different objects are often obvious, which causes most of the feature points extracted by the visual SLAM system to gather here. Since the boundaries of the depth map extracted by the convolutional neural network are inaccurate, directly using the depth value for camera pose estimation often introduces greater errors. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a dynamic visual SLAM method for indoor single-threaded semantics and depth estimation in view of the deficiencies of the prior art. The present invention improves the posture estimation algorithm of the visual SLAM system, improves the accuracy of scale information, and restores the absolute scale information missing from the monocular structured light camera. And improves the accuracy of posture estimation of the visual SLAM system in indoor dynamic environments.

[0008] The technical solution of the present invention is as follows:

[0009] A single-threaded semantic and depth estimation dynamic visual SLAM method comprises the following steps:

[0010] S1. Tracking thread: The tracking thread is responsible for real-time tracking of the camera's posture. By tracking feature points between consecutive frames, the camera's posture, that is, the position and direction of the camera in three-dimensional space, is estimated in real time through motion estimation methods, and the local map is managed at the same time.

[0011] S2, deep learning thread: The deep learning thread is responsible for providing semantic information to the tracking thread; semantic information refers to the segmented image of the specified dynamic object or organism, that is, the prior dynamic area; the deep learning thread is also responsible for combining the depth estimation and the camera pose corresponding to the key frame to select the key frame for the local mapping thread; depth estimation refers to the generation of a depth map relative to the camera for each pixel in the input image;

[0012] S3. Local mapping thread: The local mapping thread is responsible for processing the poses and map points of the filtered key frames multiple times on a regular basis;

[0013] S4. Loop detection thread:

[0014] The loop detection thread is responsible for identifying whether the current frame has similar visual content to a previous keyframe, thereby correcting the accumulated errors and improving the consistency of the map.

[0015] The method, step S1, comprises the following steps:

[0016] S11, receiving an input image and extracting ORB feature points of each frame of the image.

[0017] S12. Decide whether to perform initialization pose estimation based on the camera type. Monocular cameras lack scale information and need to use ORB feature points and semantic information from the deep learning thread to perform initialization pose estimation to obtain the camera's initial pose and map scale. For binocular cameras and RGB-D cameras, the camera pose and map scale can be directly obtained through stereo matching and sensor data, so there is no need to perform initialization pose estimation. However, regardless of the camera type, it is necessary to use the semantic information of the deep learning thread to remove dynamic feature points.

[0018] S13, performing local map tracking in real time; calculating the camera motion of the current frame relative to the previous frame by a motion estimation method;

[0019] S14. Screen new key frames in real time; by analyzing the motion between the current frame and the most recent key frame, determine whether the current frame should be selected as a new key frame, and use it as input for subsequent threads.

[0020] In the method, in step S12, the dynamic feature point elimination algorithm process adopted includes:

[0021] S121. In the tracking thread, firstly, extract ORB feature points from the input image; and at the same time, generate a priori dynamic region of the input image in the deep learning thread;

[0022] S122, using the Lucas-Kanade optical flow method to predict the position of a fixed point in a two-dimensional space at different times; calculating the movement speed of the two-dimensional point by the position difference of the two-dimensional point between two frames of images; comparing the movement speed with the speed threshold, planning the point greater than the threshold as a dynamic feature point, and planning the point less than the threshold as a static feature point;

[0023] S123, using semantic information, that is, the prior dynamic region, further filtering dynamic feature points in the static feature points; if a static feature point is within the prior dynamic region, marking it as a dynamic feature point and removing it; if a static feature point is outside the prior dynamic region, retaining the static feature point.

[0024] The method, S2, in the deep learning thread, comprises the following steps:

[0025] S21, receives an input image and uses the BiSeNetV2 network for inference to output the depth estimation and semantic information of the image; the depth estimation provides the distance information between each pixel and the camera, while the semantic information is used to specify the segmented image of a dynamic object or organism;

[0026] S22, passing the semantic information to the tracking thread, which uses the semantic information as a priori dynamic area to further remove dynamic feature points and improve the accuracy of camera posture tracking;

[0027] S23. At the same time, the deep learning thread receives the key frame created by the tracking thread; through the key frame screening algorithm, it decides whether to use the key frame for initialization in subsequent threads or to insert it into the existing key frame set.

[0028] The method described above, the BiSeNetV2 neural network algorithm used in step S21: introduces a semantic guidance depth branch into the BiSeNetV2 network, and introduces a corresponding loss function; the loss function - Loss m The expression is as follows:

[0029]

[0030] In the formula, s and d represent the semantic information and depth information output by the network respectively; x and y represent the horizontal and vertical pixels of the image respectively; and They represent the horizontal gradient and vertical gradient of the semantic graph predicted by the network respectively; and Represent the horizontal and vertical gradients of the network prediction depth respectively;

[0031] The network depth branch and semantic branch use Huber loss and standard softmax cross entropy loss respectively; Loss deep and Loss seg The specific expression is as follows:

[0032]

[0033] In the formula, y d Represents the pixel depth value predicted by the network; N and N c Represent the total number of pixels and the number of pixels of GT category c respectively; y c and p c Represent the true category and the pixel category predicted by the network respectively;

[0034] Therefore, the specific expression of the total loss function Loss used in this network is as follows:

[0035] Loss = 0.45 * Loss seg +0.45*Loss deep +0.1*Loss m .

[0037] In the method described above, the key-frame screening algorithm adopted in step S23 is as follows:

[0038] Calculate the difference in camera poses between two key frames in the deep learning thread, set a similarity threshold, and determine whether key-frame initialization work is required; the calculation formula is as follows:

[0039]

[0040] In the formula, A ij and B ij are two matrices A and B of size i*j, representing the transformation matrices of key frames A and B respectively; S represents similarity. When A ij = B ij , S = 0, indicating that the transformation matrices of the two key frames are the same;

[0041] Introduce the concept of camera pose similarity. Assume that within a certain time period, the poses of the camera at different times have a certain similarity;

[0042] Set a threshold (thred), and substitute the transformation matrices of the current key frame and the previous key frame into the above formula; compare the calculated similarity (S) with the threshold. When S > thred, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key-frame initialization and recalculation of the camera pose. When S < thred, it is considered that the pose estimation of the current key frame is accurate, and the key frame is sent to the local mapping thread and inserted into the existing key-frame set.

[0043] In the method described above, the workflow of step S3 is as follows:

[0044] S31. Receive instructions and information from the deep learning thread; if it is for key-frame initialization, recalculate the camera pose based on the depth estimation, establish and insert a new key frame; otherwise, directly insert the key frame;

[0045] S32. Perform map point screening and new map point reconstruction; in the local map, eliminate map points with poor quality and reconstruct map points through new observation data;

[0046] S33. When there are more than 2 key frames in the local map, perform local bundle adjustment (BA) optimization, i.e., local BA optimization; by simultaneously optimizing the poses of key frames and the positions of map points, improve the consistency and accuracy of the entire map;

[0047] S34. Check and eliminate redundant key frames among the key frames adjacent to the current frame, and send the screened key frames to the loop closure detection thread, i.e., local key-frame screening; the screened key frames will be used for loop closure detection to find possible loop closure loops.

[0048] The described method, step S4 workflow is as follows:

[0049] First, receive the key frames from the local mapping thread and filter out the key frames with continuity, i.e., candidate frame selection;

[0050] Secondly, the Sim3 transform between the current frame and the candidate closed-loop frame is calculated;

[0051] Then, local BA is performed to further optimize the camera pose and map point positions in the essential graph;

[0052] Finally, global BA is performed, using all keyframes and map points in the algorithm as optimization variables; and the map update is completed.

[0053] By adopting the above scheme, the present invention has the following beneficial effects:

[0054] 1. Based on the bilateral semantic neural network BiSeNetV2, a deep branch is introduced to achieve simultaneous prediction of semantic information and depth map. A semantically guided deep branch is introduced to optimize the problem of unstable depth map boundaries.

[0055] 2. Use the real-time semantic segmentation network to output the prior dynamic area, and cooperate with the LK optical flow method to eliminate dynamic and static feature points.

[0056] 3. Give the monocular camera absolute scale information. Use neural network to predict the depth map to solve the problem of inaccurate depth caused by visual SLAM when affected by environmental factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is the overall flow chart of the present invention;

[0058] Figure 2 This is the flow chart of the dynamic feature point removal algorithm;

[0059] Figure 3 This is the improved BiSeNetV2 network structure diagram; DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments.

[0061] Example 1

[0062] A single-threaded semantic and depth estimation dynamic visual SLAM method uses a single bilateral convolutional neural network to perform multiple tasks to achieve simultaneous output of semantic and depth estimation information, saving calculation parameters while providing the environment's spatial structure information and position information, as well as more accurate 3D point depth information. In addition, the present invention also designs a semantic segmentation-guided depth estimation branch to improve the accuracy of boundary depth information obtained by the convolutional neural network.

[0063] The overall framework of this method is as follows Figure 1 , including the following steps:

[0064] S1. Tracking thread:

[0065] As one of the primary tasks of the algorithm of the present invention, the tracking thread is responsible for real-time tracking of the camera's pose. By tracking feature points between consecutive frames, the camera's pose, i.e., the position and direction of the camera in three-dimensional space, is estimated in real time through motion estimation methods, and the local map is managed at the same time. The following is a detailed description of its workflow:

[0066] S11, receiving an input image and extracting ORB feature points of each frame of the image.

[0067] S12. Decide whether to perform initialization pose estimation based on the camera type. This is because the monocular camera lacks scale information and needs to use ORB feature points and semantic information from the deep learning thread to perform initialization pose estimation to obtain the initial camera pose and map scale. For binocular cameras and RGB-D cameras, they can directly obtain the camera pose and map scale through stereo matching and sensor data, so there is no need to perform initialization pose estimation. However, regardless of the camera type, it is necessary to use the semantic information of the deep learning thread to remove dynamic feature points. This will be discussed in detail later.

[0068] S13, performing local map tracking in real time, calculating the camera motion of the current frame relative to the previous frame by using a motion estimation method.

[0069] S14, screening new key frames in real time. By analyzing the motion between the current frame and the most recent key frame, it is determined whether the current frame should be selected as a new key frame, and the frame is used as input for subsequent threads.

[0070] Through the above steps, the tracking thread can track the camera's position and manage the local map in real time, providing stable and accurate input for subsequent work threads.

[0071] Regarding the dynamic feature point elimination algorithm used in step S12 of the above workflow:

[0072] like Figure 2As shown, the present invention adopts a combination of semantic segmentation and RANSAC algorithm to avoid the influence of dynamic feature points on the initial pose estimation. Even if the initial pose estimation does not need to be performed in the binocular camera and RGB-D camera, the improved method of the present invention still uses the semantic information from the deep learning thread to remove dynamic feature points to improve the accuracy of camera pose tracking.

[0073] The specific process includes:

[0074] S121, in the tracking thread, first extract ORB feature points from the input image, and at the same time generate a priori dynamic region of the input image in the deep learning thread.

[0075] S122. Use the Lucas-Kanade optical flow method to predict the position of fixed points in two-dimensional space at different times. Calculate the movement speed of the two-dimensional point by the position difference of the two-dimensional point between two frames of images. Compare the movement speed with the speed threshold. The speed greater than the threshold is planned as a dynamic feature point, and the speed less than the threshold is planned as a static feature point.

[0076] S123, using semantic information, i.e., the prior dynamic region, further filtering dynamic feature points in the static feature points. If a static feature point is within the prior dynamic region, it is marked as a dynamic feature point and removed. If a static feature point is outside the prior dynamic region, the static feature point is retained.

[0077] The algorithm can solve the challenges of missing texture, repeated texture and large scale changes in images. By using the information of deep learning threads, it overcomes the problem of inaccurate optical flow estimation in these areas of the Lucas-Kanade optical flow method.

[0078] By combining semantic segmentation and RANSAC algorithm, dynamic feature points are eliminated, thereby improving the accuracy of camera pose tracking.

[0079] S2. Deep learning thread:

[0080] As the second key step of the method of the present invention, the deep learning thread is responsible for providing semantic information to the tracking thread. Semantic information refers to the segmented image of a specified dynamic object or organism, which we call the prior dynamic region. In addition, the deep learning thread is also responsible for selecting key frames for the local mapping thread by combining depth estimation and the camera pose corresponding to the key frames. Depth estimation refers to generating a depth map relative to the camera for each pixel in the input image.

[0081] In the deep learning thread, the input image is received and the BiSeNetV2 network is used for inference to output the depth estimation and semantic information of the image. The depth estimation provides the distance information between each pixel and the camera, while the semantic information is used to specify the segmented image of dynamic objects or organisms.

[0082] The semantic information is passed to the tracking thread, which uses the semantic information as a priori dynamic area to further eliminate dynamic feature points and improve the accuracy of camera pose tracking.

[0083] At the same time, the deep learning thread receives the keyframes created by the tracking thread. Through the keyframe screening algorithm, it decides whether to use the keyframe for initialization in subsequent threads or insert it into the existing keyframe collection.

[0084] Through the above steps, the deep learning thread can provide semantic information, namely the prior dynamic area, to the tracking thread to effectively remove dynamic feature points. At the same time, the deep learning thread also participates in the screening of key frames to ensure that appropriate key frames are used for initialization or insertion operations in the local mapping thread. In this way, the work of the deep learning thread provides important support for the smooth progress of the entire algorithm.

[0085] Neural network algorithm used for S2:

[0086] The present invention adopts the BiSeNetV2 network with excellent real-time performance and accuracy. Figure 3 As shown. The BiSeNetV2 network uses a semantic branch (Semantic Branch) and a detail branch (Detail Branch) to extract the semantic information and spatial position information of the input image respectively. It uses a aggregation layer (BGA-Layer) to fuse the outputs of the above two branches, and then SegHead outputs the prediction information. In order to ensure single-threaded semantic segmentation and depth estimation, the present invention additionally adds DeepHead to the BiSeNetV2 network to output the depth value of the pixel point. Among them, DeepHead is composed of convolution and ReLU activation layers with kernel_size=3, stride=1, padding=1. In order to solve the problem that the depth estimation output by the neural network has inaccurate boundaries, the present invention designs a semantically guided depth branch, as shown by the direction between SegHead and DeepHead in the figure.

[0087] For the semantic information-guided depth estimation part, the present invention introduces a corresponding loss function. m The expression is as follows:

[0088]

[0089] In the formula, s and d represent the semantic information and depth information output by the network respectively; x and y represent the horizontal and vertical pixels of the image respectively; and They represent the horizontal gradient and vertical gradient of the semantic graph predicted by the network respectively; and Represent the horizontal and vertical gradients of the network prediction depth respectively.

[0090] The network depth branch and semantic branch use Huber loss and standard softmax cross entropy loss respectively. deep and Loss seg The specific expression is as follows:

[0091]

[0092] In the formula, y d Represents the pixel depth value predicted by the network; N and N c Represent the total number of pixels and the number of pixels of GT category c respectively; y c and p c Represent the true category and the pixel category predicted by the network, respectively.

[0093] Therefore, the specific expression of the total loss function Loss used in this network is as follows:

[0094] Loss = 0.45 * Loss seg +0.45*Loss deep +0.1*Loss m

[0095] For the key frame screening algorithm used in S2:

[0096] The present invention calculates the difference in camera pose between two key frames in the deep learning thread, sets a similarity threshold, and determines whether key frame initialization is required. The calculation formula is as follows:

[0097]

[0098] In the formula, A ij and B ij are two matrices A and B of size i*j, representing the transformation matrices of key frames A and B respectively; S represents the similarity. ij =B ij When S=0, it is considered that the transformation matrices of the two key frames are consistent.

[0099] When actually deploying a visual SLAM system, the movement of the camera is often smooth and has a certain translation. Based on this assumption, this algorithm introduces the concept of camera pose similarity, assuming that at a certain moment, the poses of the camera at different times have a certain similarity. This moment can be several frames of images.

[0100] The specific implementation method is to set a threshold (thred), and substitute the transformation matrices of the current key frame and the previous key frame into the above formula. The calculated similarity (S) is compared with the threshold. When S > thred, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key frame initialization and recalculation of the camera pose; when S < thred, it is considered that the pose estimation of the current key frame is accurate, and the key frame is sent to the local mapping thread and inserted into the existing key frame set.

[0101] S3. Local mapping thread:

[0102] The local mapping thread is responsible for continuously and regularly processing the poses of the filtered key frames and the map points multiple times. Its main workflow is as follows:

[0103] S31. Receive instructions and information from the deep learning thread. If it is for key frame initialization, recalculate the camera pose according to the depth estimation, establish and insert a new key frame. Otherwise, directly insert the key frame.

[0104] S32. Perform map point screening and reconstruction of new map points. In the local map, eliminate map points with poor quality and reconstruct map points through new observation data.

[0105] S33. When there are more than 2 key frames in the local map, perform local bundle adjustment (BA) optimization of the local map, that is, local BA optimization. By simultaneously optimizing the poses of key frames and the positions of map points, the consistency and accuracy of the entire map are improved.

[0106] S34. Check and eliminate redundant key frames among the key frames adjacent to the current frame, and send the filtered key frames to the loop closure detection thread. That is, local key frame screening. The purpose of local key frame screening is to remove redundant key frames, reduce computational overhead, and improve the efficiency of the system. The filtered key frames will be used for loop closure detection to find possible loop closure loops.

[0107] Through the above workflow, the local mapping thread continuously processes the poses of the filtered key frames and the map points, maintains the accuracy and consistency of the map, and provides appropriate key frames for loop closure detection. This can improve the robustness and performance of the SLAM system.

[0108] S4. Loop closure detection thread:

[0109] The loop detection thread is responsible for identifying whether the current frame has similar visual content to a previous keyframe, thereby correcting the accumulated errors and improving the consistency of the map. Its workflow is as follows:

[0110] First, receive the key frames from the local mapping thread and filter out the key frames with continuity, which is the candidate frame selection.

[0111] Secondly, the Sim3 transform between the current frame and the candidate closed-loop frame is calculated.

[0112] Then, local BA is performed to further optimize the camera poses and map point positions in the essential graph.

[0113] Finally, perform global BA, using all keyframes and map points in the algorithm as optimization variables, and complete the map update.

[0114] Through the above workflow, the loop detection thread can identify closed-loop frames and correct the accumulated errors by optimizing the camera pose and the position of map points to improve the consistency of map construction. This can enhance the robustness and accuracy of the SLAM system and provide more reliable map information.

[0115] Example 2

[0116] 1. Select the working camera type. First, start the tracking thread and deep learning thread. The tracking thread extracts ORB features, while the deep learning thread obtains the depth map and prior dynamic region semantic map of the input image. The LK optical flow method is used in conjunction with the prior dynamic region to remove dynamic feature points. For the convenience of calculation, the pixels of the 3x3 area are brought into the constraint equation. In the example, the pixel movement speed threshold V is set according to the actual scene operation situation. x 、V y If u>V x or v>V y , then the feature point is considered to be a dynamic feature point; otherwise, it is a static feature point. It is determined whether the remaining static feature points fall within the prior dynamic region to further distinguish dynamic and static feature points.

[0117] 2. If it is a monocular camera, initialize it. For the monocular initialization process, bring the 3D point depth information d1 and d2 obtained by the deep learning thread into the epipolar geometry constraint d1p 1 =d2p 2 +t, and the coordinates of the target point in the image coordinate system are known 1 、p 2Therefore, the unique displacement t of the camera can be solved. If it is a binocular camera, the camera pose can be directly obtained through stereo matching. If it is an RGB-D camera, the depth data can be directly obtained from the sensor, and the camera pose can be obtained through a similar calculation process as the monocular camera mentioned above. Finally, the camera pose corresponding to each frame of the image is output to select key frames and transmit the key frames to the local mapping thread.

[0118] 3. Substitute the transformation matrix of the current keyframe and the previous keyframe into the calculation formula for the camera pose difference In the example, set the threshold S tes =0.7. If S>S tes , then the camera pose estimated by the current key frame is considered to be more accurate, and the key frame is inserted; if S tes , it is considered that the current keyframe is affected by the environment, the depth value obtained is inaccurate, and the camera pose error is large. Extract the depth estimate predicted by the corresponding neural network, recalculate the camera pose of the current keyframe and insert it into the current keyframe. Perform BA optimization, adjust the camera pose and position information corresponding to the new map point, and filter and remove redundant keyframes.

[0119] 4. Perform loop detection and loop correction. Loop detection uses bag of words to detect loop key frames, and then uses the sim3 algorithm for similarity transformation; loop correction performs global BA optimization of intrinsic images.

[0120] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.​

Claims

1. A dynamic visual SLAM method with single-threaded semantics and depth estimation, It is characterized in that The following steps are involved: S1. Tracking thread: The tracking thread is responsible for real-time tracking of the camera's pose. By tracking feature points between consecutive frames, the camera's pose, i.e., the position and direction of the camera in three-dimensional space, is estimated in real time through motion estimation methods, and the local map is managed at the same time. S2, deep learning thread: The deep learning thread is responsible for providing semantic information to the tracking thread; semantic information refers to the segmented image of a specified dynamic object or organism, that is, the prior dynamic area; The deep learning thread is also responsible for selecting keyframes for the local mapping thread by combining depth estimation with the camera pose corresponding to the keyframes; depth estimation refers to generating a depth map relative to the camera for each pixel in the input image; S3, local mapping thread: The local mapping thread is responsible for continuously and regularly processing the poses and map points of the filtered key frames multiple times; S4, loop detection thread: The loop detection thread is responsible for identifying whether the current frame has similar visual content to a previous keyframe, thereby correcting the accumulated errors and improving the consistency of the map.

2. The method according to claim 1, It is characterized in that The step S1 comprises the following steps: S11, receiving an input image and extracting ORB feature points of each frame of the image. S12. Decide whether to perform initialization pose estimation based on the camera type. Monocular cameras need to use ORB feature points and semantic information from the deep learning thread to perform initialization pose estimation to obtain the camera's initial pose and map scale. Stereo cameras and RGB-D cameras directly obtain the camera's pose and map scale through stereo matching and sensor data. Regardless of the camera type, the semantic information of the deep learning thread needs to be used to remove dynamic feature points. S13, performing local map tracking in real time; calculating the camera motion of the current frame relative to the previous frame by a motion estimation method; S14. Screen new key frames in real time; by analyzing the motion between the current frame and the most recent key frame, determine whether the current frame should be selected as a new key frame, and use it as input for subsequent threads.

3. The method according to claim 2, It is characterized in that In step S12, the dynamic feature point elimination algorithm process adopted includes: S121. In the tracking thread, firstly extract ORB feature points from the input image; and at the same time, generate a priori dynamic region of the input image in the deep learning thread; S122, using the Lucas-Kanade optical flow method to predict the position of a fixed point in a two-dimensional space at different times; calculating the movement speed of the two-dimensional point by the position difference of the two-dimensional point between two frames of images; comparing the movement speed with the speed threshold, planning the point greater than the threshold as a dynamic feature point, and planning the point less than the threshold as a static feature point; S123, using semantic information, that is, the prior dynamic region, further filtering dynamic feature points in the static feature points; if a static feature point is within the prior dynamic region, marking it as a dynamic feature point and removing it; if a static feature point is outside the prior dynamic region, retaining the static feature point.

4. The method according to claim 1, wherein, S2 in the deep learning thread includes the following steps: S21, receiving an input image and performing inference using the BiSeNetV2 network to output the depth estimation and semantic information of the image; the depth estimation provides the distance information between each pixel and the camera, while the semantic information is used to specify the segmentation image of dynamic objects or organisms; S22, transmitting the semantic information to the tracking thread, and the tracking thread uses this semantic information as a prior dynamic region to further eliminate dynamic feature points and improve the accuracy of camera pose tracking; S23, at the same time, the deep learning thread receives the key frames created by the tracking thread; through the key frame screening algorithm, it is decided whether to use this key frame for initialization in subsequent threads or insert it into the existing key frame set.

5. The method according to claim 4, wherein, The BiSeNetV2 neural network algorithm adopted in step S21: A semantic guidance depth branch is introduced into the BiSeNetV2 network, and a corresponding loss function is introduced; the expression of the loss function - Lossm is as follows: In the formula, s and d represent the semantic information and depth information output by the network respectively; x and y represent the horizontal and vertical pixels of the image respectively; and They represent the horizontal gradient and vertical gradient of the semantic graph predicted by the network respectively; and point respectively represent the horizontal gradient and vertical gradient of the network-predicted depth; The network depth branch and the semantic branch respectively adopt the Huber loss and the standard softmax cross-entropy loss; they are represented by Lossdeep and Lossseg respectively, and the specific expressions are as follows: In the formula, yd represents the depth value of the pixel point predicted by the network; N and Nc respectively represent the total number of pixels and the number of pixels with GT category c; yc and pc respectively represent the true category and the pixel category predicted by the network; Therefore, the specific expression of the total loss function Loss adopted by this network is as follows: Loss = 0.45 * Lossseg + 0.45 * Lossdeep + 0.1 * Lossm.

6. The method according to claim 4, wherein, The key frame screening algorithm adopted in step S23 is: Calculating the difference in camera pose between two key frames in the deep learning thread, setting a similarity threshold, and judging whether key frame initialization work is required; the calculation formula is as follows: In the formula, Aij and Bij are two matrices A and B of size i*j, representing the transformation matrices of key frames A and B respectively; S represents the similarity. When Aij = Bij, S = 0, and it is considered that the transformation matrices of the two key frames are the same; Introducing the concept of camera pose similarity, assuming that at a certain moment, the poses of the camera at different times have a certain similarity; Setting a threshold (thred), and substituting the transformation matrices of the current key frame and the previous key frame into the above formula; comparing the calculated similarity (S) with the threshold; when S > thred, it is considered that the pose estimation of the current key frame is inaccurate, and the depth estimation of the neural network is sent to the local mapping thread for key frame initialization and recalculating the camera pose; when S < thred, it is considered that the pose estimation of the current key frame is accurate, and this key frame is sent to the local mapping thread to insert it into the existing key frame set.

7. The method according to claim 1, It is characterized in that The workflow of step S3 is as follows: S31, receiving deep learning thread instructions and information; if key frame initialization is performed, recalculating the camera pose according to the depth estimation, creating and inserting a new key frame; otherwise, directly inserting the key frame; S32, performing map point screening and new map point reconstruction; in the local map, removing map points of poor quality, and reconstructing the map points through new observation data; S33, when the number of key frames in the local map is greater than 2, BA optimization of the local map is performed, i.e., local BA optimization; by simultaneously optimizing the pose of the key frames and the positions of the map points, the consistency and accuracy of the entire map are improved; S34, checking and removing redundant key frames in key frames adjacent to the current frame, and sending the filtered key frames to the closed-loop detection thread, i.e., local key frame filtering; the filtered key frames will be used for closed-loop detection to find possible closed-loop circuits.

8. The method according to claim 1, It is characterized in that The workflow of step S4 is as follows: First, receive the key frames from the local mapping thread and filter out the key frames with continuity, i.e., candidate frame selection; Secondly, the Sim3 transform between the current frame and the candidate closed-loop frame is calculated; Then, local BA is performed to further optimize the camera pose and map point positions in the essential graph; Finally, global BA is performed, using all keyframes and map points in the algorithm as optimization variables; and the map update is completed.

Images