A semantic SLAM method based on instance segmentation
By using instance segmentation and object-level matching technology in SLAM system, the problems that inter-frame constraints and object-level information are not considered in dynamic scenarios are solved, and the robustness and feature point matching accuracy of the SLAM system are improved, and it is suitable for single and double Image scene.
Patent Information
- Application Number
- CN202111497853.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-09
AI Technical Summary
The prior art does not consider inter-frame constraints and object-level information in dynamic scenarios, resulting in feature point matching errors and poor robustness.
The semantic SLAM method based on instance segmentation is adopted, object-level semantic information is extracted through the network instance segmentation module, combined with ORB feature point information, and fine-grained feature point classification is used using the KD-TREE data structure, and the KM algorithm is used for object-level inter-frame matching. Establish an Object_KeyFrame_DataBase three-dimensional data structure for loopback detection.
It improves the robustness of the SLAM system in dynamic scenarios, reduces feature point matching errors, improves loop detection efficiency, and is highly applicable, and supports dynamic object recognition in monotonous and binocular situations.
Smart Images

Figure CN114187447B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of ORB-SLAM2 system and deep learning instance segmentation, and specifically, to a semantic SLAM method based on instance segmentation. Background Art
[0002] With the continuous advancement of science and technology in contemporary society, people's demand for convenience in life is also increasing. With the rise of artificial intelligence, from small household service robots to large RoboTaxi, how to help robots better and more accurately establish the surrounding environment has become increasingly important. For simultaneous positioning and mapping tasks, visual SLAM is one of the best choices. For example, ORB-SLAM2, RGB-D SLAM-V2, etc., have been widely used due to their advantages such as fast sensor acquisition speed, low cost, short latency, and high accuracy. Taking ORB-SLAM2 as an example, the system is prone to loss in high-dynamic scenes, and because its underlying visual information uses the ORB feature descriptor, it can only compare the most basic feature points, and does not use object-level information.
[0003] With the rapid development of deep learning technology, many visual problems have better and faster solutions. Neural networks can easily identify objects of interest in an image, including their specific categories and precise boundaries. These results can combine neural networks with visual SLAM technology, combined with semantic information, to better help SLAM systems perceive the world.
[0004] In recent years, many researchers have proposed a variety of different semantic SLAM methods. The ORB-SLAM2 system proposed by Raul Mur-Artal is a complete SLAM solution based on monocular, binocular, and RGB-D feature points. It uses the bag-of-words (DBoW2) model to cluster feature points and match feature points during inter-frame matching and loop detection. Since ORB feature points are 32-dimensional 01 vectors, matching based on vector distance does not conform to human intuition. For example, the vector distances between feature points on the car and feature points on the ground are very close, and from a human perspective, these feature points will not be matched. Similarly, for loop detection, loop detection based on the DBoW2 model only considers whether the two frames being compared match in the word vectors during loop closure, and does not make any requirements for the spatial structure of the two frames.
[0005] MASK-SLAM proposed by Masaya Kaneko of the University of Tokyo is a monocular SLAM system combined with MASK-RCNN, which can effectively segment semantic-level information such as the sky and vehicles, and remove feature points belonging to dynamic objects, converting the SLAM system in dynamic scenes into a SLAM system in static scenes. However, this method is only applicable to monocular SLAM systems, and all semantic dynamic feature points are removed, without considering the different states of multiple types of objects, resulting in the loss of relevant information and the loss of camera pose, which is not applicable in some scenarios.
[0006] DynaSLAM proposed by Berta Bescos combines MASK-RCNN with ORB-SLAM2, improving the adaptability of the SLAM system to dynamic scenes. This method also gives objects prior dynamic information when using a single camera, and excludes these feature points when tracking; when using a dual camera, it uses a multi-view geometry method to calculate the angle between the current frame and the corresponding point of the reference frame. If it is greater than 30°, it is considered a dynamic point and excluded. The influence of these points is also removed when building the map, and the background repair method is used to obtain a static map. However, this method does not consider the constraints of the object on the reference point from the object level, and there are many mismatches, which will also lead to incorrect judgments about the motion state of the object.
[0007] In summary, the prior art has the problem of not considering inter-frame constraints and object-level information in dynamic scenes, resulting in feature point matching errors and poor robustness. Summary of the invention
[0008] The purpose of the present invention is to provide a semantic SLAM method based on instance segmentation to address the deficiencies in the above-mentioned prior art, so as to solve the problems in the prior art of not considering inter-frame constraints and object-level information in dynamic scenes, resulting in feature point matching errors and poor robustness.
[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0010] The present application provides a semantic SLAM method based on instance segmentation, which comprises the following steps: S1, acquiring an image sequence; S2, extracting feature point information and semantic information; S3, fusing feature point information and semantic information; S4, dynamic object detection and removal; S5, object-level frame matching; S6, object-level loop detection.
[0011] Furthermore, in step S2, the extraction of feature point information and semantic information is performed simultaneously.
[0012] Furthermore, the extraction of semantic information is done through the network instance segmentation module.
[0013] Furthermore, the network instance segmentation is object-level segmentation.
[0014] Furthermore, the network instance segmentation is combined with the MASK-RCNN network.
[0015] Furthermore, in step S2, the MASK-RCNN network is trained.
[0016] Furthermore, step S3 performs fine-grained feature point classification on the feature points inside each object in the image.
[0017] Furthermore, fine-grained feature point classification is performed through the KD-TREE data structure.
[0018] Furthermore, step S5 uses object-level matching to constrain inter-frame matching.
[0019] Furthermore, step S5 uses the KM algorithm to find the optimal match.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] First, the present invention uses a neural network to perform instance segmentation on the scene, which can effectively identify dynamic objects in both single-eye and binocular situations, thereby improving the robustness of the SLAM system in dynamic scenes;
[0022] Second, the present invention aims at the shortcoming that the DBoW2 model only focuses on vector level information and cannot distinguish feature point matching at a higher level. It proposes to use object-level matching to constrain inter-frame matching, which effectively solves the problem of feature point matching errors leading to loss in some scenarios.
[0023] Third, the present invention proposes to grid the image, associate it with semantic information, and establish a three-dimensional data structure of Object_KeyFrame_DataBase, which can effectively use the location information of the object rather than just the category information. During loop detection, the associated key frames can be efficiently retrieved, making the loop detection more efficient.
[0024] Fourth, the present invention uses the libtorch library, which can separate the training and deployment of the network, and decouple the network model from the SLAM system, making it convenient to modify the network model without affecting the direct use of the SLAM system. The method of the present invention has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram of a semantic SLAM method based on instance segmentation provided by the present invention;
[0026] Figure 2A schematic diagram of image grid division in step S32 of a semantic SLAM method based on instance segmentation provided by the present invention;
[0027] Figure 3 A schematic diagram of a transformation method of three-dimensional map points in a binocular system in a semantic SLAM method based on instance segmentation provided by the present invention;
[0028] Figure 4 It is a schematic diagram of a key frame association method based on a bag-of-words model in the prior art;
[0029] Figure 5 The spatial structure of a three-dimensional keyframe database KeyFrame DataBase in the semantic SLAM method based on instance segmentation provided by the present invention;
[0030] Figure 6 A flowchart of a semantic SLAM method based on instance segmentation provided by the present invention. DETAILED DESCRIPTION
[0031] In order to make the implementation process of the present invention clearer, it will be described in detail below with reference to the accompanying drawings.
[0032] Embodiment 1:
[0033] The present invention provides a semantic SLAM method based on instance segmentation, such as Figure 1 As shown, the method comprises the following steps:
[0034] S1, obtain image sequence;
[0035] The camera captures images to form an image sequence. The image sequence can be multiple images in static scenes or multiple images in dynamic scenes. The images in dynamic scenes can be multiple images with large differences or multiple images with small differences. The embodiment of the present invention uses an RGB camera to obtain an image sequence in a dynamic scene. The images captured by the camera and the timestamp information released by the ROS system are passed into the SLAM system. Since the ORB feature point extraction needs to be performed on a grayscale image, in the SLAM system, the image is first converted into a grayscale image, and then enters the ORB feature point extraction thread and the network instance segmentation thread at the same time. The difference between the input images of the ORB feature point extraction thread and the network instance segmentation thread is that the input image of the ORB feature point extraction thread needs to be converted into a grayscale image, while the input image of the network instance segmentation thread can be converted into a grayscale image or not.
[0036] S2, extract feature point information and semantic information;
[0037] The extraction of feature point information is completed by the ORB feature point extraction module, and the extraction of semantic information is completed by the network instance segmentation module, wherein the network instance segmentation is an object-level segmentation, that is, it can distinguish different objects of the same or different types. Compared with the semantic segmentation that can only distinguish types, the object-level segmentation is closer to the actual situation and has more application value. The extraction of feature point information and semantic information is performed simultaneously, that is, the following steps S21 and S22 are performed simultaneously.
[0038] S21, extracting feature point information of the image using an ORB feature point extraction module;
[0039] ORB feature points are composed of key points and feature descriptors. Its key points are also called "Oriented FAST", which is an improved form of FAST corner points. Feature descriptors are also called BRIFE. Among them, FAST corner points are known for their fast speed. It only needs to judge whether a pixel point has a large difference in pixel value with its surrounding pixels, so many feature points can be quickly extracted. In order to avoid the problem of excessive concentration of feature points, non-maximum suppression should be performed afterwards. Because FAST corner points do not have scale invariance and rotation invariance, the ORB feature point extraction module adopted by the present invention introduces feature pyramids and grayscale centroid methods to solve the above problems. It has scale invariance and rotation invariance, which can effectively cope with the changes in the position, scale, and direction of feature points in the image brought about by motion scenes, and is closer to the actual scene. For feature descriptors, the BRIFE descriptor is a binary descriptor, and its description vector consists of 01, which encodes two random pixels near the key point. If the former is large, 1 is taken, otherwise 0 is taken. Due to the use of binary expression, its calculation speed and storage are very fast.
[0040] Due to the consideration of rotation and scaling, as well as its fast extraction capability, ORB feature points can better meet practical needs and are widely used in SLAM systems.
[0041] S22, using the network instance segmentation module to extract object-level semantic information of the image.
[0042] The network instance segmentation of the image is achieved through the MASK-RCNN network. The MASK-RCNN network obtains category information, bounding box information, and MASK mask information. This information is used to filter and classify feature points to extract object-level semantic information. The steps to build the MASK-RCNN network are as follows:
[0043] S221, build the MASK-RCNN network;
[0044] The specific construction steps are as follows:
[0045] Step 1: Build an FPN network with ResNet101 as the backbone network as the feature extraction part. It is mainly composed of four scales, and the length and width between each two are twice the ratio. Its main purpose is to extract feature maps of different scales for use in the RPN network and subsequent tasks.
[0046] Step 2: Build the RPN network, receive the multi-scale feature maps output by the FPN network, assign anchor boxes of different sizes to each pixel position and distinguish between the foreground and background to determine whether there is a target. Then regress the proposed target box through ROI Align to obtain a more accurate target box position.
[0047] Step 3: Input the feature map within the target box extracted by the RPN network into the target detection branch and the mask prediction branch to complete the tasks of classification, positioning, and mask acquisition.
[0048] S222, train the built MASK-RCNN network;
[0049] For different tasks, different datasets are used to train the MASK-RCNN network, which can improve the accuracy of the MASK-RCNN network in completing classification tasks in different tasks, so as to extract more accurate object-level semantic information. Use PyTorch and local datasets to train the MASK-RCNN network. Local datasets include KITTI dataset and TUM dataset; the local dataset for training the MASK-RCNN network in outdoor scenes uses the KITTI dataset, which is a set of datasets jointly established by Karlsruhe Institute of Technology and Toyota Technological University Chicago for a variety of computer vision tasks, including vehicles, road signs and other target information; the local dataset for training the MASK-RCNN network in indoor scenes uses the TUM dataset, which is a set of continuous indoor images collected by the Technical University of Munich in Germany using a depth camera, including computers, tables, chairs and other target information.
[0050] The loss function is used to determine whether the training of the MASK-RCNN network is complete. The MASK-RCNN network is divided into three branches and has three outputs, so the corresponding loss function consists of three parts:
[0051] L=L cls +L box +L mask
[0052] Where L cls is the loss function for classification, the specific form is:
[0053] L cls(pi,pi*)=-log(pipi*+(1-pi)(1-pi*))
[0054] pi represents the probability of the current category, and pi* represents the probability of not being the current category;
[0055] Where L box is the loss function of the bounding box, and its specific form is:
[0056] L box =smooth L1 (ti-ti*)
[0057] ti is the predicted bounding box position, ti* is the true value;
[0058] Where L mask The loss function of the mask is in the form of:
[0059] L mask =-(tlog(o)+(1-t)log(1-o))
[0060] t is the true value and o is the predicted value.
[0061] Positive samples are bounding boxes with an intersection-over-union (IOU) greater than 0.6, and negative samples are bounding boxes with an IOU less than 0.6. During training, the ratio of positive to negative samples is set to 1:3, and the training is completed when the loss function converges.
[0062] S223, calling the MASK-RCNN network trained in step S222 in the SLAM system. Use libtorch to save the trained MASK-RCNN network as a script file and load it in the SLAM system.
[0063] S3, integrating feature point information and semantic information; the specific steps are as follows:
[0064] S31, traverse all the extracted feature points, obtain the target information of the corresponding position on the corresponding MASK mask through its two-dimensional coordinate information p(x, y), and add its index Index to the data structure of the corresponding object. Each object corresponds to a data structure.
[0065] S32, the KD-TREE data structure is used to perform fine-grained feature point classification on the feature points inside each object, that is, the description vector of the feature points is hierarchically classified as fine-grained feature point classification, and the vector information of the feature points inside the object is fine-grained, so that many unnecessary judgments can be removed during feature point matching, thereby accelerating the matching process. At the same time, the current image is gridded, such as Figure 2As shown, it is divided into 28*32 grid images. A semantic category is assigned to each grid according to the semantic information corresponding to the MASK mask, and an object-level key frame database is established for loop detection. The specific association method is introduced in S61.
[0066] S4, dynamic object detection and removal;
[0067] The present invention is a system that combines monocular and binocular. When the monocular inputs one image at a time, the initialization information is less; when the binocular inputs two images at a time, the initialization information is more. The present invention can effectively identify dynamic objects in both monocular and binocular situations, and improves the robustness of the SLAM system in dynamic scenes.
[0068] S41, monocular system;
[0069] Due to the scale uncertainty of the monocular system, the distance information of the feature points cannot be obtained through a single frame of information. Therefore, the prior semantic category is used to filter and remove the prior dynamic objects. All objects of a category are removed by semantic category. For example, when indoors, if the category of the object is human, the category of human is removed; when outdoors, if the category of the object is car, the category of car is removed. This will lead to the recognition accuracy of the SLAM system in some scenarios, such as parking lots.
[0070] S42, binocular system.
[0071] S421, obtain the correspondence between the feature points of the left and right views by row scanning. Count the feature points of each row of the right image and match them using stereo matching. Then search the feature point pi of the i-th row of the left image on the i-th row of the right image to obtain the best matching point qi, where i represents the number of any row, and p and q represent the feature points on the left and right sides of the image, respectively. Take qi as the center and r as the radius for further search, with a search radius of 10 pixels, and perform sub-pixel interpolation optimization on the results to obtain more accurate matching points.
[0072] S422, triangulate the feature points by corresponding feature points, obtain the feature points corresponding to the left and right views after triangulation, and obtain the distance by the two rays from the optical center to the feature points in space, that is, obtain the distance information of the feature points. Through triangulation, the two-dimensional points on the left and right views are converted into points in the established three-dimensional space, which are called three-dimensional map points.
[0073] S423, when the SLAM system is running, the frame closest to the current frame is the reference frame. The reference frame is used as reference information, and each new frame needs to be transformed with the previous reference frame. The transformation matrix between the current frame and the reference frame is calculated through the background feature points, and the three-dimensional map points corresponding to the reference frame are transformed accordingly to determine whether the map error between it and the current frame is less than the threshold. When more than 90% of the three-dimensional map point errors are greater than the threshold, the current object is considered to be a real moving object. Specifically, objects that are absolutely stationary are found through semantic information, and the three-dimensional map points corresponding to these objects are used to perform preliminary pose calculations to obtain the initial pose transformation matrix Tinit, which is a preliminary estimate of the transformation from the reference frame to the current frame. The three-dimensional map points corresponding to the current frame and the reference frame are changed through Tinit. Let Plast be the feature point corresponding to the reference frame, and Pcurr be the feature point corresponding to the current frame. If:
[0074] |Plast-Pcurr|>th
[0075] like Figure 3 As shown, C1 and C2 represent the left and right cameras respectively, the dark color represents the original camera position, the light color represents the current position, and the origin represents the map point in space. When the camera moves, if the change relationship of the map point conforms to the camera's transformation matrix, it is considered a static point. If the error is greater than the threshold, the feature point is considered a dynamic point. th is the threshold, which is determined by prior information. If 90% or more of the feature points of an object are dynamic points, it is considered to be a dynamic object, and all feature points corresponding to the dynamic object need to be removed.
[0076] S5, object-level inter-frame matching;
[0077] S51, since static objects conform to the same transformation matrix, the solution can be performed, while dynamic objects do not conform to the same transformation matrix and cannot be performed. After removing the dynamic objects in the current frame, the remaining objects are static objects that can be used for matching. Then the KM algorithm is used, and the IOU is used as the weight information of whether the objects between the two frames are the same object to find the optimal match, so that the total weight is maximized and the matching accuracy is higher. The present invention takes into account the inter-frame constraints in dynamic scenes, the feature point matching accuracy is high, and the robustness is improved.
[0078] Specifically, first, calculate the weight information W of pairwise object matching between two frames, that is, calculate the intersection-over-union value of the bounding boxes of two objects, and add a certain offset to make the matching have better translational invariance, so that the matching accuracy is relatively high, not easily mis-matched, and thus has better robustness. Then, use the KM algorithm to find a set of optimal matches. The specific method is to let V1 be the Index information of all objects in the reference frame, V2 be the Index information of all objects in the current frame, and all edges <i, j> ∈ G, where W(i, j) represents the matching weight from the i-th vertex in V1 to the j-th vertex in V2. Store the graph information using an adjacency matrix, initialize the vertex labels using a greedy algorithm, and use the Hungarian algorithm to find a complete match. If not found, modify the vertex labels, add edges and search again until a complete match is reached, that is, a one-to-one correspondence has been found, and all vertex labels in the left subset in the KM algorithm have corresponding relationships. In this way, the pairwise matching information of objects between two frames is obtained.
[0079] S52. Use the KD-TREE data structure to perform fine-grained matching of feature points within the matched corresponding objects; at the same time, traverse the word vectors of the current frame and the reference frame. When the word IDs are the same, match the feature points. For each feature point within the current word of the current frame, find its corresponding feature point, and the minimum distance Dist needs to be satisfied when searching. 1 <th and Dist 1 <0.8*Dist 2 。
[0080] S53. After successful matching, optimize the pose information of the current frame through the three-dimensional map points corresponding to the feature points in the previous frame. In the motion model, the velocity vector will be calculated as the initial value, and by fixing the three-dimensional map points, the pose of the current frame is optimized using graph optimization.
[0081] S54. After successful pose optimization, project the three-dimensional map points associated with the adjacent frames of the current frame, and calculate the vector distance from the projected points to the feature points at the corresponding positions in the current frame. If the distance is less than th, a new set of matching points is considered to be found. If the total number of final matching points is greater than 30, the tracking is considered successful; otherwise, it is considered lost and re-localization is performed.
[0082] S6. Loop detection at the object level.
[0083] S61. A key frame means that every time a frame is successfully established and the tracking is successful, and the key frame has enough new information. Every time a key frame is obtained, it is necessary to enter the loop detection thread and add the current key frame information to the KeyFrame DataBase. Existing key frame association methods based on the bag-of-words model are as Figure 4As shown in the figure, ABCD are four different categories, and the current keyframe on the right contains four categories. The corresponding keyframe database KeyFrame DataBase also contains four categories, and each category is associated with some keyframes. Through this association method, similar keyframes can be found during loop matching. However, this association method does not take position information into account, and there is a certain degree of mismatching. The improved method is as follows Figure 5 As shown in the figure, dark, gray, and light colors represent three categories respectively, and white represents the background. The left side represents the information observed by the current key frame. Through grid division, different colors represent different categories. The three-dimensional data structure on the right is the key frame database KeyFrame DataBase. Its length and width are consistent with the grid division method of the image, and the depth is the number of all categories, ignoring the background. For example, if there are three categories, a data structure with a depth of 3 is established, so that different categories at different locations can be associated with different key frames, and the semantic information of all key frames at corresponding locations is saved through this three-dimensional data structure.
[0084] S62, count the maximum similarity number of all related key frames, scale it by a certain coefficient and set it as a threshold, and key frames greater than the threshold will enter the next step of screening. Specifically, count the key frames with the same category as the current frame at the corresponding position, and CommonGrids is the number of grids that match the two frames. After counting all the key frames that meet the requirements and forming related frames, count their maximum number of grids (MaxCommonGrids), and set 0.8*MaxCommonGrids as the threshold. Key frames less than the threshold will not enter the next step, that is, key frames that do not meet the requirements are removed. In this way, the key frames that are closest to the current key frame in position and category can be selected. Compared with Figure 4 As shown, the closest situation is considered through the word vector. The present invention takes the position information into consideration and has a higher screening efficiency.
[0085] S63, non-maximum suppression is performed, and the frame with the best matching effect is selected for the key frames with related relationships. Specifically, the common view information of all the frames to be selected is calculated, and the key frames with common view relationships are considered as a group, and only the key frame with the largest CommonGrids is retained in a group of key frames.
[0086] S64, perform continuity detection, after loop detection for three consecutive frames of the current frame, if the obtained key frame also has a common view relationship, it is considered that the current loop matching is successful, and the Sim3 matrix is calculated. That is, the loop matching at the image level is successful, and the transformation matrix between the two key frames is calculated to see if it is consistent with the associated three-dimensional map point. If it is consistent, the map is corrected.
[0087] Figure 6 The overall flow chart of the method of the present invention is shown.
[0088] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A semantic SLAM method based on instance segmentation, characterized in that: The method comprises the following steps: S1, acquiring an image sequence; S2, extracting feature point information and semantic information; S3, fusing feature point information and semantic information; S4, detecting and removing dynamic objects; S5, object-level frame matching; S6, object-level loop detection; The extraction of the feature point information and the semantic information in step S2 is performed simultaneously, the extraction of the semantic information is completed by a network instance segmentation module, the network instance segmentation is an object-level segmentation, the network instance segmentation is combined with a MASK-RCNN network, and the MASK-RCNN network is trained in step S2; The step S5 uses object-level matching to constrain inter-frame matching, and the step S5 uses the KM algorithm to find the optimal match; the step S5 includes the following steps: S51, first, calculate the weight information W of the pairwise matching of the two objects between the two frames, that is, calculate the intersection-and-union ratio value of the bounding boxes of the two objects and add the offset. Then, let V1 be the index information of all objects in the reference frame, V2 be the index information of all objects in the current frame, and all edges<i,j> ∈G, where W(i,j) represents the matching weight from the i-th vertex in V1 to the j-th vertex in V2; the adjacency matrix is used to store the graph information, the greedy algorithm is used to initialize the top label, and the Hungarian algorithm is used to find a complete match. If it cannot be found, the top label is modified and the edge is added to search again. When a complete match is reached, it ends, that is, a one-to-one correspondence has been found, and all the top labels in the left subset in the KM algorithm have a corresponding relationship; S52, using the KD-TREE data structure to perform fine-grained matching of feature points in the matched corresponding objects; at the same time, traversing the word vectors of the current frame and the reference frame, when the word IDs are the same, matching the feature points; S53, after the matching is successful, optimizing the pose information of the current frame through the three-dimensional map points corresponding to the corresponding feature points of the previous frame; S54, after the pose optimization is successful, the three-dimensional map points associated with the adjacent frames of the current frame are projected, and the vector distance with the feature points corresponding to the current frame is calculated. If the distance is less than th, it is considered that a new set of matching points is found. If the final total number of matching points is greater than 30, the tracking is considered successful, otherwise it is considered lost and repositioning is performed; Wherein, th is a threshold value, and the expression of the threshold value is |Plast-Pcurr| > th, wherein Plast is a feature point corresponding to a reference frame, and Pcurr is a feature point corresponding to a current frame.
2. The semantic SLAM method based on instance segmentation according to claim 1, characterized in that The step S3 performs fine-grained feature point classification on the feature points inside each object in the image.
3. The semantic SLAM method based on instance segmentation according to claim 2, characterized in that The fine-grained feature point classification is completed through the KD-TREE data structure.
