A dynamic scene visual slam optimization method based on semantic and geometric constraints

Abstract

The application discloses a dynamic scene visual SLAM optimization method based on semantic and geometric constraints, and has the following steps: on the basis of ORB-SLAM3, a new feature point and descriptor extraction module, a semantic segmentation module and a geometric detection module are integrated; the feature point and descriptor extraction module uses HFNet to replace traditional feature extraction, generates local and global descriptors; semantic segmentation is performed on an input image through the semantic segmentation module to obtain semantic information of an object; then, a feature point is tracked through Lucas-Kanade optical flow, and motion consistency detection is performed to accurately remove dynamic feature points; a frame inter-rotation estimation method is used to determine a rotation angle between continuous frames; and a more optimal local descriptor is selected to improve the robustness of a SLAM system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot positioning and navigation, and more particularly to a dynamic scene visual SLAM optimization method based on semantic and geometric constraints. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) is essential for robot vision, which helps camera pose estimation and mapping of unknown environments. Visual SLAM systems use cameras as sensor inputs to extract image information for localization and mapping. It can locate its position in real time in an unknown environment and simultaneously construct an environment three-dimensional map. It is a very key technology in the field of computer vision and robotics, and has a wide range of applications.

[0003] Most of the current SLAM frameworks are based on static assumptions, however, in real-world scenarios, moving objects are inevitable, which limits the application of SLAM systems in real-world scenarios. Most SLAM methods are based on feature points, however, feature-based visual SLAM systems are severely affected by the quality of feature extraction. First, the traditional extracted features are less robust to changes in lighting conditions, and the global features extracted using the bag of words (BOW) will destroy the spatial information in the scene and reduce the loop closure performance. Second, dynamic feature points in dynamic environments can significantly interfere with the accuracy and robustness of the system. Some excellent works have begun to use deep learning to solve these potential problems, some researches use deep learning to extract feature points and obtain better tracking accuracy, some people use deep learning derived semantic information to remove dynamic feature points to improve the robustness of the system, but the CNN-based feature extraction network will cause the problem of poor rotation robustness. In addition, in Crowd-SLAM, the authors found that removing too many feature points can reduce accuracy. Therefore, improving the poor system robustness caused by deep learning and reducing the interference of dynamic objects on the system are of great significance to improve the robustness of the SLAM system. SUMMARY

[0004] The technical problem to be solved by the present application is how to solve the problem of SLAM system robustness and accuracy decline caused by dynamic objects, poor robustness of traditional feature extraction method in environment with changing lighting conditions, and poor rotation robustness of feature extraction network based on deep learning.

[0005] To solve the above technical problems, the application provides the following technical solutions: a semantic and optical flow constraint-based simultaneous localization and mapping method, which performs semantic segmentation on input images through a semantic segmentation module to obtain semantic information of initial objects, then tracks feature points through a Lucas-Kanade optical flow and performs motion consistency detection to accurately remove dynamic feature points, determines the rotation angle between consecutive frames through an inter-frame rotation estimation method, and selects a more optimal descriptor to improve the robustness of the SLAM system. The method comprises the following steps:

[0006] Step 1: obtaining an RGB image and a depth image of an image frame through an RGB camera;

[0007] Step 2: inputting the RGB image obtained in step 1 into a semantic segmentation module and an HF-Net feature extraction module respectively;

[0008] Step 3: performing semantic segmentation on the RGB image obtained in step 1 through a semantic segmentation network YOLACT++ to obtain semantic information of feature points;

[0009] Step 4: converting the RGB image obtained in step 1 into a gray image, constructing an image pyramid, and extracting local features and generating HF-Net descriptors at each layer through the HF-Net;

[0010] Step 5: tracking feature points through a Lucas-Kanade optical flow and performing motion consistency checking to make an initial evaluation on the motion state of the feature points;

[0011] Step 6: combining the semantic information of the objects and the initial evaluation result of the motion consistency detection to accurately remove dynamic feature points;

[0012] Step 7: estimating the inter-frame rotation angle through an inter-frame rotation estimation method to determine whether to use the descriptor of the HF-Net or to recalculate the ORB descriptor;

[0013] Step 8: calculating the Euclidean distance between a new key frame and key frames stored in a key frame library, and selecting the key frame with the minimum Euclidean distance as a candidate key frame;

[0014] Step 9: performing subsequent local mapping and loop detection on the image frame from which the dynamic points are removed.

[0015] Further, the semantic segmentation network in step 3 adopts YOLACT++, YOLACT++ is added to the YOLACT framework on the basis of deformable convolution; adopt a more reasonable proportion, allocation strategy and anchor, so that each anchor can be better allocated to the correct target, and the use of anchor is more intensive, which can improve the detection and segmentation ability of small target objects, and introduce the compression-excitation mechanism (Squeeze-and-Excitation, SE) on the basis of reducing the FC layer, while ensuring the detection accuracy, improve the calculation accuracy. The semantic information of the object is obtained by performing semantic segmentation on the input image through the YOLACT++ semantic segmentation network.

[0016] Further, in step 5, Lucas-Kanade optical flow tracking feature points are used for motion consistency detection, and the motion state of the feature points in the image is preliminarily judged. First, the LK optical flow method is used to track the feature points of the previous frame extracted by the HFNet, and the optical flow vector is obtained according to the result of the current frame. The points successfully tracked in the previous frame and the current frame are marked as P1 and P2, which are recorded as P1=[u1 v1 1] P2=[u2 v2 1], wherein u and v represent pixel coordinates, and the RANSAC algorithm is applied to filter out abnormal optical flow values. The polar line L1 is calculated:

[0017]

[0018] wherein L1 represents the polar line corresponding to the pixel point P1 in the previous frame image, representing the three components of the polar line in the form of a vector, F is the fundamental matrix between the previous frame and the current frame, P1 represents the matching point corresponding to the successfully tracked point in the current frame image in the previous frame image, representing the homogeneous coordinates of the successfully tracked point in the previous frame image.

[0019] The distance D between the pixel point in the current frame and its corresponding polar line is calculated,

[0020]

[0021] wherein D represents the distance of P2 to its corresponding limit, P2 represents the point matched in the previous frame image in the current frame image, P1 represents the corresponding matching point in the previous frame image, F is the fundamental matrix between the previous frame and the current frame, X represents the first dimension parameter of the polar line vector, and Y represents the second dimension parameter of the polar line vector. If the value of D is greater than the threshold value δ1, the threshold value is set according to the size of the environment light, it is considered to be a potential dynamic point.

[0022] Further, in step 6, Lucas-Kanade optical flow is used to track feature points, motion consistency detection is performed, and semantic information obtained by the semantic segmentation network YOLACT++ is used to accurately remove dynamic feature points. The specific steps are as follows:

[0023] After preliminary screening of dynamic feature points and static feature points in the current frame by using the optical flow method and motion consistency detection, the SLAM system divides the target detection frame into 9 regions, the overall target detection frame is called a mother detection frame, and the 9 divided regions are called sub-detection frames. According to whether the proportion of dynamic feature points and static feature points is greater than a threshold value δ2, the initial threshold value is 0.5, each frame is evaluated and classified as a dynamic sub-detection frame or a static sub-detection frame. If the system identifies a dynamic sub-detection frame, in order to ensure that there is no missing potential dynamic area in the mask, the adjacent sub-detection frame is adjusted to a dynamic state. If most of the sub-detection frames of the target detection are dynamic sub-detection frames, it is considered that the object in the target detection frame is a high dynamic object, and then all are removed.

[0024] Further, in step 7, the inter-frame rotation angle is estimated by the inter-frame rotation estimation method, and it is determined whether to use the local descriptor of HFNet or to recalculate the ORB descriptor. The specific steps are as follows:

[0025] The least squares method is used to optimize the inter-frame rotation estimation with the optical flow vector obtained in step 5. First, the perpendicular bisector of each optical flow vector is calculated

[0026] ax+by+c=0

[0027] In the formula, a, b, and c represent the coefficients corresponding to the line, and x and y represent the points on the straight line of the optical flow vector, including the two end points of the optical flow vector.

[0028] The center of the image is selected as the starting point of optimization, because most of the inter-frame rotations usually occur near the center of the image, which can greatly reduce the optimization time. The optimization function f(x, y) is constructed and solved,

[0029]

[0030] In order to save the time of iteration, BFGS optimization is used, and the gradient of the objective function is calculated as follows:

[0031]

[0032] In the formula, represents the partial derivative of f(x, y) in the x direction, represents the partial derivative of f(x, y) in the y direction,

[0033]

[0034]

[0035] Each update yields the optimal point P, and each update of P occurs in a specified direction, enabling f(x,y) to converge quickly. The updated f(x,y) is shown below:

[0036] f(x k +α k p k,x ,y k +α k p k,y ).

[0037] In the formula, p k,x and p k,y The direction of iteration is determined by the following formula:

[0038]

[0039] Hessian matrix It reflects the local curvature information of the objective function near the most recent iteration point, providing a more accurate descent direction. The formula is calculated by the following equation:

[0040]

[0041] In the formula, s k Represents the change vector, y k ρ represents the gradient change vector. k This represents the update range.

[0042] s k =(x k+1 -x k ,y k+1 -y k )

[0043]

[0044]

[0045] After iterative optimization, f(x) k ,y k The optimal point P is obtained as the frame rotation center. If the motion between two frames is relatively close to the rotation of one frame, then the distance from point P to all endpoints of the optical flow vector should be the same. The distance from one endpoint P1 of the optical flow vector to P and the distance from one endpoint P2 of the optical flow vector to OP will have a certain distance difference D. If the distance difference D is too large, the vector will be filtered out.

[0046] || (P-P1) ||2-|| (P-P2) ||2≤D

[0047] where P represents the frame rotation center, P1 and P2 represent two endpoints of the optical flow vector. If too many vectors are discarded, it means that the motion between the two frames is not significant involving frame rotation. Then the angles from both ends to the frame rotation center of the remaining vectors are used to estimate the rotational motion between the current frame and the previous frame.

[0048]

[0049] where P 1i and P 2i represent two endpoints of each optical flow vector, and the threshold angle is set to 20 degrees. If the angle is greater than the threshold angle, the system considers that the descriptor generated by HFNet should not be used due to potential frame rotation. Instead, it recalculates the ORB descriptor of the feature points between the current frame and the previous frame and adopts the same method as ORB-SLAM3 for subsequent matching. Conversely, if the angle is smaller, the descriptor in HFNet is used.

[0050] Further, in step 7, when using the descriptor in HFNet, BOW is still used to accelerate matching, but the distance calculation is changed from Hamming distance to Euclidean norm as follows:

[0051] Gdist = || gdes1 - gdes2 ||2

[0052] where gdes1 and gdes2 represent the matched descriptors; therefore, when the system estimates that the scene has rotated, the HFNet descriptor is selected for the feature points on the previously extracted keyframes according to the estimated angle, or the ORB descriptor is recalculated, and the matching calculation method is selected according to whether rotation is detected. This effectively combines the rotation robustness of traditional extraction methods and the precision advantage of deep learning, enabling the system to operate normally in cases where significant frame rotation can cause a decline in matching performance and tracking failure.

[0053] Further, in step 8, the Euclidean distance between the new keyframe and the keyframes stored in the keyframe library is calculated, and the keyframe with the smallest Euclidean distance is selected as the candidate keyframe. Specifically:

[0054] When the local mapping thread receives a new keyframe, it calculates its global descriptor vector, denoted as gobaldes, and saves the global descriptor of the current keyframe to the keyframe library. Then the Euclidean norm (2-norm) between the global descriptor vectors of the current keyframe and other keyframes in the library is calculated:

[0055] Gdist = || gdes1 - gdes2 ||2

[0056] In the formula, gdes1 and gdes2 represent the global descriptors extracted by HFNet, each descriptor consisting of 4096 floating points. After obtaining the global descriptors of the key frame, the system calculates the distance with all the global descriptors stored in the key frame library. The smaller the Gdist is, the higher the similarity between the two frames is, thus increasing the possibility of loop closure detection. The system selects the frame with the highest similarity as the candidate frame according to the similarity of all the descriptors in the key frame library.

[0057] Further, only the image frames using static features in step 9 are used for subsequent local mapping and loop closure detection.

[0058] Compared with the prior art, the present application has the beneficial effects that: the present application removes dynamic feature points by combining geometric consistency detection and semantic information, ensures the accurate removal of dynamic feature points, and improves the accuracy and robustness of the visual SLAM system. A new feature extraction and matching mechanism is added, local feature extraction and global feature extraction are performed using HFNet, which can significantly improve the tracking and loop closure detection performance of the visual SLAM system. The inter-frame rotation angle is estimated by the inter-frame rotation estimation method to determine whether to use the HFNet descriptor or re-extract the ORB descriptor, which effectively combines the rotation robustness of the traditional extraction method and the accuracy advantage of deep learning, enabling the deep learning-based system to operate normally in the case where significant frame rotation may cause a decline in matching performance and tracking failure. The robustness of the system in complex environments is ensured. In addition, in loop closure detection, the global descriptors generated by HFNet replace the original BOW method, which can solve the problem of limited scene description ability and easy loss of spatial information about the depicted object based on the BOW method. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the overall flowchart of the present application;

[0060] Figure 2 is the example segmentation network framework diagram of the present application;

[0061] Figure 3 is the process diagram of removing dynamic feature points of the present application. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0063] As Figure 1 shown is an embodiment of a dynamic scene visual SLAM optimization method based on semantic and geometric constraints, comprising the following steps:

[0064] Step one: the present example first acquires RGB images and depth images through an RGBD camera, providing image frames for the HF-Net network and YOLACT++ segmentation network. The entire method adopts two branches for parallel processing. Branch one performs semantic segmentation on the RGB image to obtain object semantic information, and branch two extracts features through HF-Net and generates local and global descriptors,

[0065] Step two: as Figure 2 shown is the YOLACT network structure diagram. In order to improve the detection ability of small target objects, the present embodiment adopts YOLACT++ as the segmentation network. The input image is subjected to semantic segmentation through the YOLACT++ semantic segmentation network to obtain object semantic information;

[0066] Step three: the input RGB image is converted into a gray image, and an image pyramid is constructed. HF-Net extracts local features and generates HF-Net descriptors at each layer;

[0067] Step four: Lucas-Kanade optical flow tracking feature points are used to perform motion consistency detection, and the motion state of the feature points in the image is preliminarily judged,

[0068] S4.1: according to the result of the current frame, the optical flow vector is obtained, and the successfully tracked points in the previous frame and the current frame are marked as P1 and P2, which are recorded as P1=[u1 v1 1]P2=[u2 v21], wherein u and v represent pixel coordinates, and the RANSAC algorithm is applied to filter out abnormal optical flow values, and the polar line L1 is calculated as:

[0069]

[0070] wherein L1 represents the polar line corresponding to the pixel point P1 in the previous frame image, represents the three components of the polar line in the form of a vector, F is the fundamental matrix between the previous frame and the current frame, P1 represents the matching point corresponding to the successfully tracked point in the current frame image in the previous frame image, represents the homogeneous coordinates of the successfully tracked point in the previous frame image.

[0071] S4.2: the distance D between the pixel point in the current frame and its corresponding polar line is calculated,

[0072]

[0073] In the formula, D represents the distance of P2 to its corresponding limit, P2 represents the point matched in the previous frame image in the current frame image, P1 represents the corresponding matching point in the previous frame image, F is the basis matrix between the previous frame and the current frame, X represents the first dimension parameter of the epipolar vector, and Y represents the second dimension parameter of the epipolar vector. If the D value is greater than a threshold value δ1, the threshold value is set according to the size of the light of the environment, it is considered to be a potential dynamic point.

[0074] Step five: as shown in Figure 3 The flow chart for removing dynamic feature points is shown in FIG. 5. First, the optical flow-epipolar method is used to preliminarily screen the dynamic feature points and the static feature points of the current frame. The target detection frame is divided into 9 regions. The overall target detection frame is referred to as a mother detection frame, and the 9 divided regions are referred to as sub-detection frames. In combination with the semantic information of the feature points, whether the proportion of the dynamic feature points and the static feature points is greater than a threshold value δ2 is evaluated, and the initial threshold value is set to 0.5. Each frame is classified as a dynamic sub-detection frame or a static sub-detection frame. If the system identifies a dynamic sub-detection frame, in order to ensure that there is no omission of potential dynamic regions in the mask, the adjacent sub-detection frame is adjusted to be in a dynamic state. If most of the target detection frames are dynamic sub-detection frames, it is considered that the object in the target detection frame is a high dynamic object, and then all the dynamic feature points in the target detection frame are removed.

[0075] Step six: estimate the inter-frame rotation angle by using the inter-frame rotation estimation method, and determine whether to use the local descriptor of the HFNet or to recalculate the ORB descriptor;

[0076] S6.1: Calculate the perpendicular bisector of each optical flow vector

[0077] ax+by+c=0

[0078] In the formula, a, b, and c represent the coefficients corresponding to the line, and x and y represent the points corresponding to the line, including the two end points of the optical flow vector.

[0079] S6.2: Select the image center as the starting point of optimization, construct and solve the optimization function f(x,y),

[0080]

[0081] S6.3: Calculate the gradient of the objective function by using the BFGS optimization:

[0082]

[0083] Each update can obtain an optimal point P. Each update of P is performed in a specified direction, so that f(x,y) can quickly converge. The updated f(x,y) is as follows:

[0084] f(x k +αk p k,x ,y k +α k p k,y ).

[0085] In the formula, p k,x and p k,y The direction of iteration is determined by the following formula:

[0086]

[0087] Hessian matrix It reflects the local curvature information of the objective function near the most recent iteration point, providing a more accurate descent direction. The formula is calculated by the following equation:

[0088]

[0089] In the formula, s k Represents the change vector, y k ρ represents the gradient change vector. k This represents the update range.

[0090] s k =(x k+1 -x k ,y k+1 -y k )

[0091]

[0092]

[0093] S6.4: After iterative optimization, f(x) k ,y k The optimal point P is obtained as the frame rotation center. If the motion between two frames is relatively close to the rotation of one frame, then the distance from point P to all endpoints of the optical flow vector should be the same. The distance from one endpoint P1 of the optical flow vector to P and the distance from one endpoint P2 of the optical flow vector to OP will have a certain distance difference D. If the distance difference D is too large, the vector will be filtered out.

[0094] ||(P-P1)||2-||(P-P2)||2≤D

[0095] In the formula, P represents the frame rotation center, and P1 and P2 represent the two endpoints of the optical flow vector. If too many vectors are discarded, it means that there is no significant frame rotation between the two frames.

[0096] S6.5: Estimate the rotational motion between the current frame and the previous frame by using the angles from the two ends of the remaining vectors to the frame rotation center.

[0097]

[0098] P 1i and P 2i represent the two end points of each optical flow vector; the threshold angle is set to 20. If the angle is larger than the threshold angle, the system considers that due to potential frame rotation, the descriptor generated by HFNet should not be used. Instead, it re-computes the ORB descriptor of the feature points between the current frame and the previous frame, and adopts the same method as ORB-SLAM3 for the following matching. On the contrary, if the angle is smaller, the descriptor in HFNet is used.

[0099] Step seven: select the HFNet descriptor for the feature points on the previously extracted keyframes according to the estimated angle, or re-extract the ORB descriptor, and select the matching calculation method according to whether rotation is detected.

[0100] Step eight: when the local mapping thread receives a new keyframe, it calculates its global descriptor vector, denoted as gobaldes, and saves the global descriptor of the current keyframe to the keyframe library.

[0101] Step nine: then calculate the Euclidean norm (2-norm) between the global descriptor vector of the current keyframe and the global descriptor vector of other keyframes in the library:

[0102] Gdist = |gdes1-gdes2||2

[0103] In the formula, gdes1 and gdes2 represent the global descriptors extracted by HFNet, each descriptor consisting of 4096 floating-point numbers.

[0104] Step ten: after obtaining the global descriptor of the keyframe, calculate the distance with all the global descriptors stored in the keyframe library. The smaller the Gdist, the higher the similarity between the two frames, thus increasing the possibility of loop closure detection. The system selects the frame with the highest similarity as the candidate frame according to the similarity of all descriptors in the keyframe library.

[0105] Step eleven: use image frames containing only static features for subsequent local mapping and loop closure detection.

[0106] The above method effectively eliminates dynamic feature points by combining semantic information and geometric detection, determines the rotation angle between consecutive frames through the frame rotation estimation method, selects a more suitable local descriptor, and improves the robustness of the SLAM system, thus meeting the needs of visual SLAM in actual scenarios.

[0107] In summary, the dynamic SLAM method based on instance segmentation and depth features provided by the application comprises the following steps: on the basis of ORB-SLAM3, a new feature point and descriptor extraction module, a semantic segmentation module and a geometric detection module are integrated, the feature point and descriptor extraction module uses HFNet instead of the traditional feature extraction, generates local and global descriptors, the input image is subjected to semantic segmentation through the semantic segmentation module to obtain an initial mask, then the Lucas-Kanade optical flow is tracked to feature points and mobile consistency checking is performed to accurately remove dynamic feature points, the frame rotation estimation method is used to determine the rotation angle between consecutive frames, and the more optimal local descriptor is selected to improve the robustness of the SLAM system. The application uses the image collected by the RGB-D camera, and through semantic segmentation, optical flow tracking and mobile consistency checking, inter-frame rotation estimation, the dynamic feature points can be accurately removed, the SLAM system has good robustness in the scene with large light changes and moving objects, and can be applied to most actual scenes.

[0108] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Any modification, equivalent replacement and improvement made on the basis of the above description for those skilled in the art should be included in the protection scope of the claims of the application.

Claims

1. A dynamic scene visual SLAM optimization method based on semantic and geometric constraints, characterized in that, Comprising the following steps: Step 1: Obtain the RGB image and depth image of the image frame through the RGB camera; Step 2: The RGB image obtained in step 1 is respectively input into a semantic segmentation module and an HF-Net feature extraction module; Step 3: The RGB image obtained in step 1 is subjected to semantic segmentation through a semantic segmentation network YOLACT++, to obtain semantic information of feature points; Step 4: The RGB image obtained in step 1 is converted into a gray image, an image pyramid is constructed, and HF-Net extracts local features at each layer and generates an HF-Net descriptor; Step 5: Track the feature points using Lucas-Kanade optical flow and perform motion consistency check to make an initial evaluation of the motion state of the feature points; Step 6: Combine the semantic information of the object and the initial evaluation result of the motion consistency check to accurately remove dynamic feature points; Step 7: Estimate the inter-frame rotation angle through geometric consistency constraint and optical flow vector, to determine whether to use the descriptor of HF-Net or recalculate the ORB descriptor; the specific steps are as follows: use the optical flow vector obtained in step 5 to optimize the inter-frame rotation estimation by least squares method, first, calculate the perpendicular bisector of each optical flow vector ; where, , , denotes the coefficient corresponding to the line, and denotes the point on the straight line of optical flow vector, including the two end points of the optical flow vector; Select the image center as the starting point of optimization, construct and solve the optimization function , In order to save the time of iteration, BFGS optimization is adopted, and the gradient of the objective function is calculated as follows: wherein denotes the partial derivative in the direction denotes the partial derivative in the direction ； . Each update gets the optimal point , Each update goes in the specified direction, making it quickly converges, and the updated is as follows: . wherein and represents the iteration direction, determined by the following equation: . Hessian matrix reflects the local curvature information of the objective function near the current iteration point, and provides a more accurate descent direction, The formula is calculated by the following formula: ; wherein represents a change vector, represents a gradient change vector, represents an update magnitude; ; ; ; through iterative optimization The optimal point P is obtained as the center of frame rotation. If the motion between two frames is relatively close to a frame rotation, the distance from P to the end point of all optical flow vectors should be the same. The distance from one end point P1 of an optical flow vector to P and the distance from one end point P2 of an optical flow vector to OP will have a certain distance difference D. If the distance difference D is too large, the vector will be filtered out. ; where P represents the center of frame rotation, and P1 and P2 represent two end points of the optical flow vector; first, filter some optical flow vectors, if too many vectors are discarded, it means that the motion between two frames has no significant frame rotation, then use the remaining vectors to estimate the rotational motion between the current frame and the previous frame from the angles of the two ends to the center of frame rotation; ; where and represent the two end points of each optical flow vector; the threshold angle is set to 20 degrees, if the angle is larger than the threshold angle, the system considers that due to potential frame rotation, the descriptor generated by HFNet should not be used, instead, it re-computes the ORB descriptor of the feature points between the current frame and the previous frame, and adopts the same method as ORB-SLAM3 for the following matching, vice versa, if the angle is smaller than the threshold angle, the descriptor in HFNet is used; Step 8: Calculate the Euclidean distance between the new key frame and the key frame stored in the key frame library, and select the key frame with the smallest Euclidean distance as the candidate key frame; Step 9: The image frame after removing dynamic points is subjected to subsequent local mapping and loop detection.

2. The method of claim 1, wherein, The semantic segmentation network adopts YOLACT++, which is based on the YOLACT framework and adds deformable convolution, adopts a more reasonable scale, allocation strategy and anchor, so that each anchor can be better allocated to the correct target, and the use of anchor is more intensive, improving the detection and segmentation ability of small target objects, introducing a compression-activation mechanism on the basis of reducing the FC layer, improving the calculation precision while ensuring the detection precision, and obtaining object semantic information through semantic segmentation of the input image by the YOLACT++ semantic segmentation network.

3. The method of claim 1, wherein, Track the feature points using Lucas-Kanade optical flow and perform motion consistency check to make an initial evaluation of the motion state of the feature points, and the specific steps are as follows: Firstly, the LK optical flow method is used to track the feature points of the previous frame extracted by HFNet. The optical flow vector is obtained according to the result of the current frame. The successfully tracked points in the previous frame and the current frame are marked as and , which are recorded as , wherein and represent pixel coordinates. The RANSAC algorithm is applied to filter out abnormal optical flow values, and the epipolar line is calculated. wherein, represents a pixel point the corresponding epipolar line in the previous frame image, represents three components of the epipolar line in vector form, F is the fundamental matrix between the previous frame and the current frame, represents the matching point in the previous frame image corresponding to the successfully tracked point in the current frame image, represents the homogeneous coordinates of the successfully tracked point in the previous frame image; the distance D between the pixel point in the current frame and the corresponding epipolar line is calculated: ; where D represents the distance to its corresponding epipole, represents the point in the current frame image that matches the point in the previous frame image, represents the corresponding matching point in the previous frame image, F is the fundamental matrix between the previous and current frame, represents the first dimensional parameter of the epipolar vector, represents the second dimensional parameter of the epipolar vector, which is considered a potential dynamic point if the D value is too large.

4. The method of claim 1, wherein, After the dynamic feature points and static feature points of the current frame are preliminarily screened by using the optical flow method and motion consistency detection, the SLAM system divides the target detection frame into 9 regions, the overall target detection frame is called a mother detection frame, the 9 divided regions are called sub-detection frames, and the proportion of dynamic feature points and static feature points is greater than a threshold value based on semantic information, the initial threshold value is 0.5, each frame is evaluated and classified as a dynamic sub-detection frame or a static sub-detection frame, if the system identifies a dynamic sub-detection frame, in order to ensure that there is no missing potential dynamic region in the mask, the adjacent sub-detection frame is adjusted to a dynamic state, if most of the target detection frame is a dynamic sub-detection frame, it is considered that the object in the target detection frame is a high dynamic object, and then all the dynamic objects are removed.

5. The method of claim 1, wherein, When using the descriptor in HFNet, BOW is still used to accelerate matching, but the distance calculation is changed from Hamming distance to Euclidean norm, as follows: ; where gdes1 and gdes2 represent the matched descriptors; thus, when the system estimates that a rotation of the scene has occurred, it selects the HFNet descriptors for the feature points on the previously extracted keyframe according to the estimated angle, or reextracts the ORB descriptors, and selects the matching computation method according to whether a rotation was detected.

6. The method of claim 1, wherein, The global descriptor generated by HFNet replaces the original BOW method, and ORB-SLAM3 uses the bag-of-words (BOW) method for loop detection, however, the BOW-based method has limited scene description ability and is easy to lose spatial information about the depicted object, when the local mapping thread receives a new key frame, it calculates its global descriptor vector, denoted as gobaldes, and saves the global descriptor of the current key frame to the key frame library, then calculates the Euclidean 2-norm between the global descriptor vector of the current key frame and other key frames in the library: ； In the formula, gdes1 and gdes2 represent global descriptors extracted by HFNet, each descriptor consisting of 4096 floating points, after obtaining the global descriptor of the key frame, the distance with all global descriptors stored in the key frame library is calculated; the smaller Gdist is, the higher the similarity between the two frames is, thereby increasing the possibility of loop closure detection, and the system selects the frame with the highest similarity as the candidate frame according to the similarity of all descriptors in the key frame library.

Images