Monocular vision dynamic SLAM method based on semantics

Abstract

The invention relates to a monocular vision dynamic SLAM method based on semantics, and belongs to the technical field of computer vision. The method comprises the following steps: 1) carrying out semantic segmentation on a monocular vision image by adopting a deep learning model, and identifying an unknown motion state region and a pure static state region; 2) performing data enhancement on the original image; (3) extracting a SuperPoint feature point by using a SuperPoint network; 4) matching the SuperPoint feature points of the pure static region, and obtaining feature correspondence through confidence coefficient screening; 5) ensuring the accuracy and robustness of subsequent pose estimation through constant-speed motion model tracking, key frame reference tracking, and SuperGlue and SuperGlue-based 2D-3D tracking and repositioning; and 6) constructing a 3D sparse map of the environment. According to the method, an ORB-SLAM 2 system in a monocular camera mode is improved, and a feature point tracking algorithm is optimized.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a semantic-based monocular vision dynamic SLAM method. Background Art

[0002] Monocular SLAM (Simultaneous Localization and Mapping) technology has attracted widespread attention due to its low hardware cost and wide range of applications. This technology can simultaneously complete positioning and mapping tasks using only a single camera, and has important application value in robot navigation, augmented reality, autonomous driving and other fields. In recent years, monocular SLAM technology has aroused widespread research interest in fields such as computer vision and robotics. This technology has made great progress in both theoretical research and practical applications. However, monocular SLAM technology still faces many challenges in practical applications. Since it only relies on image information from a single camera, the system often encounters difficulties in dealing with complex and dynamic environments. Especially in the initialization stage and in scenes with drastic changes in illumination, existing algorithms often perform poorly. These problems have, to a certain extent, limited the application of monocular SLAM in a wider range of scenarios.

[0003] Specifically, the existing monocular vision SLAM methods have the following main problems:

[0004] (1) Initialization is difficult. The traditional monocular ORBSLAM 2 system requires multiple frames of image data to complete initialization, which may lead to initialization failure or delay in the case of rapid motion or drastic scene changes.

[0005] (2) Poor environmental adaptability. In degraded environments with large changes in lighting, the monocular ORBSLAM 2 is prone to feature point tracking failure, which in turn causes the entire algorithm to fail.

[0006] Therefore, how to improve the initialization efficiency and environmental adaptability of the monocular vision SLAM system has become a key issue that needs to be solved urgently. Summary of the invention

[0007] In view of this, the object of the present invention is to provide a semantic-based monocular visual dynamic SLAM method.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A semantics-based monocular visual dynamic SLAM method, the method comprising the following steps:

[0010] Step 1: Use the DeepLabV3 semantic segmentation deep learning model to perform semantic segmentation on the monocular visual image and identify areas of unknown motion state and purely static state;

[0011] Step 2, perform data enhancement on the original image, that is, generate new enhanced image data by image inversion transformation;

[0012] Step 3: For the original image and enhanced image, the SuperPoint network is used to replace the traditional ORB feature point extraction method to extract SuperPoint feature points with stronger representation ability and robustness;

[0013] Step 4: Initialization stage: Use the SuperGlue algorithm to match the SuperPoint feature points in the purely static area, combine the matching results of the original image and the enhanced image sequence, and obtain high-quality feature correspondence through confidence screening to improve the reliability of initialization;

[0014] Step 5, tracking phase, the accuracy and robustness of subsequent pose estimation are ensured through constant speed motion model tracking, key frame reference tracking, SuperGlue-based 2D-3D tracking, SuperGlue-based 2D-2D tracking and relocalization;

[0015] Step 6: construct a 3D sparse map of the environment based on the matched purely static feature points.

[0016] Further, the step 1 comprises:

[0017] 1-1 For the original image captured by the monocular camera, the DeepLabV3 deep learning model is used for semantic segmentation to divide the image into regions with different semantic labels; that is, each pixel contains corresponding semantic information;

[0018] 1-2 Based on the segmented semantic regions, each key semantic category is analyzed, and the motion characteristics are judged according to the semantic information, and they are divided into pure static state areas and unknown motion state areas.

[0019] Furthermore, step 2 includes: for the original grayscale image I(x c ,y c ) to perform inverse color transformation, where x c and c Represent the row and column coordinates of the pixel respectively, and the pixel value range is [0,255]; the enhanced image I′(x c ,y c ) is obtained from the formula I′(x c ,y c )=255-I(x c ,yc ) is calculated, laying the foundation for the quantity and quality of feature points extracted later.

[0020] Further, the step 3 comprises:

[0021] 3-1 For the original image, the SuperPoint deep learning network is used to process the original image. The network completes two key tasks at the same time: one is to detect the significant SuperPoint feature points in the image, and the other is to generate the descriptor corresponding to the SuperPoint feature points;

[0022] 3-2 For the generated enhanced image, the SuperPoint network is used again to extract feature information to complete the two key tasks of SuperPoint feature point detection and descriptor generation;

[0023] 3-3 Based on the extracted SuperPoint feature points and combined with the semantic segmentation information obtained in step 1, the motion attribute analysis of the feature points is performed; specifically, by judging the semantic category of the area where the feature points are located, the SuperPoint feature points located in the purely static object area are identified and screened out, and marked as purely static SuperPoint feature points.

[0024] Further, the step 4 comprises:

[0025] 4-1 Improve the monocular initialization; for the purely static SuperPoint feature points from the original image and enhanced image at two different moments, use the SuperGlue algorithm to perform inter-frame matching on the purely static SuperPoint feature points in the two types of image sequences; specifically, based on SuperGlue, match the SuperPoint feature points in the original image at two moments before and after k-1 and k, and independently match the SuperPoint feature points in the enhanced image at the corresponding moment, so as to obtain two sets of independent feature point matching pairs and matching pair confidences;

[0026] 4-2 Perform quality assessment on the above two sets of matching results; when the matching pairs of purely static feature points in the original image and the enhanced image appear at the same pixel position, compare their matching confidences; if the matching confidence of the feature points in the enhanced image is higher than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the enhanced image; if the matching confidence of the feature points in the enhanced image is lower than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the original image; thereby eliminating the matching pairs with relatively low confidences and improving the overall quality of feature matching;

[0027] 4-3 For the high-quality feature matching pairs screened, the basic matrix is ​​calculated based on the epipolar geometric constraints to ensure the geometric consistency of feature point matching;

[0028] 4-4 Use the basic matrix and matching pairs to complete the triangulation operation of the feature points and establish the initial 3D map points;

[0029] 4-5 The Bundle Adjustment optimization method is used to jointly optimize the three-dimensional coordinates of the feature points and the camera posture to improve the accuracy and reliability of system initialization.

[0030] Further, the step 5 comprises:

[0031] 5-1 Constant speed motion model tracking includes: using the posture information of the previous frame and the estimated motion speed to estimate the initial posture of the current frame in the world coordinate system; the mathematical formula of the constant speed motion model is: T cw =VT lw ; Among them, T cw Represents the pose transformation matrix of the current frame in the world coordinate system; T lw Represents the pose transformation matrix of the camera in the world coordinate system of the previous frame; V represents the movement speed of the camera V = T cl =T cw T wl ; Among them, T cl Indicates the pose transformation of the current frame relative to the previous frame; T wl Represents the inverse of the pose transformation matrix of the previous frame in the world coordinate system; after obtaining the initial pose, the pose of the current frame is optimized using the 3D-2D projection relationship; finally, whether the tracking is successful is determined by counting the number of successfully matched feature points; if the tracking is successful, the optimized pose is output, otherwise the reference key frame is tracked;

[0032] 5-2 The tracking process of the reference key frame includes: based on visual similarity, select the frame closest to the current frame as the reference key frame; match the feature points of the current frame with the reference key frame, and use the bag-of-words model to accelerate the matching process; based on the obtained 2D-3D correspondence, calculate the initial pose of the current frame through the PnP algorithm; further optimize the pose by minimizing the reprojection error; count the number of successfully matched feature points to determine whether the tracking is successful; if the tracking fails, switch to the 2D-3D tracking method based on SuperGlue;

[0033] 5-3 The 2D-3D tracking process based on SuperGlue includes: using the SuperGlue algorithm to obtain the 2D-2D feature point matching pairs between the previous frame and the current frame; according to the correspondence between the 2D-2D feature points and the 3D map points in ORB-SLAM2, the matching pairs are converted into 2D-3D correspondences; the corresponding 3D map point coordinates are found by the index of the 2D point in the previous frame; using the obtained 2D-3D matching pairs, the EPnP algorithm is used to solve the pose transformation of the current frame relative to the previous frame; if the number of available 3D-2D corresponding points is lower than the set threshold, the system automatically switches to the 2D-2D tracking mode based on SuperGlue;

[0034] 5-4 The 2D-2D tracking process based on SuperGlue includes: obtaining the 2D-2D feature point matching pairs between the previous frame and the current frame through the SuperGlue algorithm; using the matching pairs to calculate the basic matrix formula is x2 T Fx1=0; where x1 and x2 are the corresponding feature points in the previous frame and the current frame respectively, and F is the basic matrix; the basic matrix is ​​converted into the essential matrix E, and then the rotation matrix and translation vector of the relative pose transformation are solved; based on the estimated pose transformation, new 3D map points are created by triangulation method; then the scale factor is calculated through the 2D-3D point correspondence relationship, and the newly established 3D points are converted to the global coordinate system; the current frame and the previous frame are set as key frames, and key frame management, map point management and local BA optimization are performed to ensure the stability and accuracy of the system; if the number of 2D-3D matching points is lower than the set threshold, the system enters the relocalization mode;

[0035] 5-5 The relocalization process includes: using the bag-of-words model to search the key frame database for the candidate key frame that is most similar to the current frame; calculating the initial pose of the current frame through the EPnP algorithm based on the retrieved candidate key frame; pose optimization; entering the iterative optimization phase, alternately performing the following operations: (1) Reprojection matching: reprojection matching of feature points based on the current pose (2) BA optimization: optimizing the pose of the current frame to improve the accuracy of pose estimation.

[0036] Further, the step 6 comprises:

[0037] 6-1 Convert the selected 2D feature points into 3D map points through triangulation method, calculate the number of observation frames of each 3D point, and record the key frame information of the observed map points;

[0038] 6-2 Eliminate map points with excessive reprojection errors and merge duplicate map points;

[0039] 6-3 Perform local BA optimization to optimize the map point position and key frame pose;

[0040] 6-4 Save the optimized map point information.

[0041] The beneficial effects of the present invention are:

[0042] (1) The present invention significantly improves the initialization efficiency and robustness of the system by introducing the SuperGlue algorithm for feature matching and combining the matching results of the original image and the enhanced image sequence.

[0043] (2) The present invention adopts a multi-level tracking strategy, including constant velocity model prediction, key frame reference, and SuperGlue-based 2D-3D and 2D-2D tracking, which greatly improves the positioning accuracy and environmental adaptability of the system in complex environments.

[0044] (3) The present invention effectively improves the stability of the system in dynamic scenes and lighting changing environments through the pure static feature point screening and high-quality matching pair selection mechanism.

[0045] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 It is the algorithm flow chart of the present invention;

[0048] Figure 2 It is the distribution result of feature points extracted from the original image by SuperPoint algorithm;

[0049] Figure 3 To enhance the distribution of feature points extracted by the SuperPoint algorithm;

[0050] Figure 4 It is the matching result of the feature points of the previous and next two frames of the original image through the SuperGlue algorithm;

[0051] Figure 5 To enhance the matching results of the feature points of the two frames before and after the image through the SuperGlue algorithm;

[0052] Figure 6 This is the running result of the algorithm of the present invention on the Odometry sequence 01 of the KITTI dataset;

[0053] Figure 7This is a local enlarged view of the running results of the algorithm of the present invention on the Odometry sequence 01 of the KITTI dataset. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0055] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0056] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0057] like Figure 1 As shown, a semantic-based monocular visual dynamic SLAM method of the present invention comprises the following steps:

[0058] Step 1: Use the DeepLabV3 semantic segmentation deep learning model to perform semantic segmentation on the monocular visual image and identify areas of unknown motion state and purely static state;

[0059] Step 2: Perform data enhancement on the original image, that is, generate new enhanced image data through image inversion transformation, which lays the foundation for improving the quantity and quality of feature points;

[0060] Step 3: For the original image and enhanced image, the SuperPoint network is used to replace the traditional ORB feature point extraction method to extract SuperPoint feature points with stronger representation ability and robustness;

[0061] Step 4: Initialization stage: Use the SuperGlue algorithm to match the SuperPoint feature points in the purely static area, combine the matching results of the original image and the enhanced image sequence, and obtain high-quality feature correspondence through confidence screening to improve the reliability of initialization;

[0062] Step 5, tracking phase, the accuracy and robustness of subsequent pose estimation are ensured through constant speed motion model tracking, key frame reference tracking, SuperGlue-based 2D-3D tracking, SuperGlue-based 2D-2D tracking and relocalization;

[0063] Step 6: construct a 3D sparse map of the environment based on the matched purely static feature points.

[0064] Step 1 specifically includes

[0065] 1-1 For the original image captured by the monocular camera, the DeepLabV3 deep learning model is used for semantic segmentation to divide the image into regions with different semantic labels. That is, each pixel contains corresponding semantic information.

[0066] 1-2 Based on the segmented semantic regions, the algorithm further analyzes each key semantic category (including purely static regions such as roads, buildings, and street lights, as well as regions with unknown motion states such as vehicles and people). Specifically, the motion characteristics are judged based on the semantic information and divided into purely static state regions and regions with unknown motion states.

[0067] The original grayscale image I(x c ,y c ) to perform inverse color transformation, where x c and c Represent the row and column coordinates of the pixel respectively, and the pixel value range is [0,255]. The enhanced image I′(x c ,y c ) is obtained from the formula I′(x c ,y c )=255-I(x c ,y c ) is calculated. This data enhancement method lays the foundation for the quantity and quality of feature points extracted subsequently.

[0068] Step 3 specifically includes

[0069] 3-1 As Figure 2 and Figure 3 As shown in the figure, for the original image, the SuperPoint deep learning network is used to process the original image. The network completes two key tasks at the same time: one is to detect the significant SuperPoint feature points in the image, and the other is to generate the descriptor corresponding to the SuperPoint feature points.

[0070] 3-2 For the enhanced image generated in the previous steps, the SuperPoint network is used again to extract feature information to complete the two key tasks of SuperPoint feature point detection and descriptor generation.

[0071] 3-3 Based on the SuperPoint feature points extracted in the previous steps, combined with the semantic segmentation information obtained in step 1, the motion attribute analysis of the feature points is performed. Specifically, by judging the semantic category of the area where the feature points are located, the SuperPoint feature points located in the area of ​​pure static objects (such as buildings, roads, etc.) are identified and screened, and marked as pure static SuperPoint feature points.

[0072] Step 4 specifically includes

[0073] 4-1 As Figure 4 and Figure 5 As shown in the figure, improvements are made for monocular initialization. For the purely static SuperPoint feature points from the original image and enhanced image at two different moments, the SuperGlue algorithm is used to perform inter-frame matching on the purely static SuperPoint feature points in the two types of image sequences. Specifically, SuperGlue is used to match the SuperPoint feature points in the original image at two moments before and after k-1 and k, and the SuperPoint feature points in the enhanced image at the corresponding moment are matched independently, thereby obtaining two sets of independent feature point matching pairs and matching pair confidences.

[0074] 4-2 Perform quality assessment on the above two sets of matching results. When the matching pairs of purely static feature points in the original image and the enhanced image appear at the same pixel position, compare their matching confidences; if the matching confidence of the feature points in the enhanced image is higher than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the enhanced image; if the matching confidence of the feature points in the enhanced image is lower than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the original image; in order to eliminate the matching pairs with relatively low confidences, the overall quality of feature matching is improved.

[0075] 4-3 For the high-quality feature matching pairs screened, the basic matrix is ​​calculated based on the epipolar geometric constraints to ensure the geometric consistency of feature point matching.

[0076] 4-4 Use the basic matrix and matching pairs to complete the triangulation operation of the feature points and establish the initial 3D map points.

[0077] 4-5 The Bundle Adjustment optimization method is used to jointly optimize the three-dimensional coordinates of the feature points and the camera posture, thereby improving the accuracy and reliability of system initialization.

[0078] Step 5 specifically includes

[0079] 5-1 Constant speed motion model tracking includes the following steps. Using the pose information of the previous frame and the estimated motion speed, the initial pose of the current frame in the world coordinate system is estimated. The mathematical formula of the constant speed motion model is: cw =VT lw Among them, T cw Represents the pose transformation matrix of the current frame in the world coordinate system; T lw Represents the pose transformation matrix of the camera in the world coordinate system of the previous frame; V represents the movement speed of the camera V = T cl =T cw T wl Among them, T cl Indicates the pose transformation of the current frame relative to the previous frame; T wl Represents the inverse of the pose transformation matrix of the previous frame in the world coordinate system. After obtaining the initial pose, the pose of the current frame is optimized using the 3D-2D projection relationship. Finally, the number of successfully matched feature points is counted to determine whether the tracking is successful. If the tracking is successful, the optimized pose is output, otherwise the reference key frame is tracked.

[0080] 5-2 The tracking process of the reference key frame includes the following steps. Based on visual similarity, select the frame closest to the current frame as the reference key frame. Match the feature points of the current frame with the reference key frame, and use the bag-of-words model to accelerate the matching process. Based on the obtained 2D-3D correspondence, calculate the initial pose of the current frame through the PnP algorithm. The pose is further optimized by minimizing the reprojection error. Count the number of successfully matched feature points to determine whether the tracking is successful. If the tracking fails, switch to the 2D-3D tracking method based on SuperGlue.

[0081] 5-3 The 2D-3D tracking process based on SuperGlue includes the following steps. The SuperGlue algorithm is used to obtain the 2D-2D feature point matching pairs between the previous frame and the current frame. According to the correspondence between the 2D-2D feature points and the 3D map points in ORB-SLAM2, the matching pairs are converted into 2D-3D correspondences. Specifically, the corresponding 3D map point coordinates are found by the index of the 2D point in the previous frame. Using the obtained 2D-3D matching pairs, the EPnP algorithm is used to solve the pose transformation of the current frame relative to the previous frame. If the number of available 3D-2D corresponding points is lower than the set threshold, the system automatically switches to the 2D-2D tracking mode based on SuperGlue.

[0082] 5-4 The 2D-2D tracking process based on SuperGlue includes the following steps. The 2D-2D feature point matching pairs between the previous frame and the current frame are obtained through the SuperGlue algorithm. The basic matrix formula is calculated using the matching pairs: Among them, x1 and x2 are the corresponding feature points in the previous frame and the current frame respectively, and F is the basic matrix. The basic matrix is ​​converted into the essential matrix E, and then the rotation matrix and translation vector of the relative pose transformation are solved. Based on the estimated pose transformation, new 3D map points are created by triangulation method. Then the scale factor is calculated through the 2D-3D point correspondence, and the newly established 3D points are converted to the global coordinate system. The current frame and the previous frame are set as key frames, and key frame management, map point management and local BA optimization are performed to ensure the stability and accuracy of the system. If the number of 2D-3D matching points is lower than the set threshold, the system enters the relocalization mode.

[0083] 5-5 The relocalization process includes the following steps. The bag-of-words model is used to search the keyframe database for the candidate keyframe that is most similar to the current frame. Based on the retrieved candidate keyframes, the initial pose of the current frame is calculated using the EPnP algorithm. Pose optimization. Entering the iterative optimization phase, the following operations are performed alternately: (1) Reprojection matching: Reprojection matching of feature points based on the current pose (2) BA optimization: Optimize the pose of the current frame to improve the pose estimation accuracy.

[0084] Step 6: The process of building a 3D sparse map of the environment based on the matched purely static feature points includes the following steps

[0085] 6-1 Convert the filtered 2D feature points into 3D map points through triangulation method, calculate the observation frame number information of each 3D point, and record the key frame information of the observed map point.

[0086] 6-2 Eliminate map points with too large reprojection errors and merge duplicate map points.

[0087] 6-3 Perform local BA optimization to optimize the map point positions and key frame poses.

[0088] 6-4 Save the optimized map point information.

[0089] To verify the advantages of the present invention in initialization efficiency and system robustness, we compared it with the monocular mode of the ORB-SLAM2 system proposed by Mur-Artal et al. (ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras, IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255-1262, Oct. 2017) on the Odometry sequence 01 of the KITTI dataset. The experimental results show that in terms of initialization efficiency, as shown in Table 1, the present invention only needs 2 frames of images to complete initialization, which is 200% higher than the 6 frames required by ORB-SLAM2, significantly improving the startup speed of the system.

[0090] Table 1

[0091] algorithm Initialize the required number of frames Improved initialization efficiency This algorithm 2 +200％ ORB-SLAM2 6 Baseline

[0092] In terms of system robustness, as shown in Table 2, the present invention can effectively track all 1101 frame sequences, and the tracking success rate reaches 100%. In contrast, the traditional ORB-SLAM2 algorithm can only continuously track 228 frames, and the tracking rate is only 20.7%.

[0093] Table 2

[0094] algorithm Effective tracking frames / total frames Tracking Rate This algorithm 1101 / 1101 100.0％ ORB-SLAM2 228 / 1101 20.7％

[0095] like Figure 6 and Figure 7 The figure shows the reconstruction result of this algorithm on the KITTI dataset Odometry sequence 01. The black point cloud represents the reconstructed environment map. From the reconstruction results, it can be seen that this algorithm can accurately restore the camera motion trajectory and construct a sparse 3D environment map.

[0096] In summary, the experimental results show that the present invention has achieved significant improvements in both initialization efficiency and system robustness: the number of frames required for initialization is reduced from 6 frames to 2 frames, and the tracking success rate is increased from 20.7% to 100%. These improvements make the system more adaptable and reliable in actual scenarios.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A semantic-based monocular visual dynamic SLAM method, characterized by: The method comprises the following steps: Step 1: Use the DeepLabV3 semantic segmentation deep learning model to perform semantic segmentation on the monocular visual image and identify areas of unknown motion state and purely static state; Step 2, perform data enhancement on the original image, that is, generate new enhanced image data by image inversion transformation; Step 3: For the original image and enhanced image, the SuperPoint network is used to replace the traditional ORB feature point extraction method to extract SuperPoint feature points with stronger representation ability and robustness; Step 4: Initialization stage: Use the SuperGlue algorithm to match the SuperPoint feature points in the purely static area, combine the matching results of the original image and the enhanced image sequence, and obtain high-quality feature correspondence through confidence screening to improve the reliability of initialization; Step 5, tracking phase, the accuracy and robustness of subsequent pose estimation are ensured through constant speed motion model tracking, key frame reference tracking, SuperGlue-based 2D-3D tracking, SuperGlue-based 2D-2D tracking and relocalization; Step 6: Build a 3D sparse map of the environment based on the matched purely static feature points.

2. The semantic-based monocular visual dynamic SLAM method according to claim 1, characterized in that: The step 1 comprises: 1-1 For the original image captured by the monocular camera, the DeepLabV3 deep learning model is used for semantic segmentation to divide the image into regions with different semantic labels; that is, each pixel contains corresponding semantic information; 1-2 Based on the segmented semantic regions, each key semantic category is analyzed, and the motion characteristics are judged according to the semantic information, and they are divided into pure static state areas and unknown motion state areas.

3. The semantic-based monocular visual dynamic SLAM method according to claim 2, characterized in that: The step 2 comprises: for the original grayscale image I(x c ,y c ) to perform inverse color transformation, where x c and c Represent the row and column coordinates of the pixel respectively, and the pixel value range is [0,255]; the enhanced image I′(x c ,y c ) is obtained from the formula I′(x c ,y c )=255-I(x c ,y c ) is calculated, laying the foundation for the quantity and quality of feature points extracted later.

4. The semantic-based monocular visual dynamic SLAM method according to claim 3, characterized in that: The step 3 comprises: 3-1 For the original image, the SuperPoint deep learning network is used to process the original image. The network completes two key tasks at the same time: one is to detect the significant SuperPoint feature points in the image, and the other is to generate the descriptor corresponding to the SuperPoint feature points; 3-2 For the generated enhanced image, the SuperPoint network is used again to extract feature information to complete the two key tasks of SuperPoint feature point detection and descriptor generation; 3-3 Based on the extracted SuperPoint feature points and combined with the semantic segmentation information obtained in step 1, the motion attribute analysis of the feature points is performed; specifically, by judging the semantic category of the area where the feature points are located, the SuperPoint feature points located in the purely static object area are identified and screened out, and marked as purely static SuperPoint feature points.

5. The semantic-based monocular visual dynamic SLAM method according to claim 4, characterized in that: The step 4 comprises: 4-1 Improve the monocular initialization; for the purely static SuperPoint feature points from the original image and enhanced image at two different moments, use the SuperGlue algorithm to perform inter-frame matching on the purely static SuperPoint feature points in the two types of image sequences; specifically, based on SuperGlue, match the SuperPoint feature points in the original image at two moments before and after k-1 and k, and independently match the SuperPoint feature points in the enhanced image at the corresponding moment, so as to obtain two sets of independent feature point matching pairs and matching pair confidence matching results; 4-2 Perform quality assessment on the above two sets of matching results; when the matching pairs of purely static feature points in the original image and the enhanced image appear at the same pixel position, compare their matching confidences; if the matching confidence of the feature points in the enhanced image is higher than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the enhanced image; if the matching confidence of the feature points in the enhanced image is lower than the matching confidence of the corresponding feature points in the original image, then retain the matching pairs in the original image; thereby eliminating the matching pairs with relatively low confidences and improving the overall quality of feature matching; 4-3 For the high-quality feature matching pairs screened, the basic matrix is ​​calculated based on the epipolar geometric constraints to ensure the geometric consistency of feature point matching; 4-4 Use the basic matrix and matching pairs to complete the triangulation operation of the feature points and establish the initial 3D map points; 4-5 The Bundle Adjustment optimization method is used to jointly optimize the three-dimensional coordinates of the feature points and the camera posture to improve the accuracy and reliability of system initialization.

6. The semantic-based monocular visual dynamic SLAM method according to claim 5, characterized in that: The step 5 comprises: 5-1 Constant speed motion model tracking includes: using the posture information of the previous frame and the estimated motion speed to estimate the initial posture of the current frame in the world coordinate system; the mathematical formula of the constant speed motion model is: T cw =VT lw ; Among them, T cw Represents the pose transformation matrix of the current frame in the world coordinate system; T lw Represents the pose transformation matrix of the camera in the world coordinate system of the previous frame; V represents the movement speed of the camera V = T cl =T cw T wl ; Among them, T cl Indicates the pose transformation of the current frame relative to the previous frame; T wl Represents the inverse of the pose transformation matrix of the previous frame in the world coordinate system; after obtaining the initial pose, the pose of the current frame is optimized using the 3D-2D projection relationship; finally, whether the tracking is successful is determined by counting the number of successfully matched feature points; if the tracking is successful, the optimized pose is output, otherwise the reference key frame is tracked; 5-2 The tracking process of the reference key frame includes: based on visual similarity, select the frame closest to the current frame as the reference key frame; match the feature points of the current frame with the reference key frame, and use the bag-of-words model to accelerate the matching process; based on the obtained 2D-3D correspondence, calculate the initial pose of the current frame through the PnP algorithm; further optimize the pose by minimizing the reprojection error; count the number of successfully matched feature points to determine whether the tracking is successful; if the tracking fails, switch to the 2D-3D tracking method based on SuperGlue; 5-3 The 2D-3D tracking process based on SuperGlue includes: using the SuperGlue algorithm to obtain the 2D-2D feature point matching pairs between the previous frame and the current frame; according to the correspondence between the 2D-2D feature points and the 3D map points in ORB-SLAM2, the matching pairs are converted into 2D-3D correspondences; the corresponding 3D map point coordinates are found by the index of the 2D point in the previous frame; using the obtained 2D-3D matching pairs, the EPnP algorithm is used to solve the pose transformation of the current frame relative to the previous frame; if the number of available 3D-2D corresponding points is lower than the set threshold, the system automatically switches to the 2D-2D tracking mode based on SuperGlue; 5-4 The 2D-2D tracking process based on SuperGlue includes: obtaining the 2D-2D feature point matching pairs between the previous frame and the current frame through the SuperGlue algorithm; using the matching pairs to calculate the basic matrix formula is: Among them, x1 and x2 are the corresponding feature points in the previous frame and the current frame respectively, and F is the basic matrix; the basic matrix is ​​converted into the essential matrix E, and then the rotation matrix and translation vector of the relative pose transformation are solved; based on the estimated pose transformation, new 3D map points are created by triangulation method; then the scale factor is calculated through the 2D-3D point correspondence relationship, and the newly established 3D points are converted to the global coordinate system; the current frame and the previous frame are set as key frames, and key frame management, map point management and local BA optimization are performed to ensure the stability and accuracy of the system; if the number of 2D-3D matching points is lower than the set threshold, the system enters the relocalization mode; 5-5 The relocalization process includes: using the bag-of-words model to search the key frame database for the candidate key frame that is most similar to the current frame; calculating the initial pose of the current frame through the EPnP algorithm based on the retrieved candidate key frame; pose optimization; entering the iterative optimization phase, alternately performing the following operations: (1) Reprojection matching: reprojection matching of feature points based on the current pose (2) BA optimization: optimizing the pose of the current frame to improve the accuracy of pose estimation.

7. The semantic-based monocular visual dynamic SLAM method according to claim 6, characterized in that: The step 6 comprises: 6-1 Convert the selected 2D feature points into 3D map points through triangulation method, calculate the number of observation frames of each 3D point, and record the key frame information of the observed map points; 6-2 Eliminate map points with excessive reprojection errors and merge duplicate map points; 6-3 Perform local BA optimization to optimize the map point position and key frame pose; 6-4 Save the optimized map point information.

Images