A dynamic scene visual slam method, system and device fusing deep semantic and spatiotemporal consistency constraints

Abstract

A dynamic scene visual SLAM method, system and device fusing deep semantic and space-time consistency constraints, based on ORB-SLAM3 framework, adding a dual-thread parallel processing mechanism of tracking thread and semantic thread: obtaining an initial semantic mask of dynamic objects through a lightweight Bisenet V2 semantic segmentation network, combining depth image gradient information to perform window local inflation correction, and improving the edge accuracy of the mask; constructing a multi-frame dynamic marker history cache queue, adopting an exponential weighted time continuity constraint to correct accidental missed detection and misjudgment of single-frame semantic segmentation, and completing dynamic feature point rough screening; fusing optical flow tracking error and re-projection error to generate space consistency fusion error, generating an adaptive threshold through K-means clustering, removing potential dynamic feature points through space consistency inspection, and finally completing positioning and dense mapping based on pure static feature points, giving consideration to positioning accuracy and real-time performance, and being widely applicable to positioning and navigation of autonomous mobile robots in dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of dynamic scene visual simultaneous localization and mapping technology, specifically to a dynamic scene visual SLAM method, system, and device that integrates deep semantics and spatiotemporal consistency constraints. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technique that allows a subject equipped with specific sensors to estimate its own pose and simultaneously construct an environmental model through sensor observations during movement, even in the absence of prior environmental information. It is a crucial foundation for autonomous mobile robots, drones, AR / VR, and other fields. Visual SLAM, based on camera sensors, offers advantages such as low cost, small size, and rich environmental information, making it the mainstream research direction in SLAM technology.

[0003] Currently, most mainstream visual SLAM algorithms are designed based on the assumption of a static environment. However, in real indoor and outdoor scenes, there are dynamic interference factors such as moving pedestrians, vehicles, and movable objects, which can lead to deviations in feature matching. This can cause problems such as the accumulation of pose estimation errors, localization drift, and map distortion, which severely limit the application of visual SLAM in real dynamic scenes.

[0004] Existing dynamic scene visual SLAM solutions are mainly divided into two categories: traditional dynamic elimination methods based on geometric constraints and semantic-assisted dynamic elimination methods based on deep learning.

[0005] In geometrically constrained schemes, the classic ORB-SLAM3 algorithm uses the random sample consistency method to eliminate mismatched points in order to reduce the interference of dynamic objects on feature matching. However, it cannot effectively distinguish between dynamic feature points and mismatched points in highly dynamic scenes, and the improvement in positioning accuracy is very limited. Another scheme constructs the topological relationship of feature points through triangulation and combines point and line features with scene structure constraints to filter static feature points. However, these schemes generally suffer from high computational complexity, poor real-time performance, dependence on scene structure characteristics, and weak generalization ability, making them difficult to adapt to complex and ever-changing real dynamic scenes.

[0006] Deep learning-based semantic-assisted solutions provide prior information about dynamic objects through semantic segmentation or object detection. For example, some solutions use the SegNet network to segment dynamic pedestrians or combine instance segmentation networks with multi-view geometry to filter dynamic feature points. However, such solutions generally have several drawbacks: First, the semantic segmentation network structure is complex and computationally intensive, which seriously affects the algorithm's running speed and makes it difficult to meet real-time positioning requirements. Second, the semantic mask edge segmentation accuracy is low, and it is prone to oversegmentation or undersegmentation, leading to the accidental deletion of static feature points or the missed detection of dynamic feature points. Third, it can only detect dynamic objects of predefined categories and cannot identify non-predefined potential dynamic objects such as moved furniture. Fourth, subsequent geometric constraints mostly rely on manually preset fixed thresholds, which have poor adaptability and insufficient robustness in different dynamic scenarios, and cannot simultaneously guarantee positioning accuracy and running efficiency in dynamic scenarios. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to provide a dynamic scene visual SLAM method, system, and device that integrates deep semantics and spatiotemporal consistency constraints. Based on the classic ORB-SLAM3 algorithm, it adds dual-thread parallel processing of semantic thread and tracking thread, adds a dynamic feature removal module, corrects semantic mask edges through depth information, corrects single-frame semantic errors by combining the temporal continuity constraints of multi-frame memory, and removes potential dynamic feature points by using spatial consistency checks with adaptive thresholds, thereby achieving high-precision and robust localization in dynamic scenes while taking into account the real-time performance of the algorithm.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A dynamic scene visual SLAM method that integrates deep semantics and spatiotemporal consistency constraints specifically includes the following steps: Step 1: Acquire RGB and depth images and perform parallel processing in two threads. In the tracking thread, the input RGB image is preprocessed to complete image denoising, ORB feature extraction and inter-frame matching. In the semantic thread, the lightweight semantic segmentation network Bisenet V2 is used to perform pixel-level semantic segmentation on the RGB image to obtain the initial semantic mask of the dynamic object. Combined with the depth gradient information of the depth image, the edges of the initial semantic mask are locally dilated to obtain the corrected semantic mask. The corrected mask image is then passed to the tracking thread to complete the initial labeling of dynamic and static feature points. Step 2: In the tracking thread, based on the corrected semantic mask, a multi-frame history cache queue with dynamic feature point marking is constructed. The weighted dynamic history ratio is calculated according to the time continuity using a weighted strategy to correct the missed detections and misjudgments in single-frame semantic segmentation, complete the coarse screening of dynamic feature points, and obtain the initial static feature point set. Step 3: Calculate the initial camera pose of the current frame based on the initial static feature point set, fuse optical flow tracking error and reprojection error to generate spatial consistency fusion error, use K-means clustering algorithm to adaptively generate partitioning threshold, remove remaining potential dynamic feature points, and obtain a pure static feature point set; Step 4: Based on the pure static feature point set, perform camera pose optimization, local mapping and loop closure detection, and output the camera positioning trajectory and dense point cloud map of the environment in the dynamic scene.

[0009] The specific implementation method of the semantic thread in step one is as follows: The lightweight semantic segmentation network Bisenet V2 is used to segment RGB images. Real dynamic classes that appear more frequently than a set threshold in indoor scenes are selected as segmentation targets, and an initial semantic mask with 255 pixels in dynamic regions and 0 pixels in static regions is output. Median filtering is applied to the depth image to eliminate salt-and-pepper noise; the Sobel operator is used to calculate the depth gradient magnitude of the filtered depth image and quantize the degree of depth abrupt change of pixels. Extract edge pixels from the initial semantic mask, construct non-overlapping sub-windows, calculate the mean depth gradient of edge pixels within the window; set a gradient threshold, perform local dilation correction on windows with insignificant depth abrupt changes, repeat the verification until the gradient threshold is met or the maximum number of dilations is reached, and generate the corrected semantic mask.

[0010] The median filtering constructs a 3×3 neighborhood centered on each pixel, and replaces the original depth value of the center pixel with the median depth value within the neighborhood; The size of the non-overlapping sub-window is 30×30 pixels; The real-world dynamic classes include people, cats, and dogs.

[0011] The specific method for step two is as follows: For each ORB feature point, a dynamic labeling history cache queue of length 5 is constructed to record the dynamic and static labeling results of the current frame and the previous 4 frames. A label of 1 indicates that the feature point is dynamic in the current frame, and a label of 0 indicates that it is static. An exponential weighting strategy is used to design the weight vector, with higher weights for smaller frame intervals. After normalizing the weights, the weighted dynamic historical proportion of each feature point is calculated. Set a dynamic proportion threshold. When the weighted dynamic historical proportion is higher than the threshold, it is determined to be a dynamic point. When it is lower than the threshold, it is determined to be a static point, thus completing the coarse screening of dynamic feature points. After each frame is processed, the cache queue is updated, the oldest frame marker is removed, and the initial marker for the next frame is added.

[0012] The decay coefficient of the exponential weighting strategy is 0.8; The dynamic percentage threshold is 0.5.

[0013] The specific method for step three is as follows: Inter-frame tracking of feature points is achieved using the pyramid LK optical flow method, and the optical flow tracking error of each feature point is calculated. Calculate the reprojection error of 3D map points corresponding to feature points based on the initial pose; Set an error balance coefficient to fuse optical flow tracking error and reprojection error to generate spatial consistency fusion error; The feature points are divided into a dynamic point set and a static point set according to the prior label. The fusion error of the two sets is subjected to K-means binary clustering to generate an adaptive partitioning threshold. By filtering out mislabeled static points based on a threshold and removing potential dynamic points with errors greater than the threshold, a pure set of static feature points is obtained.

[0014] The error balance coefficient is 0.5; The optical flow tracking error is the Euclidean distance between the true coordinates of the feature point and the optical flow tracking coordinates. The reprojection error is the deviation between the optical flow observation coordinates and the predicted coordinates obtained by projecting 3D map points onto the current frame.

[0015] In step four, the absolute trajectory error (ATE) and relative pose error (RPE) are used as evaluation metrics for the algorithm performance. Both ATE and RPE are statistically measured using root mean square error (RMSE), mean (Mean), and standard deviation (SD). The absolute trajectory error is used to measure the global deviation between the estimated trajectory and the true trajectory. Its root mean square error is calculated as follows: in, The total number of frames for the trajectory. For the first The true pose matrix of the frame, For the first The smaller the estimated pose matrix ATE of the frame, the closer the estimated trajectory of the SLAM algorithm is to the real trajectory, and the higher its global positioning accuracy. Relative pose error is used to measure the local pose deviation between adjacent frames. It is divided into translational relative error and rotational relative error. The root mean square error is calculated using the following formula: in, The number of adjacent frame pairs used to calculate the relative error. , For the first The true relative translation vector and the estimated relative translation vector of a group of adjacent frames , For the first The true relative rotation matrix and the estimated relative rotation matrix of a group of adjacent frames. , These are the root mean square errors of the translational and rotational relative pose errors, respectively. The smaller the values ​​of each RPE component, the closer the relative pose estimated by the SLAM algorithm is to the true relative pose, and the more stable the local pose estimation is.

[0016] A dynamic scene visual SLAM system integrating deep semantics and spatiotemporal consistency constraints, comprising: The data preprocessing module is used to perform denoising preprocessing, ORB feature extraction, and inter-frame matching on RGB images in the tracking thread; The semantic segmentation and mask correction module is used to perform pixel-level semantic segmentation on RGB images in the semantic thread, combine depth gradient information to complete mask edge correction, and output the initial labels of dynamic and static feature points. The temporal continuity constraint module is used to construct a multi-frame dynamic labeling cache queue, and corrects single-frame missed detections and misjudgments by weighted dynamic historical proportions to complete the dynamic feature point coarse screening. The spatial consistency verification module is used to fuse optical flow tracking error and reprojection error, and adaptively divides dynamic and static feature points through clustering to output a pure set of static feature points. The localization and mapping module is used for camera pose optimization, local mapping and loop closure detection based on clean static feature points, and outputs localization trajectory and dense point cloud map.

[0017] A dynamic scene visual SLAM device that integrates deep semantics and spatiotemporal consistency constraints includes: Image acquisition device, used to acquire RGB and depth images of dynamic scenes; A processor configured to perform the method of any one of claims 1 to 8 when executing a computer program; Memory for storing the computer program; A display used to show the camera positioning trajectory and the results of dense environmental mapping; And a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention addresses the problem of ambiguous semantic mask edge segmentation by proposing a window local dilation correction method based on depth gradient. It utilizes the abrupt changes in depth between dynamic objects and the background to adaptively correct mask edges, avoiding over-segmentation and under-segmentation, and improving the accuracy of initial labeling of dynamic and static feature points.

[0019] 2. To address the problem of accidental missed detections and misjudgments in single-frame semantic segmentation, this invention designs a multi-frame memory-enhanced temporal continuity constraint. By using the exponentially weighted dynamic historical proportion, the temporal continuity of motion is used to correct single-frame errors and reduce the interference of occlusion and motion blur on dynamic detection.

[0020] 3. To address the issues of insufficient detection of potential dynamic objects and poor robustness of fixed thresholds, this invention proposes an adaptive threshold spatial consistency test. It integrates optical flow tracking error and reprojection error to characterize the dynamic and static properties of feature points, and achieves adaptive threshold partitioning through K-means clustering. This eliminates the need for manual threshold presets, effectively detects semantically undefined potential dynamic objects, and avoids the accidental deletion of valid static feature points.

[0021] 4. This invention adopts a lightweight Bisenet V2 network and a dual-thread parallel architecture, which ensures high accuracy while taking into account real-time performance. The tracking time for a single frame is only 38.3ms. On the TUM high dynamic dataset, the absolute trajectory error is reduced by up to 97.38%, and the relative pose error is reduced by up to 95.16%. The positioning accuracy and robustness are significantly better than the existing mainstream dynamic SLAM algorithms.

[0022] In summary, this invention, based on the ORB-SLAM3 architecture, effectively solves the problems of localization drift and mapping distortion in visual SLAM under dynamic scenes by using a dynamic feature removal module with deep semantic mask correction, temporal continuity constraints, and adaptive spatial consistency checks. At the same time, it takes into account the real-time performance of the algorithm and can be widely applied to autonomous localization and navigation in real dynamic scenes such as indoor service robots and industrial inspection robots. Attached Figure Description

[0023] Figure 1 This is a diagram showing the overall architecture of the dynamic visual SLAM system that integrates deep semantics and spatiotemporal consistency constraints according to the present invention.

[0024] Figure 2 This is a schematic diagram of the edge of the window local expansion correction mask according to the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the temporal continuity constraint of the multi-frame memory enhancement of the present invention.

[0026] Figure 4 This is a block diagram of the adaptive threshold spatial consistency check structure of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments: See Figure 1 A dynamic scene visual SLAM method that integrates deep semantics and spatiotemporal consistency constraints, specifically including: S1 acquires RGB and depth images and performs parallel processing using dual threads. Specifically, in the tracking thread, the input RGB and depth images are preprocessed for denoising, ORB feature points are extracted, and feature matching between consecutive frames is completed, providing a foundation for subsequent dynamic culling and pose estimation. This invention retains the tracking thread, local mapping thread, loop closure detection thread, and multi-map system core modules of ORB-SLAM3, and adds an independent semantic thread and dynamic feature culling module. The semantic thread and tracking thread run in parallel to avoid blocking the main tracking process.

[0028] In the semantic thread, dynamic semantic masks are generated and corrected.

[0029] This invention employs Bisenet V2, a lightweight semantic segmentation network pre-trained on the MS COCO dataset, to perform pixel-level semantic segmentation on the RGB image input to the semantic thread. It selects high-frequency dynamic categories such as people, cats, and dogs as segmentation targets and outputs an initial binary semantic mask with 255 pixels in the dynamic region and 0 pixels in the static region.

[0030] To address the issue of blurred edge segmentation in the initial mask, edge correction based on depth consistency is implemented: First, a 3×3 median filter is applied to the input depth image to eliminate salt-and-pepper noise in the original depth map. The Sobel operator is then used to convolve the filtered depth map, calculating the x and y-direction depth gradient components for each pixel and fusing them to obtain the depth gradient magnitude, quantifying the degree of depth abrupt changes at each pixel. All edge pixels of the initial semantic mask are extracted, and a 30×30 non-overlapping sub-window is constructed. The mean depth gradient of edge pixels within each sub-window is calculated. A gradient threshold is set; when the mean gradient of the window is below the threshold, it indicates that there is a missed segmentation of the mask edges in that region. The window is then locally dilated, and the mean gradient is recalculated after dilation. This process is repeated until the threshold is met or the maximum number of dilations is reached, ultimately generating a semantic mask with accurate edges and completing the initial labeling of dynamic and static feature points. The correction process is as follows: Figure 2 As shown.

[0031] S2 performs a temporal continuity constraint test for multi-frame memory enhancement. To address the issue of occasional missed detections and misjudgments in single-frame semantic segmentation due to occlusion and motion blur, a multi-frame memory-enhanced verification mechanism is constructed by leveraging the temporal continuity of object motion. The process is as follows: Figure 3 As shown: For each ORB feature point, a dynamic labeling history cache queue of length 5 is constructed to record the dynamic and static labeling results of the current frame and the previous 4 frames. A label of 1 indicates dynamic and 0 indicates static.

[0032] An exponential weighting strategy is adopted to design the weight vector. The smaller the time difference between the historical frame and the current frame, the higher the labeling reliability. The weight increases exponentially, and the decay coefficient is set to 0.8. After normalizing the weights, the weighted dynamic historical proportion of each feature point is calculated, which is the weighted sum of each frame label and its corresponding weight in the cache queue.

[0033] A dynamic proportion threshold of 0.5 is set. When the weighted dynamic historical proportion is greater than 0.5, the feature point continuously exhibits dynamic characteristics in recent times. Even if the current frame's semantic segmentation misses a static feature point, it is still judged as a dynamic feature point. When the proportion is less than 0.5, it is judged as a static feature point, and the false detection result of the current frame is corrected. After each frame is processed, the cache queue is updated, the oldest frame mark is removed, and the initial mark of the next frame is added to achieve real-time updating of historical memory.

[0034] S3 eliminates dynamic points through spatial consistency checks using adaptive thresholds. To address the issue of poor robustness between semantically undefined potential dynamic objects and fixed thresholds, an adaptive threshold spatial consistency check is performed. The structural diagram is as follows: Figure 4 As shown: First, the pyramid LK optical flow method is used to achieve inter-frame tracking of feature points. Based on gray-level invariance and the least squares method, the optical flow tracking error of each feature point is solved, which is the Euclidean distance between the actual coordinates of the feature point and the optical flow tracking coordinates. Based on the initial static feature point set obtained by coarse screening, the initial pose of the camera in the current frame is calculated. According to the camera projection model, the 3D map points corresponding to the feature points are projected onto the current frame, and the reprojection error is calculated, which is the deviation between the optical flow observed coordinates and the projected predicted coordinates. An error balance coefficient λ=0.5 is set to fuse the optical flow tracking error and the reprojection error to generate a spatial consistency fusion error, which comprehensively characterizes the dynamic and static characteristics of the feature points.

[0035] All feature points in the current frame are divided into a dynamic point set D and a static point set S according to their prior dynamic and static labels. K-means binary clustering is then performed on the fusion errors of the two sets: For set D, clustering yields true dynamic point clusters with large errors and mislabeled static point clusters with small errors. The mean of the two cluster centers is taken as an adaptive threshold, and feature points with errors less than the threshold are selected and included in the static point candidate set. For set S, clustering yields true static point clusters with small errors and potential dynamic point clusters with large errors. An adaptive threshold is generated, and potential dynamic points with errors greater than the threshold are removed.

[0036] By merging the static points from the two screenings, a pure set of static feature points is obtained. The threshold is adaptively adjusted according to the distribution of feature points in each frame, eliminating the need for manually preset thresholds for different scenarios.

[0037] S4.SLAM Localization and Mapping: The pure static feature point set is input into the pose optimization module of ORB-SLAM3 to complete the nonlinear optimization of the camera pose; key frames are selected for local mapping and loop closure detection, and finally the global positioning trajectory of the camera is output; based on the accurate camera pose and dynamic region mask, point clouds are generated only for static regions to construct a dense 3D point cloud map without dynamic ghosting.

[0038] The operating platform is Ubuntu 20.04, with an 11th Gen Intel i7-11800H CPU at 2.30GHz, 16GB of RAM, and an NVIDIA GeForce RTX 3060 Laptop GPU. Validation was performed using the TUM RGB-D dynamic dataset from the Technical University of Munich, selecting the fr3_walking high dynamic range sequence and the fr3_sitting low dynamic range sequence, covering various camera motion modes.

[0039] Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) are used as evaluation metrics for algorithm performance. Pose error is determined by statistical measures such as root mean square error (RMSE), mean, and standard deviation (SD), and is an intuitive reflection of the positioning accuracy and robustness in dynamic scenarios. The absolute trajectory error is used to measure the global deviation between the estimated trajectory and the true trajectory. Its root mean square error is calculated using the formula (1): (1) in, The total number of frames for the trajectory. For the first The true pose matrix of the frame, For the first The smaller the estimated pose matrix ATE of the frame, the closer the estimated trajectory of the SLAM algorithm is to the real trajectory, and the higher its global positioning accuracy. Relative pose error is used to measure the local pose deviation between adjacent frames. It is divided into translational relative error and rotational relative error. The root mean square error calculation formulas (2) and (3) are as follows: (2) (3) in, The number of adjacent frame pairs used to calculate the relative error. , For the first The true relative translation vector and the estimated relative translation vector of a group of adjacent frames , For the first The true relative rotation matrix and the estimated relative rotation matrix of a group of adjacent frames. , These are the root mean square errors (RMSEs) of the translational and rotational relative pose errors, respectively. The smaller the values ​​of each RPE component, the closer the relative pose estimated by the SLAM algorithm is to the true relative pose, and the more stable the local pose estimation.

[0040] A dynamic scene visual SLAM system integrating deep semantics and spatiotemporal consistency constraints, comprising: The data preprocessing module is used for denoising preprocessing of RGB images, ORB feature extraction, and inter-frame matching. The semantic segmentation and mask correction module is used to generate an initial semantic mask through Bisenet V2, combine it with deep gradient information to complete the mask edge correction, and output the initial labels of dynamic and static feature points. The temporal continuity constraint module is used to construct a multi-frame dynamic labeling cache queue. It corrects single-frame missed detections and misjudgments by using exponentially weighted dynamic historical proportions, thus completing the dynamic feature point coarse screening. The spatial consistency verification module is used to fuse optical flow tracking error and reprojection error, and adaptively divides dynamic and static feature points through K-means clustering to output a pure set of static feature points; The localization and mapping module is used for camera pose optimization, local mapping and loop closure detection based on clean static feature points, and outputs localization trajectory and dense point cloud map.

[0041] A dynamic scene visual SLAM device that integrates deep semantics and spatiotemporal consistency constraints includes: Image acquisition device, RGB-D camera, used to acquire RGB and depth images of dynamic scenes; A processor is used to store a computer program and execute the computer program to implement the dynamic scene visual SLAM method that integrates deep semantics and spatiotemporal consistency constraints as described in steps one to five. A monitor is used to display the camera's positioning trajectory and the results of dense environmental mapping.

[0042] This invention compares and analyzes the proposed dynamic scene visual SLAM method that integrates deep semantics and spatiotemporal consistency constraints with four mainstream visual SLAM algorithms: ORB-SLAM3, DS-SLAM, Dyna-SLAM, Blitz-SLAM, and CFP-SLAM. The method is validated in eight dynamic sequences of ORB-SLAM3. The results show that the proposed method has significant improvements over other methods in terms of root mean square error (RMSE) and standard deviation (SD) of absolute trajectory error, as well as real-time performance. This demonstrates that the proposed model can effectively solve the problem of localization drift in traditional visual SLAM in dynamic scenes.

[0043] The experimental results are shown in the table below.

[0044] Model RMSE(m) SD(m) Single-frame tracking time (ms) ORB-SLAM3 0.4846 0.2936 15.2 DS-SLAM 0.0247 0.0161 59.4 Dyna-SLAM 0.0164 0.0086 153.9 Blitz-SLAM 0.0153 0.0078 - CFP-SLAM 0.0141 0.0072 - Ours 0.0127 0.0063 38.3

Claims

1. A dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints, characterized in that: Includes the following steps: Step 1: Acquire RGB and depth images and perform parallel processing in two threads. In the tracking thread, the input RGB image is preprocessed to complete image denoising, ORB feature extraction and inter-frame matching. In the semantic thread, the lightweight semantic segmentation network Bisenet V2 is used to perform pixel-level semantic segmentation on the RGB image to obtain the initial semantic mask of the dynamic object. Combined with the depth gradient information of the depth image, the edges of the initial semantic mask are locally dilated to obtain the corrected semantic mask. The corrected mask image is then passed to the tracking thread to complete the initial labeling of dynamic and static feature points. Step 2: In the tracking thread, based on the corrected semantic mask, a multi-frame history cache queue with dynamic feature point marking is constructed. The weighted dynamic history ratio is calculated according to the time continuity using a weighted strategy to correct the missed detections and misjudgments in single-frame semantic segmentation, complete the coarse screening of dynamic feature points, and obtain the initial static feature point set. Step 3: Calculate the initial camera pose of the current frame based on the initial static feature point set, fuse optical flow tracking error and reprojection error to generate spatial consistency fusion error, use K-means clustering algorithm to adaptively generate partitioning threshold, remove remaining potential dynamic feature points, and obtain a pure static feature point set; Step 4: Based on the pure static feature point set, perform camera pose optimization, local mapping and loop closure detection, and output the camera positioning trajectory and dense point cloud map of the environment in the dynamic scene.

2. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 1, characterized in that: The specific implementation method of the semantic thread in step one is as follows: The lightweight semantic segmentation network Bisenet V2 is used to segment RGB images. Real dynamic classes that appear more frequently than a set threshold in indoor scenes are selected as segmentation targets, and an initial semantic mask with 255 pixels in dynamic regions and 0 pixels in static regions is output. Median filtering was applied to the depth image to eliminate salt-and-pepper noise. The Sobel operator is used to calculate the depth gradient magnitude of the filtered depth image and quantize the degree of depth abrupt change of pixels. Extract the edge pixels of the initial semantic mask, construct a non-overlapping sub-window, and calculate the mean depth gradient of the edge pixels within the window; Set a gradient threshold, perform local dilation correction on windows with insignificant depth abrupt changes, repeat the verification until the gradient threshold is met or the maximum number of dilations is reached, and generate the corrected semantic mask.

3. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 2, characterized in that: The median filtering constructs a 3×3 neighborhood centered on each pixel, and replaces the original depth value of the center pixel with the median depth value within the neighborhood; The size of the non-overlapping sub-window is 30×30 pixels; The real-world dynamic classes include people, cats, and dogs.

4. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 1, characterized in that: The specific method for step two is as follows: For each ORB feature point, a dynamic labeling history cache queue of length 5 is constructed to record the dynamic and static labeling results of the current frame and the previous 4 frames. A label of 1 indicates that the feature point is dynamic in the current frame, and a label of 0 indicates that it is static. An exponential weighting strategy is adopted to design the weight vector, with higher weights for smaller frame intervals. After normalizing the weights, the weighted dynamic historical proportion of each feature point is calculated. Set a dynamic proportion threshold. When the weighted dynamic historical proportion is higher than the threshold, it is determined to be a dynamic point. When it is lower than the threshold, it is determined to be a static point, thus completing the coarse screening of dynamic feature points. After each frame is processed, the cache queue is updated, the oldest frame marker is removed, and the initial marker for the next frame is added.

5. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 4, characterized in that: The decay coefficient of the exponential weighting strategy is 0.8; The dynamic percentage threshold is 0.

5.

6. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 1, characterized in that: The specific method for step three is as follows: The pyramid LK optical flow method is used to achieve inter-frame tracking of feature points, and the optical flow tracking error of each feature point is calculated. Calculate the reprojection error of 3D map points corresponding to feature points based on the initial pose; Set an error balance coefficient to fuse optical flow tracking error and reprojection error to generate spatial consistency fusion error; The feature points are divided into a dynamic point set and a static point set according to the prior label. The fusion error of the two sets is subjected to K-means binary clustering to generate an adaptive partitioning threshold. By filtering out mislabeled static points based on a threshold and removing potential dynamic points with errors greater than the threshold, a pure set of static feature points is obtained.

7. The dynamic scene visual SLAM method integrating deep semantics and spatiotemporal consistency constraints according to claim 6, characterized in that: The error balance coefficient is 0.5; The optical flow tracking error is the Euclidean distance between the true coordinates of the feature point and the optical flow tracking coordinates. The reprojection error is the deviation between the optical flow observation coordinates and the predicted coordinates obtained by projecting 3D map points onto the current frame.

8. The method according to claim 1, characterized in that, In step four, the absolute trajectory error (ATE) and relative pose error (RPE) are used as evaluation metrics for the algorithm performance. Both ATE and RPE are statistically measured using root mean square error (RMSE), mean (Mean), and standard deviation (SD). The absolute trajectory error is used to measure the global deviation between the estimated trajectory and the true trajectory. Its root mean square error is calculated as follows: in, The total number of frames for the trajectory. For the first The true pose matrix of the frame, For the first The smaller the estimated pose matrix ATE of the frame, the closer the estimated trajectory of the SLAM algorithm is to the real trajectory, and the higher its global positioning accuracy. Relative pose error is used to measure the local pose deviation between adjacent frames. It is divided into translational relative error and rotational relative error. The root mean square error is calculated using the following formula: in, The number of adjacent frame pairs used to calculate the relative error. , For the first The true relative translation vector and the estimated relative translation vector of a group of adjacent frames , For the first The true relative rotation matrix and the estimated relative rotation matrix of a group of adjacent frames. , These are the root mean square errors of the translational and rotational relative pose errors, respectively. The smaller the values ​​of each RPE component, the closer the relative pose estimated by the SLAM algorithm is to the true relative pose, and the more stable the local pose estimation is.

9. The dynamic scene visual SLAM system according to any one of claims 1 to 8, characterized in that, include: The data preprocessing module is used to perform denoising preprocessing, ORB feature extraction, and inter-frame matching on RGB images in the tracking thread; The semantic segmentation and mask correction module is used to perform pixel-level semantic segmentation on RGB images in the semantic thread, combine depth gradient information to complete mask edge correction, and output the initial labels of dynamic and static feature points. The temporal continuity constraint module is used to construct a multi-frame dynamic labeling cache queue, and corrects single-frame missed detections and misjudgments by weighted dynamic historical proportions to complete the dynamic feature point coarse screening. The spatial consistency verification module is used to fuse optical flow tracking error and reprojection error, and adaptively divides dynamic and static feature points through clustering to output a pure set of static feature points. The localization and mapping module is used for camera pose optimization, local mapping and loop closure detection based on clean static feature points, and outputs localization trajectory and dense point cloud map.

10. A dynamic scene visual SLAM device that integrates deep semantics and spatiotemporal consistency constraints, characterized in that, include: Image acquisition device, used to acquire RGB and depth images of dynamic scenes; A processor configured to perform the method of any one of claims 1 to 8 when executing a computer program; Memory for storing the computer program; A display screen is used to show the camera positioning trajectory and the dense mapping results of the environment; And a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images