Lightweight dynamic feature point elimination method based on deep learning

Abstract

The invention relates to a lightweight dynamic feature point elimination method based on deep learning. Accurate elimination of dynamic feature points is realized through cooperation of three core modules: a deep learning module generates a dynamic mask by adopting a'target detection-cutting-batch semantic segmentation 'strategy; the mask repairing module optimizes a mask by using an RGB-D camera depth map and a region growing algorithm to make up for semantic segmentation defects; the motion consistency detection module rejects unknown dynamic feature points through depth constraint and an adaptive threshold mechanism. Through TUM RGBD dynamic data set verification, the absolute trajectory error (ATE) root-mean-square error in a high dynamic scene is reduced by 97% to the maximum, the precision in a low dynamic scene is remarkably improved, the system frame rate is maintained to be 15-20 fps, the real-time requirement is met, the core contradiction that the real-time performance and the positioning precision of a VSLAM system in a dynamic environment are difficult to consider at the same time is effectively solved, and the positioning precision and robustness are greatly improved.

Description

Technical Field

[0001] This invention relates to a lightweight dynamic feature point removal method based on deep learning designed for dynamic environments, belonging to the field of Visual Simultaneous Localization and Mapping (VSLAM) technology. Background Technology

[0002] VSLAM technology acquires environmental images through visual sensors, enabling robots or mobile devices to autonomously locate and map their environments. It is widely used in fields such as autonomous driving, robot navigation, and augmented reality. The core assumption of traditional VSLAM systems is that the environment is static, meaning that all objects within the camera's field of view are stationary. However, in real-world application scenarios (such as streets, shopping malls, and indoor crowd flow scenarios), there are many dynamic objects (people, vehicles, etc.). The feature points of these dynamic objects may be misidentified by the system as static background feature points, leading to a significant increase in localization errors or even system failure.

[0003] To solve the positioning problem in dynamic environments, existing technologies are mainly divided into three categories:

[0004] 1. Optical flow method: This method determines the dynamics by tracking the movement trajectory of feature points. However, it is sensitive to changes in lighting and object occlusion, and has poor robustness in large-scale motion scenarios, making it difficult to guarantee positioning accuracy.

[0005] 2. Geometric constraint method: Based on the assumption that "static feature points account for the majority", static points are filtered through multi-view geometric constraints. However, in highly dynamic scenes (where dynamic objects account for a high proportion), this assumption does not hold, leading to a sharp decline in anti-interference ability.

[0006] 3. Deep learning method: It identifies dynamic objects and removes their feature points through semantic segmentation, but it has three major drawbacks: (i) It cannot identify unknown dynamic objects that have not been trained; (ii) It is easy to misjudge static objects as dynamic objects, resulting in the loss of static feature points; (iii) It has high computing power requirements and is difficult to meet the real-time requirements of VSLAM system.

[0007] In summary, existing technologies cannot simultaneously solve the three core problems of "accurate removal of dynamic feature points, identification of unknown dynamic objects, and real-time system operation." The positioning accuracy and real-time performance of VSLAM systems in dynamic environments are difficult to balance, which limits their practical application scope. Summary of the Invention

[0008] To overcome the shortcomings of existing VSLAM systems, such as large positioning errors in dynamic environments, poor robustness, high computational requirements, and inability to identify unknown dynamic objects, existing optical flow methods, geometric constraint methods, and deep learning methods, this invention provides a lightweight dynamic feature point removal method based on deep learning. While ensuring the real-time performance of the system, it accurately removes dynamic feature points (including predefined dynamic objects and unknown dynamic objects), significantly improving the positioning accuracy and robustness of VSLAM systems in dynamic environments.

[0009] A lightweight dynamic feature point removal method based on deep learning is proposed. It uses an RGB-D camera to acquire RGB and depth maps, and removes dynamic feature points through a deep learning module, a mask repair module, and a motion consistency detection module.

[0010] Using an RGB-D camera as a visual sensor, it can simultaneously acquire RGB images and depth maps. First, deep learning methods are used to process the RGB images to identify predefined dynamic objects. Second, a mask inpainting algorithm is used to process the depth map, enhancing the deep learning's ability to identify predefined dynamic objects and removing most dynamic feature points. Finally, a motion consistency detection algorithm is used to remove the few missed dynamic feature points. The process includes the following steps:

[0011] Step 1: The deep learning module first extracts the rectangular dynamic regions of predefined dynamic objects through the lightweight object detection submodule, and then performs batch semantic segmentation on the dynamic regions through the high-precision semantic segmentation submodule, outputting the precise pixel dynamic mask of the predefined dynamic objects.

[0012] Step 2: The mask repair module determines whether repair is needed based on the area ratio of the dynamic mask to the dynamic region. If repair is needed, the depth map from the RGB-D camera is used for repair and enhancement. The dynamic mask is optimized by combining the region growing algorithm with the depth map to ensure that the vast majority of dynamic feature points are covered.

[0013] Step 3: The motion consistency detection module uses depth constraint detection to remove dynamic feature points that were not detected by the deep learning module.

[0014] The deep learning module in step one includes:

[0015] S1.1: The object detection submodule uses a lightweight neural network (preferably YOLOv5) to identify predefined dynamic objects (such as people, cars, cats and dogs, etc., which are likely to be dynamic) in each frame, outputs a rectangular dynamic region that defines the object, and represents the detection result as a set of rectangular regions that enclose the predefined dynamic object, which is the dynamic region.

[0016] S1.2: The semantic segmentation submodule crops dynamic regions and converts them into tensor form. It then uses a GPU interpolation function to unify the resolution of the cropped images, stacks all tensors into a batch tensor, and uses a high-precision semantic segmentation network (preferably DeepLabv3) for parallel inference to output a dynamic mask. Specifically, the semantic segmentation submodule crops the image from the dynamic regions output by the object detection submodule and converts it into tensor form. It uses a GPU interpolation function to unify the resolution of each cropped image, stacks all GPU tensors together, converts them into a batch tensor, and uses a high-precision semantic segmentation network for parallel inference to perform semantic segmentation on multiple dynamic regions at once, outputting a pixel mask of a predefined dynamic object, which is the dynamic mask.

[0017] The resolution in S1.2 adopts a dynamic resolution adjustment strategy. Specifically, when the number of dynamic regions is N, the resolution of each dynamic region is adjusted to... W and H represent the original image width and height, respectively. The total number of pixels remains unchanged, and the empty parts are filled with black pixels to maintain the aspect ratio.

[0018] The mask repair module in step two includes:

[0019] S2.1: Traverse all dynamic regions in the current frame and calculate the area ratio of the dynamic mask to the corresponding dynamic region;

[0020] S2.2: Set a high threshold seg_frac_threshH and a low threshold seg_frac_threshL: When the ratio is greater than seg_frac_threshH, take the intersection of the dynamic mask and the repair mask; when the ratio is less than seg_frac_threshL, take the union of the two; when the ratio is between the two, retain the original dynamic mask.

[0021] S2.3: The repair mask is generated by the region growing algorithm: the median of the effective depth values ​​in the dynamic mask is used as the reference depth d. The region with the depth in the range of d±depth_thresh is found. After the hole is filled by the closing operation and the noise is deleted by the opening operation, the deep continuous region with the largest area is selected as the repair mask.

[0022] S2.4: Perform morphological dilation operation on the final dynamic mask;

[0023] S2.5: If the area of ​​the original dynamic mask or the repair mask is too small, the rectangular dynamic region output by the target detection submodule shall be used as the final dynamic mask.

[0024] Specifically, each dynamic region in the current frame is traversed, and the need for repair is determined based on the proportion of the dynamic mask to its corresponding dynamic region. If repair is required, different repair strategies are selected according to the proportion. When repair is deemed necessary, a region growing algorithm is used to repair the dynamic mask. The algorithm flow is as follows: First, a baseline depth, denoted as d, is calculated based on the depth value of the region where the dynamic mask is located. Starting with the dynamic mask, regions with depths within the range of d ± depth_thresh are searched. Morphological processing is performed on these regions, using closing operations to fill small holes within the region and opening operations to remove isolated noise. Finally, the deep contiguous region with the largest area is selected as the repair mask. A high threshold `seg_frac_threshH` and a low threshold `seg_frac_threshL` are set. When the proportion is greater than `seg_frac_threshH`, the dynamic mask may incorrectly include some static feature points, requiring some to be cropped. Therefore, the final dynamic mask is the intersection of the original dynamic mask and the repaired mask. When the proportion is less than `seg_frac_threshL`, the dynamic mask may incorrectly miss some dynamic feature points, requiring some to be repaired. Therefore, the final dynamic mask is the union of the original dynamic mask and the repaired mask. When the proportion is between these two values, the dynamic mask may just cover the dynamic object, requiring no repair. Morphological dilation is performed on the final dynamic mask to ensure it covers the edges of the dynamic object, preventing the omission of dynamic feature points on the edges. In addition, this module also considers some unexpected situations. When the original dynamic mask area is too small, or the generated repair mask area is too small, it is considered a repair failure. In this case, a rollback strategy is adopted, that is, the entire rectangular dynamic region output by the target detection submodule is used as the final dynamic mask, and dynamic feature points in the dynamic region are deleted at the cost of sacrificing some static feature points.

[0025] The depth_thresh is adaptively adjusted according to the dynamic object category label, and if the number of valid depth values ​​is less than 100, it is determined that the repair has failed and the rollback strategy of S2.5 is executed.

[0026] The motion consistency detection module in step three includes:

[0027] S3.1: Obtain the pose estimation, feature points, and depth map of the current frame and the previous frame;

[0028] S3.2: Feature points near abrupt depth changes are excluded through a depth change edge detection mechanism;

[0029] S3.3: The median sampling method in the window is adopted. A square window is constructed with the feature point as the center. The median of the pixel depth value in the window is calculated as the median depth value of the feature point, and the two-dimensional feature point is converted into a three-dimensional feature point.

[0030] S3.4: Calculate the camera motion based on the camera pose difference between two frames, and inversely deduce the coordinates of the three-dimensional feature points in the current frame in the previous frame;

[0031] S3.5: Compare the calculated depth value with the actual observed median depth value. If the difference exceeds the adaptive threshold, it is determined to be a dynamic feature point and deleted. The adaptive threshold is designed based on the characteristic that the noise of the depth camera increases with the increase of depth. The larger the depth value, the larger the threshold.

[0032] Specifically, pose estimates, feature points, and depth maps of the current and previous frames are obtained for later use. A feature point exclusion mechanism for depth jump edges is used to identify feature points located near abrupt depth jumps. Since the depth values ​​near these jumps are unreliable, these feature points are not included in subsequent motion consistency checks. For each remaining feature point, a window median sampling method is used. A square window with a small side length is constructed with the feature point coordinates as the center. The median depth value of all pixels within the window is calculated as the depth value of that feature point, hereinafter referred to as the median depth value. Finally, the original two-dimensional feature points are transformed into three-dimensional feature points. Based on the camera pose difference between two frames, the camera motion is calculated, and then the coordinates of the three-dimensional feature points in the current frame in the previous frame are inferred based on the camera motion. The calculated depth value of the coordinate point in the previous frame is compared with the median depth value actually observed at that coordinate. If the difference exceeds a threshold, the feature point is considered dynamic and is deleted. The threshold here adopts an adaptive design, based on the characteristic that the noise of the depth camera increases with the increase of the depth value. The larger the depth value, the larger the threshold, to avoid accidentally deleting feature points on the distant background.

[0033] The depth transition edge detection mechanism of S3.2 is as follows: check the depth values ​​of the four points in the cross-shaped neighborhood of the feature point. If the absolute value of the difference between any depth value and the depth value of the feature point is greater than the threshold T, then the feature point is determined to be located at a depth transition edge.

[0034] The difference determination in S3.5 satisfies: absolute difference or relative difference Where adapt_thresh = abs_thresh + 0.02 × ,in This is the median depth value of the feature point in the current frame. This is the absolute value of the difference between the estimated depth value of the feature point in the previous frame and the median depth value actually observed in the previous frame. equal Divide by The business.

[0035] A lightweight dynamic RGB-D VSLAM system based on deep learning includes an RGB-D camera, a processor, and a storage medium. The processor executes instructions in the storage medium to implement a lightweight dynamic feature point removal method based on deep learning.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] Significantly improved positioning accuracy: Validated using the TUM RGBD dynamic dataset (5 sequences including fr3_walking_xyz and fr3_walking_rpy), the ATE root mean square error is reduced by up to 97% in high dynamic scenarios, the MAX error is reduced by 89%-96%, and the STD error is reduced by 91%-97%; in low dynamic scenarios, the ATE root mean square error is reduced by 32%-49%, and the relative pose error (RPE) is significantly reduced. The overall positioning accuracy far exceeds that of the existing ORB-SLAM3 system.

[0038] Significantly enhanced robustness: By repairing the mask through depth map, the problem of decreased recognition rate of dynamic objects caused by RGB image blurring is solved; the motion consistency detection module realizes the accurate removal of feature points of unknown dynamic objects, making up for the inherent defects of deep learning methods; the depth jump edge exclusion mechanism improves the reliability of depth values ​​and further reduces the false judgment rate.

[0039] Real-time performance requirements are met: The system employs a "object detection-cropping-batch semantic segmentation" strategy, lightweight neural networks (YOLOv5, MobileNetv2), and GPU parallel inference, maintaining a frame rate of 15-20fps, which meets the real-time operation requirements of the VSLAM system.

[0040] Wide range of applications: It can be applied to various environments such as high dynamics (crowd movement, vehicle movement) and low dynamics, and is suitable for multiple fields such as autonomous driving, robot navigation, and augmented reality, solving the core problem of localization failure of traditional VSLAM systems in dynamic environments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a system block diagram of a lightweight dynamic feature point removal method based on deep learning according to the present invention.

[0043] Figure 2This is a flowchart of a region growing algorithm for a lightweight dynamic feature point removal method based on deep learning, as described in this invention.

[0044] Figure 3 This is a flowchart of the motion consistency algorithm for a lightweight dynamic feature point removal method based on deep learning, as described in this invention.

[0045] Figure 4 These are experimental results of the 3D camera trajectory of the embodiment of the present invention and ORB-SLAM3 on the fr3_walking_xyz sequence of the TUM dataset.

[0046] Figure 5 The figures show the experimental results of the three-axis absolute trajectory error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_xyz sequence of the TUM dataset.

[0047] Figure 6 The figures show the experimental results of the three-axis rotation error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_xyz sequence of the TUM dataset.

[0048] Figure 7 These are experimental results of the present invention and ORB-SLAM3 on the fr3_walking_rpy sequence of the TUM dataset, showing the 3D camera trajectory.

[0049] Figure 8 The figures show the experimental results of the three-axis absolute trajectory error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_rpy sequence of the TUM dataset.

[0050] Figure 9 The figures show the experimental results of the three-axis rotation error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_rpy sequence of the TUM dataset.

[0051] Figure 10 These are experimental results of the present invention and ORB-SLAM3 on the fr3_walking_static sequence of the TUM dataset, showing the 3D camera trajectory.

[0052] Figure 11 The figures show the experimental results of the triaxial absolute trajectory error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_static sequence of the TUM dataset.

[0053] Figure 12 The figures show the experimental results of the three-axis rotation error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_static sequence of the TUM dataset.

[0054] Figure 13 These are experimental results of the three-dimensional camera trajectory of the embodiment of the present invention and ORB-SLAM3 on the fr3_walking_halfsphere sequence of the TUM dataset.

[0055] Figure 14 The figures show the experimental results of the triaxial absolute trajectory error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_halfsphere sequence of the TUM dataset.

[0056] Figure 15 The figures show the experimental results of the three-axis rotation error of the embodiments of the present invention and ORB-SLAM3 on the fr3_walking_halfsphere sequence of the TUM dataset.

[0057] Figure 16 These are experimental results of the present invention and ORB-SLAM3 on the fr3_sitting_static sequence of the TUM dataset, showing the 3D camera trajectory.

[0058] Figure 17 The figures show the experimental results of the triaxial absolute trajectory error of the embodiments of the present invention and ORB-SLAM3 on the fr3_sitting_static sequence of the TUM dataset.

[0059] Figure 18 The figure shows the experimental results of the triaxial rotation error of the embodiment of the present invention and ORB-SLAM3 on the fr3_sitting_static sequence of the TUM dataset. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] This invention proposes a lightweight dynamic feature point removal method based on deep learning, which utilizes prior semantic information and depth information to identify and remove dynamic feature points while ensuring real-time system operation.

[0062] A lightweight dynamic feature point removal method based on deep learning mainly includes the following steps and their corresponding modules:

[0063] Step 1: The deep learning module first uses a lightweight object detection submodule to extract dynamic regions, and then uses a high-precision semantic segmentation submodule to segment the dynamic regions and extract dynamic masks.

[0064] Step 2: When the semantic segmentation result is poor and dynamic feature points are not completely removed, the mask repair module uses the depth map of the RGB-D camera for repair and enhancement, and uses the region growing algorithm to repair the dynamic mask so that it can cover the vast majority of dynamic feature points.

[0065] Step 3: The motion consistency detection module uses depth constraint detection to remove feature points on dynamic objects that were not detected by the deep learning module.

[0066] like Figure 1 As shown, embodiments of the present invention incorporate a series of dynamic feature point removal methods into visual odometry. These methods primarily consist of a deep learning module, a mask restoration module, and a motion consistency detection module, which are used to implement steps one, two, and three, respectively. This embodiment mainly utilizes RGB and depth maps as inputs. The dynamic mask generated by the deep learning and mask restoration modules first removes dynamic feature points to obtain a preliminary motion estimate. Then, the motion consistency detection module removes any missed dynamic feature points. After this double dynamic feature point removal, conventional VSLAM system steps such as tracking, mapping, and global basis analysis are performed, significantly improving the localization and mapping accuracy of the VSLAM system in dynamic environments.

[0067] like Figure 1 As shown, the deep learning module in step one of this embodiment includes the following sub-steps:

[0068] S1.1: The object detection submodule uses the YOLOv5 network to identify predefined dynamic objects in each frame, such as people, cars, cats, and dogs—objects with a high probability of movement. The detection results are represented as a set of rectangular regions enclosing the pre-dynamic objects, i.e., the dynamic regions. The YOLOv5 network uses the officially provided YOLOv5s.pt pre-trained model, exported in torchscript format, with the input resolution changed to 640×480 (width×height), and deployed and integrated into the VSLAM system using libtorch.

[0069] S1.2: The semantic segmentation submodule first crops the image from the dynamic regions output by the object detection submodule and converts it into tensor form. Since the libtorch framework requires each tensor to have a consistent size during batch inference, the GPU interpolation function is used to unify the resolution of each cropped image, which is faster than using the CPU's `cv:resize` function. To accelerate semantic segmentation, a dynamic resolution adjustment strategy is adopted here: the more dynamic regions there are, the lower the resolution of each dynamic region. For example, if there are N dynamic regions in a frame, the resolution of these dynamic regions is adjusted to... To maintain a constant total pixel count of W×N, the original aspect ratio of the cropped image is preserved during resolution adjustments. Missing areas are filled with black pixels to prevent proportional distortion and maintain semantic segmentation accuracy. All GPU tensors are stacked together and converted into a batch tensor. A high-precision DeepLabv3 network is used for parallel semantic segmentation, outputting a pixel mask of predefined dynamic objects—the dynamic mask. The backbone network of DeepLabv3 is MobileNetv2, trained on the VOC2007 dataset. Because DeepLabv3 uses fully convolutional networks and dilated spatial pyramid pooling, it can accept image input of any resolution.

[0070] like Figure 1 As shown, the mask repair module in step two of this embodiment includes the following sub-steps:

[0071] S2.1: Traverse each dynamic region of the current frame and determine whether repair is needed based on the proportion of the area of ​​the dynamic mask to its respective dynamic region. If repair is needed, select different repair strategies based on the proportion.

[0072] S2.2: When it is determined that repair is needed, a region growing algorithm is used to repair the dynamic mask. The algorithm flow is as follows: Figure 2As shown. The algorithm first calculates a baseline depth, denoted as d, based on the depth values ​​of the region where the dynamic mask is located. Specifically, it records all valid depth values ​​within the area covered by the dynamic mask. If the number of valid depth values ​​is greater than 100, the median of these values ​​is taken as the baseline depth. If the number is less than 100, the depth values ​​within the dynamic mask are considered unreliable, the repair fails, and the backoff strategy in S2.5 is adopted. Then, starting with the dynamic mask, it searches for regions with depths within the range of d ± depth_thresh. Here, depth_thresh is adjusted according to the category label of the dynamic object; for example, when the category is "human," depth_thresh = 0.5 meters. Next, morphological processing is performed on this region. Closing operations are used to fill small holes within the region, and opening operations are used to remove isolated noise points. Finally, the deep contiguous region with the largest area is selected as the repair mask. If the repair mask area is too large or too small compared to the dynamic region, the repair mask is considered unusable, the repair fails, and the backoff strategy in S2.5 is adopted.

[0073] S2.3: Set a high threshold seg_frac_threshH and a low threshold seg_frac_threshL. When the proportion is greater than seg_frac_threshH, the dynamic mask may incorrectly include some static feature points, which need to be cropped. Therefore, the final dynamic mask is the intersection of the original dynamic mask and the repaired mask. When the proportion is less than seg_frac_threshL, the dynamic mask may incorrectly miss some dynamic feature points, which need to be repaired. Therefore, the final dynamic mask is the union of the original dynamic mask and the repaired mask. When the proportion is between the two, the dynamic mask may just cover the dynamic object, and no repair is needed.

[0074] S2.4: Perform morphological dilation on the final dynamic mask to ensure that it covers the edges of the dynamic object and prevent the omission of dynamic feature points on the edges.

[0075] S2.5: In addition, this module also considers some unexpected situations. When the original dynamic mask area is too small, or the generated repair mask area is too small, it is considered as a repair failure. In this case, a fallback strategy is adopted, that is, the entire rectangular dynamic region output by the target detection submodule is used as the final dynamic mask, and dynamic feature points in the dynamic region are deleted at the cost of sacrificing some static feature points.

[0076] like Figure 3 As shown, the motion consistency detection module in step three includes the following sub-steps:

[0077] S3.1: Obtain pose estimates, feature points, and depth maps of the current and previous frames for later use.

[0078] S3.2: A depth abrupt change edge feature point exclusion mechanism is used to identify feature points located near abrupt depth changes. Since depth values ​​sampled at depth edges are often an average of foreground and background values, their reliability is poor, so these feature points are excluded from subsequent motion consistency checks. The specific implementation of this protection mechanism is as follows: For a feature point... Examine the four points within its cross-shaped area respectively. , , , depth value , , and If any one of these four depth values ​​is equal to the feature point depth The absolute value of the difference is greater than the threshold. If so, the feature point is considered to be located at the depth transition edge.

[0079] S3.3: For each remaining feature point, use the median window sampling method. Construct a small square window with side length w centered on the feature point coordinates. Calculate the median depth value of all pixels within the window and use this median as the depth value of the feature point, hereinafter referred to as the median depth value. This transforms the original two-dimensional feature points into three-dimensional feature points.

[0080] S3.4: Based on the camera pose difference between two frames, calculate the camera motion, and then use the camera motion to infer the coordinates of the 3D feature points in the current frame in the previous frame. The specific calculation method is as follows, assuming the transformation matrix from the world coordinate system to the camera coordinate system in the previous frame is... The transformation matrix from the world coordinate system to the camera coordinate system in the current frame is: Let the relative pose transformation matrix between two camera frames be...

[0081]

[0082] in It can be represented as In the form of, For rotation matrix, It is a translation vector.

[0083] set up For a feature point in the current frame's 2D coordinate system, using camera intrinsic parameters and median depth value can Projected onto the current camera's 3D coordinate system, its 3D coordinates are:

[0084]

[0085] Through pose transformation matrix The three-dimensional coordinates of the feature point in the previous frame were calculated as follows:

[0086]

[0087] S3.5: Compare the calculated depth values ​​of the coordinate points And the previous frame The median depth value actually observed at the point location If the difference between the two exceeds a threshold, the feature point is considered dynamic and is deleted. The threshold here employs an adaptive design, based on the characteristic that depth camera noise increases with depth value; the larger the depth value, the larger the threshold, avoiding accidental deletion of feature points in the distant background. Specifically, an absolute threshold `abs_thresh` and a relative threshold `rel_thresh` are set, with the adaptive threshold `adapt_thresh = abs_thresh + 0.02 × ... The absolute difference in depth is defined as The relative difference in depth is defined as ,when or If the feature point is considered dynamic, it will be deleted.

[0088] For easy reference and comparison, the configurations of the embodiments of the present invention are shown in Table 1.

[0089] Table 1

[0090]

[0091] The running results of this invention's embodiment on the TUM dataset for dynamic object sequences are as follows: Figure 4 to Figure 18 The figures shown are experimental results of the embodiments of the present invention and ORB-SLAM3 on five sequences of TUM data: fr3_walking_xyz, fr3_walking_rpy, fr3_walking_static, fr3_walking_halfsphere, and fr3_sitting_static. Each sequence includes a 3D camera trajectory plot, a three-axis absolute trajectory error plot, and a three-axis rotation error plot. Figure Three Individual charts.

[0092] Table 2 shows the ATE (Absolute Trajectory Error) related indicators of the embodiments of the present invention and ORB-SLAM3 for the following five dynamic sequences, where RMSE is the root mean square value, MAX is the maximum value, and STD is the standard deviation. ATE reflects the overall accuracy of the system. In the three high-dynamic sequences fr3_walking_xyz, fr3_walking_rpy, and fr3_walking_halfsphere, the ATE of the embodiments of the present invention reduced RMSE by 92%–97%, MAX by 89%–96%, and STD by 91%–97%, demonstrating that the embodiments of the present invention have a highly significant accuracy improvement in high-dynamic scenarios. In the two low-dynamic sequences fr3_walking_static and fr3_sitting_static, RMSE was reduced by 49% and 32%, MAX by 75% and 50%, and STD by 58% and 54%, respectively, demonstrating that the embodiments of the present invention also have a significant accuracy improvement in low-dynamic environments.

[0093] Table 2

[0094]

[0095] Table 3 shows the RPE (Relative Pose Error) related indicators of the embodiments of the present invention and ORB-SLAM3 in the following five dynamic sequences. RPE reflects the local accuracy of the system, that is, the error between two frames. In the three high dynamic sequences fr3_walking_xyz, fr3_walking_rpy, and fr3_walking_halfsphere, the embodiments of the present invention reduced RMSE by 33% to 51%, MAX by 12% to 76%, and STD by 43% to 69%, which proves that the embodiments of the present invention have a significant improvement in stability under high dynamic scenes. In the two low dynamic sequences fr3_walking_static and fr3_sitting_static, RMSE was reduced by 51% and 2%, MAX by 76% and 77%, and STD by 69% and 4%, respectively. Among them, the reduction ratio is smaller in the fr3_sitting_static sequence because the RPE is already extremely small, reaching the millimeter level. This proves that the embodiments of the present invention also have a significant improvement in accuracy in low dynamic environments.

[0096] Table 3

[0097]

[0098] The average processing time per frame for the embodiments of the present invention and ORB-SLAM3 under five sequences is shown in Table 4. Although the embodiment of the present invention has a reduced running speed, it can maintain the overall system running frame rate at about 15-20 frames per second, ensuring the real-time operation of the system.

[0099] Table 4

[0100]

[0101] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A lightweight dynamic feature point removal method based on deep learning, characterized in that: Acquired using an RGB-D camera The RGB image and depth image are used together to remove dynamic feature points through a deep learning module, a mask restoration module, and a motion consistency detection module. This process includes the following steps: Step 1: The deep learning module first extracts the rectangular dynamic regions of predefined dynamic objects through the lightweight object detection submodule, and then performs batch semantic segmentation on the dynamic regions through the high-precision semantic segmentation submodule, outputting the precise pixel dynamic mask of the predefined dynamic objects. Step 2: The mask repair module determines whether repair is needed based on the area ratio of the dynamic mask to the dynamic region. If repair is needed, the dynamic mask is optimized by combining the region growing algorithm with the depth map to ensure coverage of dynamic feature points. Step 3: The motion consistency detection module uses depth constraint detection to eliminate dynamic feature points that were not detected by the deep learning module.

2. The lightweight dynamic feature point removal method based on deep learning according to claim 1, characterized in that: The deep learning module in step one includes: S1.1: The object detection submodule uses a lightweight neural network to identify predefined dynamic objects in each frame and outputs a rectangular dynamic region that defines the object. S1.2: The semantic segmentation submodule crops dynamic regions and converts them into tensor form. It uses GPU interpolation functions to unify the cropped image resolution, stacks all tensors into batch tensors, and uses a high-precision semantic segmentation network for parallel inference to output a dynamic mask.

3. The lightweight dynamic feature point removal method based on deep learning according to claim 2, characterized in that: The resolution in S1.2 adopts a dynamic resolution adjustment strategy, specifically including: when the number of dynamic regions is N, the resolution of each dynamic region is adjusted to... W and H are the original image width and height, respectively. The total number of pixels remains unchanged, and the empty parts are filled with black pixels to maintain the aspect ratio.

4. The lightweight dynamic feature point removal method based on deep learning according to claim 1, characterized in that: The mask repair module in step two includes: S2.1: Traverse all dynamic regions in the current frame and calculate the area ratio of the dynamic mask to the corresponding dynamic region; S2.2: Set a high threshold seg_frac_threshH and a low threshold seg_frac_threshL: When the ratio is greater than seg_frac_threshH, take the intersection of the dynamic mask and the repair mask; when the ratio is less than seg_frac_threshL, take the union of the two; when the ratio is between the two, retain the original dynamic mask. S2.3: The repair mask is generated by the region growing algorithm: the median of the effective depth values ​​in the dynamic mask is used as the reference depth d. The region with the depth in the range of d±depth_thresh is found. After the hole is filled by the closing operation and the noise is deleted by the opening operation, the deep continuous region with the largest area is selected as the repair mask. S2.4: Perform morphological dilation operation on the final dynamic mask; S2.5: If the area of ​​the original dynamic mask or the repair mask is too small, the rectangular dynamic region output by the target detection submodule shall be used as the final dynamic mask.

5. The lightweight dynamic feature point removal method based on deep learning according to claim 4, characterized in that: The depth_thresh is adaptively adjusted according to the dynamic object category label, and if the number of valid depth values ​​is less than 100, it is determined that the repair has failed and the rollback strategy of S2.5 is executed.

6. The lightweight dynamic feature point removal method based on deep learning according to claim 1, characterized in that: The motion consistency detection module in step three includes: S3.1: Obtain the pose estimation, feature points, and depth map of the current frame and the previous frame; S3.2: Feature points near abrupt depth changes are excluded through a depth change edge detection mechanism; S3.3: The median sampling method in the window is adopted. A square window is constructed with the feature point as the center. The median of the pixel depth value in the window is calculated as the median depth value of the feature point, and the two-dimensional feature point is converted into a three-dimensional feature point. S3.4: Calculate the camera motion based on the camera pose difference between two frames, and inversely deduce the coordinates of the three-dimensional feature points in the current frame in the previous frame; S3.5: Compare the calculated depth value with the actual observed median depth value. If the difference exceeds the adaptive threshold, it is determined to be a dynamic feature point and deleted. The adaptive threshold is designed based on the characteristic that the noise of the depth camera increases with the increase of depth. The larger the depth value, the larger the threshold.

7. A lightweight dynamic feature point removal method based on deep learning according to claim 6, characterized in that: The depth transition edge detection mechanism of S3.2 is as follows: check the depth values ​​of the four points in the cross-shaped neighborhood of the feature point. If the absolute value of the difference between any depth value and the depth value of the feature point is greater than the threshold T, then the feature point is determined to be located at a depth transition edge.

8. A lightweight dynamic feature point removal method based on deep learning according to claim 6, characterized in that: The difference determination in S3.5 satisfies: absolute difference or relative difference Where adapt_thresh = abs_thresh + 0.02 × ,in This is the median depth value of the feature point in the current frame. This is the absolute value of the difference between the estimated depth value of the feature point in the previous frame and the median depth value actually observed in the previous frame. equal Divide by The business.

9. A lightweight dynamic RGB-D VSLAM system based on deep learning, characterized in that, The device includes an RGB-D camera, a processor, and a storage medium, wherein the processor executes instructions in the storage medium to implement the lightweight dynamic feature point removal method based on deep learning as described in any one of claims 1-8.

Images