Visual SLAM (Simultaneous Localization and Mapping) method and system for adaptively optimizing dynamic region

Abstract

The invention belongs to the technical field of image processing and computer vision, and particularly relates to a visual SLAM (Simultaneous Localization and Mapping) method and a visual SLAM system for adaptively optimizing a dynamic region, which are used for carrying out semantic screening based on a multi-class semantic set difference set method and solving the problem of dynamic and static overlapping. More importantly, the strategy of self-adaptive decision based on dynamic point density fundamentally solves the two problems that a bounding box contains too many backgrounds and a slow moving target is improperly processed in the prior art, and intelligent and differentiated processing of a dynamic area is achieved. Through the dual effects of semantic screening and self-adaptive decision making, the method can generate a dynamic area mask which is much more accurate than an original bounding box and an existing fixed process method, especially in scenes that a dynamic object is close to a lens and the like, the boundary can be intelligently shrunk, precious static features mistakenly deleted by a traditional method are reserved to the maximum extent, and the dynamic area mask is more accurate. And the precision and robustness of the SLAM system are directly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and computer vision, and particularly relates to a visual SLAM method and system for adaptive optimization of a dynamic region. BACKGROUND

[0002] In various visual applications, accurately separating moving objects (foreground) and static background from images or video streams is a fundamental and key task. Taking visual SLAM (Simultaneous Localization and Mapping) as an example, its core lies in estimating the motion trajectory of a camera and constructing an environment map by matching and tracking stable static feature points in consecutive image frames. However, when using a target detection network (such as YOLO) to assist in removing dynamic objects, a thorny problem is often encountered: the target detection network outputs a rectangular bounding box that encloses the object. This representation is too rough and can cause a series of problems, severely affecting the performance of downstream tasks.

[0003] Specifically, the disadvantages are reflected in the following two aspects: First, the bounding box contains a large amount of irrelevant background. A typical example is that when a "person" is detected, the bounding box not only contains the body contour of the person, but also inevitably frames a large area of static background region behind the person, such as walls, carpets, and distant furniture. If a "one-size-fits-all" strategy is adopted to delete all visual feature points within the bounding box, these high-quality static feature points that can be used for accurate positioning and mapping are also sacrificed. This can lead to a sharp reduction in the number of available feature points, especially when dynamic objects are close to the lens and occupy a large part of the screen, which can directly cause the SLAM system to fail due to insufficient features for matching.

[0004] Second, it cannot handle the overlap and interaction of dynamic and static objects. In complex scenes, occlusion and overlap between objects are very common. For example, a person sitting at work (high dynamic target) may have his bounding box overlap with the bounding box of the computer monitor (low dynamic / static target) in front of him, or even completely contain the latter. If removal is performed only according to the bounding box of the "person", the features on the monitor that should be stable landmarks will also be incorrectly filtered out, which also weakens the robustness of the system.

[0005] Some existing methods attempt to use pixel-level instance segmentation networks (such as Mask R-CNN) to obtain more refined object contours, but the computational amount of such networks is huge, and it is difficult to meet the real-time requirements of SLAM and other applications. Therefore, how to refine an accurate mask (Mask) that is closer to the real motion range of the dynamic object from the rough boundary box provided by target detection without sacrificing real-time performance is the core key to improving the performance of various dynamic scene visual applications. SUMMARY

[0006] To solve the problems raised in the background art, the present application provides a visual SLAM method and system for adaptively optimizing dynamic regions.

[0007] The technical solutions of the present application are as follows: The present application provides a visual SLAM method for adaptively optimizing dynamic regions, comprising: S1: obtaining a current frame image, performing forward reasoning using a target detection network to obtain class labels and boundary box coordinate values of detected objects; S2: dividing the detected objects into a high-dynamic set and a non-high-dynamic set according to the motion attributes of the detected objects; S3: performing a set operation on the boundary boxes of all objects in the high-dynamic set and the non-high-dynamic set respectively, setting the pixel values of the regions obtained by the operation to 1 and the pixel values of the remaining regions to 0, respectively generating a high-dynamic object mask image and a non-high-dynamic object mask image, and subtracting the overlapping region of the high-dynamic object mask image from the non-high-dynamic object mask image to generate a preliminary dynamic region mask image; S4: extracting all ORB feature points of the preliminary dynamic region mask image, calculating the three-dimensional space reprojection error of each ORB feature point between the current frame and the historical key frame, and marking the ORB feature point as a dynamic feature point if the three-dimensional space reprojection error is greater than a preset error; S5: regarding the ratio of the number of dynamic feature points to the number of ORB feature points in each independent connected region composed of boundary boxes in the preliminary dynamic region mask image as the dynamic point density of the region; S6: if the dynamic point density of the region is not less than a dynamic point density threshold, using a region generation method based on depth information fusion to generate an irregular mask for the region after region growing with the identified dynamic feature points as seeds on the depth map corresponding to the region and performing a morphological dilation operation; If the dynamic point density of the region is less than the dynamic point density threshold, the position of the rectangular boundary box corresponding to the region is kept unchanged; S7: merging the irregular mask and the dynamic region corresponding to the rectangular boundary box to obtain a target dynamic region mask image.

[0008] Further, the S6 adopts the region generation method based on depth information fusion, and after region growing is performed on the depth map corresponding to the region with the identified dynamic feature points as seeds, a morphological dilation operation is performed to generate an irregular mask of the region, specifically: A blank binary mask with the same size as the current frame image is created, and all pixel values are initialized to 0; All dynamic feature points in the region are traversed, and for each dynamic feature point, the region growing operation is performed on the corresponding depth image with the dynamic feature point as a seed point; the local region obtained by each region growing is accumulated in the initialized binary mask through logical OR operation; after all seed points complete the region growing, a preliminary mask of the region is generated; The preliminary mask of the region is subjected to a morphological dilation operation with an elliptical structure element to generate an irregular mask of the region.

[0009] The region growing operation is specifically: Starting from the seed point, the seed point is diffused to its neighborhood pixels, and as long as the difference between the depth value of the neighborhood pixel and the depth value of the seed point is less than a preset depth threshold, the neighborhood pixel is included in the current growing region.

[0010] The S4 history key frame is marked as a history key frame and stored if at least one of the following conditions is met: (1) Motion distance condition: the camera motion distance from the last key frame exceeds a preset translation threshold or rotation threshold; (2) Co-view feature point condition: the number of co-view feature points between the current frame and the latest key frame is less than a preset number; (3) Frame interval condition: a fixed number of frames have passed since the last key frame was inserted; (4) Number of tracked feature points condition: the number of ORB feature points in the current frame is less than a safety threshold.

[0011] The S3 performs a set operation on the bounding boxes of all objects in the high dynamic set and the non-high dynamic set, respectively, sets the pixel values of the obtained regions to 1 and the pixel values of the remaining regions to 0, generates a high dynamic object mask image and a non-high dynamic object mask image, respectively, and subtracts the overlapping region of the high dynamic object mask image from the non-high dynamic object mask image to generate a preliminary dynamic region mask image, specifically: The bounding box regions of all objects in the high dynamic set are subjected to a set operation, the pixel values of the obtained regions are set to 1, and the pixel values of the remaining regions are set to 0 to form a high dynamic object mask image; The bounding box regions of all objects in the non-high dynamic set are subjected to a set operation, the pixel values of the obtained regions are set to 1, and the pixel values of the remaining regions are set to 0 to form a non-high dynamic object mask image; Subtracting the overlapping part of the high dynamic object mask image from the high dynamic and non-high dynamic object mask image through pixel-level logical operation, a preliminary dynamic region mask image is generated.

[0012] Further, the subtraction of the overlapping part of the high dynamic object mask image from the high dynamic and non-high dynamic object mask image is: , realized; In the formula, , , The preliminary dynamic region mask image, the high dynamic object mask image and the non-high dynamic object mask image are represented respectively.

[0013] S3 further includes, before calculating the overlapping region: For any two boundary boxes belonging to the high dynamic set and the non-high dynamic set respectively, the intersection-over-union of the two is calculated, when the intersection-over-union is greater than a preset intersection-over-union, then the pixel regions corresponding to the two boundary boxes are compared pixel by pixel, and the overlapping pixel region of the two is accurately extracted; when the intersection-over-union is not greater than the preset intersection-over-union, then the overlapping region is calculated through the boundary box coordinates.

[0014] S2 divides the detected object into the high dynamic set and the non-high dynamic set according to the motion attribute of the detected object, and specifically: Two semantic sets, the high dynamic set and the non-high dynamic set, are predefined according to the motion attribute of the detected object; all objects detected by S1 are classified into the corresponding set according to the category label of the detected object.

[0015] S7 further includes applying the target dynamic region mask image to the visual SLAM, and accurately removing the dynamic feature points of the current frame according to the target dynamic region mask image.

[0016] The application further provides a visual SLAM system for adaptively optimizing a dynamic region, comprising: A preprocessing module: acquiring a current frame image, performing forward reasoning by using a target detection network, and obtaining a category label and boundary box coordinate value of a detected object; A set division module: dividing the detected object into the high dynamic set and the non-high dynamic set according to the motion attribute of the detected object; A preliminary dynamic region mask image generation module: performing a set operation on the boundary boxes of all objects in the high dynamic set and the non-high dynamic set respectively, setting the pixel value of the region obtained by the operation to 1 and the pixel value of the remaining region to 0, generating a high dynamic object mask image and a non-high dynamic object mask image respectively, and generating a preliminary dynamic region mask image by subtracting the overlapping region of the high dynamic object mask image from the non-high dynamic object mask image. Dynamic feature point marking module: extract all ORB feature points of the preliminary dynamic region mask image, calculate the three-dimensional space reprojection error between each ORB feature point and the historical key frame, if the three-dimensional space reprojection error is greater than the preset error, mark the ORB feature point as a dynamic feature point; Dynamic point density calculation module: for each independent connected region composed of a bounding box in the preliminary dynamic region mask image, the ratio of the number of dynamic feature points to the number of ORB feature points in the region is taken as the dynamic point density of the region; Adaptive decision module: if the dynamic point density of the region is not less than the dynamic point density threshold, the region generation method based on depth information fusion is adopted, after region growing on the depth map corresponding to the region with the identified dynamic feature points as seeds, morphological dilation operation is performed to generate the irregular mask of the region; If the dynamic point density of the region is less than the dynamic point density threshold, the position of the rectangular bounding box corresponding to the region is kept unchanged; Target dynamic region mask image generation module: combine the irregular mask and the dynamic region corresponding to the rectangular bounding box as the target dynamic region mask image.

[0017] Advantages The present application is based on the method of multi-class semantic set difference, which performs "semantic screening" and solves the dynamic and static overlap problem; more importantly, the "adaptive decision based on dynamic point density" strategy fundamentally solves the two problems of "too much background contained in the bounding box" and "inappropriate processing of slow moving targets" in the prior art, and realizes intelligent and differentiated processing of dynamic regions. Through the dual effects of "semantic screening" and "adaptive decision", the present application can generate a dynamic region mask much more accurate than the original bounding box and the existing fixed process method, especially in scenes where dynamic objects are close to the lens, the boundary can be intelligently contracted, the valuable static features mistakenly deleted by traditional methods are maximally retained, and the accuracy and robustness of the SLAM system are directly improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A schematic diagram for performing preliminary screening based on semantics by using the method of the present application.

[0019] Figure 2 A graph of absolute trajectory error (ATE) for one period on the TUM RGB-D dataset fr3 / walking_xyz sequence by using the method of the present application.

[0020] Figure 3 A graph of deviation in (x, y, z) direction between the estimated trajectory and the ground truth trajectory on the TUM RGB-D dataset fr3 / walking_xyz sequence by using the method of the present application.

[0021] Figure 4 Figure 8 shows the deviation map of the (y, p, r) direction of the estimated trajectory and the ground truth trajectory on the TUM RGB-D dataset fr3 / walking_xyz sequence using the method of the present application. DETAILED DESCRIPTION

[0022] The following examples are intended to illustrate the present application and are not further limiting.

[0023] The present application provides a visual SLAM method for adaptively optimizing dynamic regions, comprising: S1: obtaining a current frame image, performing forward inference using a target detection network to obtain a class label and a bounding box coordinate value of a detected object.

[0024] For example, an RGB image with a resolution of 640x480 is obtained, and a YOLOv11 target detection network that has been trained is used to perform forward inference according to a convolution layer→an activation layer→a pooling layer→a fully connected layer / detection head, and give a prediction result, obtaining a list of all detected objects, each object containing information: {class label, bounding box coordinates}.

[0025] S2: dividing the detected objects into a high dynamic set and a non-high dynamic set according to the motion attributes of the detected objects. Specifically: Two semantic sets, a high dynamic set and a non-high dynamic set, are predefined according to the motion attributes of the detected objects; all detected objects in S1 are iterated, and the class label of each detected object is used to classify it into the corresponding set.

[0026] The high dynamic set contains object classes with high motion possibilities, such as person and car; the non-high dynamic set contains object classes with low motion possibilities or that are stationary, such as table, chair, and monitor. When a "person" class object is detected, it is placed in the high dynamic set, and when a "monitor" class object is detected, it is placed in the non-high dynamic set.

[0027] S3: performing a union operation on the bounding boxes of all objects in the high dynamic set and the non-high dynamic set, respectively, setting the pixel values of the resulting regions to 1 and the pixel values of the remaining regions to 0, respectively, to generate a high dynamic object mask image and a non-high dynamic object mask image, and subtracting the overlapping region of the high dynamic object mask image from the non-high dynamic object mask image to generate a preliminary dynamic region mask image.

[0028] The purpose of this step is to solve the dynamic and static overlap problem through semantic screening, and the present application uses a method based on the difference set of multiple semantic sets to do so. Specifically: The boundary box area of all objects in the high dynamic set is union operated, the pixel value of the obtained area is set to 1, the pixel value of the remaining area is set to 0, and a high dynamic object mask image is formed; The boundary box area of all objects in the non-high dynamic set is union operated, the pixel value of the obtained area is set to 1, the pixel value of the remaining area is set to 0, and a non-high dynamic object mask image is formed; Through pixel-level logical operation, the overlapping part of the high dynamic and non-high dynamic object mask images is subtracted from the high dynamic object mask image to generate a preliminary dynamic region mask image.

[0029] Further, the subtraction of the overlapping part of the high dynamic and non-high dynamic object mask images from the high dynamic object mask image is: , realized; In the formula, , , respectively represent the preliminary dynamic region mask image, the high dynamic object mask image, and the non-high dynamic object mask image.

[0030] As shown in Figure 1 , for example, when the boundary box of “person” (belonging to ) overlaps with the boundary box of “display” (belonging to ), the pixel value of the overlapping area in is 0, and the static feature points in this area are retained.

[0031] In addition, S3 further includes, before calculating the overlapping area: For any two boundary boxes belonging to the high dynamic set and the non-high dynamic set respectively, the intersection-over-union ratio of the two is calculated, and when the intersection-over-union ratio is greater than a preset intersection-over-union ratio (such as 0.4), the pixel regions corresponding to the two boundary boxes are compared pixel by pixel to accurately extract the overlapping pixel region of the two; when the intersection-over-union ratio is not greater than the preset intersection-over-union ratio, the overlapping area is calculated through the boundary box coordinates.

[0032] This is because, when the intersection-over-union is greater than the preset intersection-over-union, it means that the two types of object bounding boxes overlap to a high degree (for example, the bounding box of the "person" and the bounding box of the "display" overlap a lot), and if the simplified operation is used, it can cause the static features in the overlapping region (such as the display surface features) to be mistakenly deleted, so the pixel-level operation is performed. When the intersection-over-union is not greater than the preset intersection-over-union, the two types of object bounding boxes overlap to a very low degree or even do not overlap (for example, the "walking person" and the "distant cabinet"), at this time, the influence of the overlapping region on the static features can be ignored. If the pixel-level operation is still performed, it will increase unnecessary calculation (especially when the number of detected objects is large, the pixel-level comparison is time-consuming). Therefore, at this time, the simplified box-level operation is used, which greatly reduces the operation time, meets the "real-time" requirement of visual SLAM, and does not affect the core static feature retention effect.

[0033] S4: Extract all ORB (Oriented FAST and Rotated BRIEF) feature points of the preliminary dynamic region mask image, calculate the three-dimensional space reprojection error of each ORB feature point between the current frame and the historical key frame, and if the three-dimensional space reprojection error is greater than a preset error (such as 0.05 meters), mark the ORB feature point as a dynamic feature point.

[0034] Preferably, the historical key frame is a representative special frame selected from continuous video image frames during the running of visual SLAM according to a series of conditions. Specifically, if a certain historical frame meets at least one of the following conditions, it is marked as a historical key frame and stored: (1) Motion distance condition: the camera motion distance from the last key frame exceeds a preset translation threshold (such as 0.1 meters) or a rotation threshold (such as 5 degrees), ensuring that there is enough parallax between the new and old key frames for three-dimensional geometric calculation; (2) Co-visible feature point condition: the number of co-visible feature points between the current frame and the latest key frame is less than a preset number, indicating that the scene has changed significantly, and a new key frame needs to be inserted to record the new information of the environment; (3) Frame interval condition: a fixed number of frames (such as 30 frames) have passed since the last key frame was inserted, ensuring that when the camera is moving slowly or stationary, key frames are still generated periodically to maintain stable positioning; (4) Number of tracked feature points condition: the number of ORB feature points in the current frame is less than a safety threshold (such as 50), forcing the insertion of a key frame to prevent tracking loss.

[0035] By comparing with these historical key frames that have been strictly screened and have high information content and long-term stability, the present application can obtain a more stable and robust motion judgment than the method in the prior art that only compares adjacent frames, thereby significantly improving the accuracy of dynamic feature point recognition.

[0036] S5: For each independent connected region (corresponding to one or more closely adjacent high dynamic objects) composed of the bounding box in the preliminary dynamic region mask image, take the ratio of the number of dynamic feature points to the number of ORB feature points in the region as the dynamic point density of the region.

[0037] S6: If the dynamic point density of the region is not less than the dynamic point density threshold, a region generation method based on depth information fusion is adopted. After region growing is performed on the depth image corresponding to the region with the identified dynamic feature points as seeds, a morphological dilation operation is performed to generate an irregular mask for the region. If the dynamic point density of the region is less than the dynamic point density threshold, the position of the rectangular bounding box corresponding to the region is kept unchanged.

[0038] This step aims to intelligently, differentially and adaptively process dynamic regions according to the distribution of dynamic features. According to the level of dynamic point density, it is decided which way to generate the final dynamic exclusion region. The generated result can be an irregular accurate mask or an original rectangular bounding box.

[0039] When the dynamic point density is high, it indicates that the objects in the region are moving significantly. A region generation method based on depth information fusion is adopted to generate an irregular mask that tightly wraps the dynamic objects. The reason for introducing depth information is that pixel points belonging to the surface of the same object usually have continuously changing depth values. Compared with relying only on color or texture information, depth information can more accurately distinguish between foreground dynamic objects and background, especially when the object and the background color are similar. Through depth similarity judgment, discrete feature points can be expanded into complete object surfaces, avoiding missing the textureless areas of the object.

[0040] Preferably, the region generation method based on depth information fusion in S6, after region growing is performed on the depth image corresponding to the region with the identified dynamic feature points as seeds, a morphological dilation operation is performed to generate an irregular mask for the region, specifically: A blank binary mask with the same size as the current frame image is created, and all pixel values are initialized to 0; All dynamic feature points in the region are traversed. For each dynamic feature point, it is taken as a seed point, and a region growing operation is performed on the corresponding depth image. The local region obtained by each region growing operation is accumulated in the initialized binary mask through logical "or" operation. After all seed points complete region growing, the binary mask contains the regions expanded from all dynamic feature points, forming a preliminary mask for the region. In order to connect the separated small areas in the preliminary mask of the region into a whole and fill the possible holes in the object, the preliminary mask of the region is subjected to a morphological dilation operation with an elliptical structure element (for example, 31 pixels in diameter) to ensure that the whole contour of the moving object is completely contained, thereby generating an accurate dynamic mask of irregular shape of the region.

[0041] Further, the region growing operation specifically includes: Starting from a seed point, the seed point spreads to its neighborhood pixels, and as long as the depth value of the neighborhood pixel is less than the depth value of the seed point by a preset depth threshold (for example, 0.2 meters), the neighborhood pixel is included in the current growing region.

[0042] When the dynamic point density is low, it indicates that the dynamic features in the region are sparse, which may correspond to an object that moves very slowly or only has local limb movement. In order to avoid over-segmentation of such targets and lose effective dynamic information, the present application will adopt a robust conservative strategy: keeping the position of the rectangular bounding box corresponding to the region unchanged.

[0043] Through the above adaptive decision-making process, the dynamic region for filtering feature points obtained by the present application is the result of intelligent selection according to the scene dynamics: either an irregular mask for accurate rejection or a rectangular bounding box for robust rejection. In the subsequent visual SLAM process, only the feature points falling into these finally determined dynamic regions will be rejected.

[0044] By introducing the dynamic point density as a judgment basis, the present application can flexibly cope with various complex dynamic situations, whether it is a fast-moving object or a slow-moving object or even only a local motion (such as a person waving his hand), and can make reasonable processing. This overcomes the defects of the prior art that the fixed geometric model (scene flow + Gaussian model) is not sensitive to slow motion and is prone to misjudgment, and has stronger scene adaptability.

[0045] S7: Merge the dynamic regions corresponding to the irregular mask and the rectangular bounding box as a target dynamic region mask image.

[0046] After S7, the target dynamic region mask image is applied to visual SLAM, and the dynamic feature points of the current frame are accurately rejected according to the target dynamic region mask image.

[0047] The application is based on a multi-class semantic set difference method, performs "semantic screening", and solves the dynamic-static overlap problem; more importantly, the "adaptive decision based on dynamic point density" strategy fundamentally solves the two problems of "too much background included in the bounding box" and "inappropriate processing of slow-moving targets" in the prior art, and realizes intelligent and differentiated processing of dynamic regions. Through the dual effects of "semantic screening" and "adaptive decision", the application can generate a dynamic region mask that is much more accurate than the original bounding box and the existing fixed process method. Especially in scenes where dynamic objects are close to the lens, this method can intelligently shrink the boundary, maximize the retention of valuable static features that are mistakenly deleted by traditional methods, and directly improve the accuracy and robustness of the SLAM system.

[0048] The application also provides a visual SLAM system for adaptively optimizing a dynamic region, comprising: A preprocessing module: acquires a current frame image, performs forward reasoning by using a target detection network, and obtains class labels and bounding box coordinate values of detected objects; A set division module: divides the detected objects into a high-dynamic set and a non-high-dynamic set according to the motion attributes of the detected objects; A preliminary dynamic region mask image generation module: performs a union operation on the bounding boxes of all objects in the high-dynamic set and the non-high-dynamic set respectively, sets the pixel values of the regions obtained by the operation to 1 and the pixel values of the remaining regions to 0, generates a high-dynamic object mask image and a non-high-dynamic object mask image respectively, and generates a preliminary dynamic region mask image by subtracting the overlapping region of the high-dynamic object mask image from the non-high-dynamic object mask image; A dynamic feature point marking module: extracts all ORB feature points of the preliminary dynamic region mask image, calculates the three-dimensional space reprojection error of each ORB feature point between the current frame and the historical key frame, and marks the ORB feature point as a dynamic feature point if the three-dimensional space reprojection error is greater than a preset error; A dynamic point density calculation module: takes the ratio of the number of dynamic feature points to the number of ORB feature points of each independent connected region composed of bounding boxes in the preliminary dynamic region mask image as the dynamic point density of the region; An adaptive decision module: if the dynamic point density of the region is not less than a dynamic point density threshold, a region generation method based on depth information fusion is adopted, a morphological dilation operation is performed after region growing on the depth map corresponding to the region with the identified dynamic feature points as seeds, and an irregular mask of the region is generated; If the dynamic point density of the region is less than the dynamic point density threshold, the position of the rectangular bounding box corresponding to the region is kept unchanged; A target dynamic region mask image generation module: combines the irregular mask and the dynamic region corresponding to the rectangular bounding box as a target dynamic region mask image.

[0049] Experimental results and analysis To verify the effectiveness of the "semantic filtering + adaptive decision-making" two-stage optimization method described in the present application, the following comparative experiments are designed and implemented.

[0050] 1. Experimental environment and settings Hardware platform: All experiments are performed in a unified hardware environment to ensure fairness. The platform is configured as follows: Intel Core i5 CPU, 16 GB of RAM, and a NVIDIA GeForce RTX 4060 GPU.

[0051] Software environment: The operating system is Ubuntu, based on the Robot Operating System (ROS) framework, and the method described in the present application is implemented using C++ and Python languages.

[0052] Test data set: The TUM RGB-D data set, a widely recognized authoritative evaluation data set in the industry, is selected. To fully verify the performance of the present application in complex dynamic scenarios, the high dynamic sequence fr3 / walking_xyz is selected for testing. This sequence is characterized by the presence of two freely walking test subjects, whose movements generate both "dynamic occlusion" (e.g., a person walking in front of a display, blocking it) and "bounding box containing a large amount of background" (a person approaching the camera, with their detection box occupying a large area of the picture).

[0053] Core evaluation index: The most core precision evaluation index in the field of visual SLAM, the Absolute Trajectory Error (ATE), is used. This index quantifies the global positioning accuracy by directly comparing the camera estimation trajectory of the present application method with the Ground Truth trajectory recorded by a high-precision motion capture system. The Root Mean Square Error (RMSE) of the Absolute Trajectory Error (ATE) is calculated, and the smaller the value, the higher the positioning accuracy of the present application method.

[0054] 2. Core parameter selection and basis The present application method includes a key parameter: dynamic point density threshold, which is used to decide whether to shrink the bounding box of dynamic regions. To determine its optimal value, a parameter sensitivity analysis experiment is conducted. On the fr3 / walking_xyz sequence, the threshold is tested from 20% to 60%, and the results are shown in the following table: Table 1 Effect of dynamic point density threshold on root mean square error performance (unit: m) As shown in Table 1, the ATE RMSE reaches its minimum value of 0.0157 meters when the threshold is set to 40%. If the threshold is too low (e.g., 20%), the system becomes overly sensitive and may over-shrink some slowly moving objects, resulting in the loss of some features; if the threshold is too high (e.g., 60%), it cannot effectively shrink truly high-density dynamic regions, leading to the removal of static background features. Therefore, in subsequent experiments, the dynamic point density threshold was determined to be 40%, which achieves the optimal balance between removing dynamic interference and preserving static features.

[0055] 3. Performance Comparison and Verification To verify the effectiveness and advancement of the method in real-world dynamic environments, four typical high-dynamic walking sequences (fr3 / walking_static, fr3 / walking_xyz, fr3 / walking_halfsphere, fr3 / walking_rpy) from the TUM RGB-D dataset were selected for accuracy comparison tests. These sequences comprehensively cover various motion modes, including camera stationary, translation, hemispherical motion, and rotation, while two people are always freely moving within the scene, posing a significant challenge to the robustness of the SLAM system.

[0056] The method of this invention is compared with several mainstream SLAM methods currently recognized in the field, including: ORB-SLAM2: A classic baseline method that does not handle dynamic objects.

[0057] Dyna-SLAM: An excellent dynamic SLAM method that combines multi-view geometry with Mask R-CNN instance segmentation.

[0058] All experiments were run on a unified hardware and software platform to ensure fairness in the comparison. The evaluation metric used was the root mean square error (RMSE) of the absolute trajectory error (ATE), expressed in meters (m). A smaller RMSE value indicates higher positioning accuracy. The comparison results are shown in the table below: Table 2. Performance comparison of different methods on fr3 / walking dynamic sequences Table 2 shows the experimental data, which clearly demonstrates the superior performance and high robustness of the method described in this invention. The analysis is as follows: Firstly, compared with the baseline method ORB-SLAM2, the accuracy of the present application has achieved an order of magnitude improvement. As can be seen from Table 2, in all high dynamic sequences, the trajectory error of ORB-SLAM2 without processing dynamic objects is extremely large (RMSE is generally between 0.37 meters and 0.8 meters), and almost cannot complete effective positioning. While the RMSE value of the present application method on all the same sequences is stable between 0.01 meters and 0.035 meters. For example, in the most complex fr3 / walking_xyz sequence, the present application greatly reduces the error from 0.7468 meters to 0.0157 meters, and the accuracy is improved by more than 47 times. This fully shows that in a dynamic environment, accurate identification and rejection of dynamic areas are necessary prerequisites for the normal work of the SLAM system, and the two-stage optimization strategy proposed by the present application is extremely effective.

[0059] Secondly, compared with the advanced dynamic SLAM method Dyna-SLAM, the present application method shows strong competitiveness and achieves the best performance in multiple complex scenes.

[0060] In the two sequences fr3 / walking_halfsphere and fr3 / walking_rpy containing complex camera motion, the accuracy of the present application method is ranked first, indicating that the "semantic screening + adaptive decision" framework of the present application has the best suppression effect on dynamic interference under various camera poses.

[0061] In the fr3 / walking_xyz sequence, the accuracy (0.0157 meters) of the present application is significantly better than that of Dyna-SLAM (0.0269 meters).

[0062] The core reason for this series of superior performance lies in the two-stage fine processing flow of the present application: The "semantic screening" of the first stage successfully solves the feature deletion problem when the dynamic target (person) blocks the static landmark (such as display, table) through the innovative set difference operation.

[0063] The "adaptive decision" of the second stage intelligently solves the problem of the detection frame containing a large number of irrelevant static backgrounds by adaptively adjusting the bounding box based on the dynamic point density, and maximizes the retention of valuable static feature points for the SLAM system.

[0064] Moreover, Figure 2 Intuitively, it shows that the camera estimation trajectory of the present application method is highly consistent with the ground true trajectory, verifying the extremely low root mean square error value in Table 2. Figure 3 And Figure 4Further from (x,y,z) translation and (roll,pitch,yaw) rotation, wherein roll, pitch and yaw represent roll angle, pitch angle and yaw angle respectively, the deviation of the pose estimated by the method of the application from the time change is shown, and the curve jitter is small, which again proves that the application can still maintain high stability and accuracy under strong dynamic interference.

[0065] In summary, the experimental results fully demonstrate that the application effectively overcomes the limitations of traditional methods by the synergistic optimization of semantic classification and adaptive decision-making, can generate more accurate dynamic region masks, and thus significantly improves the positioning accuracy and overall robustness of the visual SLAM system in various complex dynamic scenes.

Claims

1. A method for self-adaptive optimization of dynamic regions for visual SLAM, characterized in that, Comprise: S1: obtain the current frame image, utilize the target detection network to carry out forward inference, obtain the class label and the boundary box coordinate value of the detected object; S2: according to the motion attribute of the detected object, the detected object is divided into high dynamic set, non high dynamic set; S3: respectively for the union operation of the boundary box of all objects in high dynamic set, non high dynamic set, the pixel value of the region obtained by operation is set to 1, the rest region is set to 0, respectively generate high dynamic object mask image and non high dynamic object mask image, subtract the overlap area of high dynamic object mask image and non high dynamic object mask image, generate preliminary dynamic region mask image; S4: extract all ORB feature points of preliminary dynamic region mask image, calculate the three-dimensional space re-projection error between each ORB feature point and historical key frame, if the three-dimensional space re-projection error is greater than the preset error, then the ORB feature point is marked as dynamic feature point; S5: for each independent connected region composed of boundary box in preliminary dynamic region mask image, the ratio of the number of dynamic feature points and the number of ORB feature points in the region is taken as the dynamic point density of the region; S6: if the dynamic point density of the region is not less than the dynamic point density threshold, then the region generation method based on depth information fusion is used, after region growing is carried out on the depth image corresponding to the region with the identified dynamic feature points as seeds, morphological dilation operation is carried out, and the irregular mask of the region is generated; If the dynamic point density of the region is less than the dynamic point density threshold, then keep the position of the rectangular boundary box corresponding to the region unchanged; S7: the irregular mask and the dynamic region corresponding to the rectangular boundary box are merged as the target dynamic region mask image.

2. The method of claim 1, wherein, The region generation method based on depth information fusion in S6, after region growing is carried out on the depth image corresponding to the region with the identified dynamic feature points as seeds, morphological dilation operation is carried out, and the irregular mask of the region is generated, specifically: Create a blank binary mask with the same size as the current frame image, all pixel values are initialized to 0; Iterate through all dynamic feature points in the region, for each dynamic feature point, take it as a seed point, execute region growing operation once on the corresponding depth image; the local region obtained by each region growing is accumulated in the initialized binary mask through logical "or" operation; after all seed points complete region growing, the preliminary mask of the region is generated; Perform morphological dilation operation once on the preliminary mask of the region with elliptical structure element, generate the irregular mask of the region.

3. The method of claim 2, wherein, The region growing operation, specifically: From the seed point, spread to its neighborhood pixels, as long as the difference between the depth value of the neighborhood pixel and the depth value of the seed point is less than the preset depth threshold, then the neighborhood pixel is included in the current growing region. 4.The method of claim 1, wherein, The historical key frame in S4, if a historical frame satisfies at least one of the following conditions, it is marked as a historical key frame and stored: (1) motion distance condition: the camera motion distance from the last key frame exceeds the preset translation threshold or rotation threshold; (2) The co-view feature point condition: the number of co-view feature points of the current frame and the latest key frame is lower than a preset number; (3) The frame interval condition: a fixed number of frames has passed since the last key frame insertion; (4) The number of tracked feature points condition: the number of ORB feature points of the current frame is lower than a safety threshold.

5. The method of claim 1, wherein, S3 respectively performs set operation on the bounding boxes of all objects in the high-dynamic set and the non-high-dynamic set, sets the pixel values of the obtained regions to 1 and the pixel values of the remaining regions to 0, respectively generates a high-dynamic object mask image and a non-high-dynamic object mask image, and subtracts the overlapping region of the high-dynamic object mask image from the non-high-dynamic object mask image to generate a preliminary dynamic region mask image, specifically as follows: performing set operation on the bounding box regions of all objects in the high-dynamic set, setting the pixel values of the obtained regions to 1 and the pixel values of the remaining regions to 0, and forming a high-dynamic object mask image; performing set operation on the bounding box regions of all objects in the non-high-dynamic set, setting the pixel values of the obtained regions to 1 and the pixel values of the remaining regions to 0, and forming a non-high-dynamic object mask image; subtracting the overlapping part of the high-dynamic object mask image from the non-high-dynamic object mask image through pixel-level logical operation to generate a preliminary dynamic region mask image.

6. The method of claim 5, wherein, Subtracting the overlapping part of the high dynamic object mask image and the high dynamic and non-high dynamic object mask image from the high dynamic object mask image is implemented by the formula: , and the implementation is realized. wherein , , respectively represent the preliminary dynamic region mask image, the high dynamic object mask image, and the non-high dynamic object mask image.

7. The method of claim 1, wherein, S3 further includes the following before calculating the overlapping region: For any two bounding boxes respectively belonging to the high-dynamic set and the non-high-dynamic set, the intersection-over-union ratio of the two is calculated, when the intersection-over-union ratio is greater than a preset intersection-over-union ratio, the pixel regions corresponding to the two bounding boxes are compared pixel by pixel to accurately extract the overlapping pixel region of the two; when the intersection-over-union ratio is not greater than the preset intersection-over-union ratio, the overlapping region is calculated through the bounding box coordinates. 8.The method for visual SLAM of self-adaptive optimization dynamic area according to claim 1, characterized in that, S2 divides the detected objects into the high-dynamic set and the non-high-dynamic set according to the motion attributes of the detected objects, specifically as follows: Two semantic sets, the high-dynamic set and the non-high-dynamic set, are predefined according to the motion attributes of the detected objects; all objects detected by S1 are iterated, and the objects are respectively classified into the corresponding set according to the class label of the detected object. 9.The method for visual SLAM of self-adaptive optimization dynamic area according to claim 1, characterized in that, S7 further includes applying the target dynamic region mask image to visual SLAM to accurately remove the dynamic feature points of the current frame according to the target dynamic region mask image.

10. A self-adapting optimizing dynamic region visual SLAM system, characterized in that, It includes: a preprocessing module: obtaining a current frame image, performing forward reasoning by using a target detection network to obtain a class label and bounding box coordinate value of a detected object; a set division module: dividing the detected object into the high-dynamic set and the non-high-dynamic set according to the motion attribute of the detected object; a preliminary dynamic region mask image generation module: performing set operation on the bounding boxes of all objects in the high-dynamic set and the non-high-dynamic set, setting the pixel values of the obtained regions to 1 and the pixel values of the remaining regions to 0, respectively generating a high-dynamic object mask image and a non-high-dynamic object mask image, and subtracting the overlapping region of the high-dynamic object mask image from the non-high-dynamic object mask image to generate a preliminary dynamic region mask image; The dynamic feature point marking module: extracts all ORB feature points of the preliminary dynamic region mask image, calculates the three-dimensional space reprojection error of each ORB feature point between the current frame and the historical key frame, and if the three-dimensional space reprojection error is greater than a preset error, marks the ORB feature point as a dynamic feature point; The dynamic point density calculation module: for each independent connected region composed of a bounding box in the preliminary dynamic region mask image, takes the ratio of the number of dynamic feature points to the number of ORB feature points in the region as the dynamic point density of the region; The adaptive decision module: if the dynamic point density of the region is not less than the dynamic point density threshold, a region generation method based on depth information fusion is adopted, after region growing on the depth map corresponding to the region with the identified dynamic feature points as seeds, a morphological dilation operation is performed to generate an irregular mask of the region; If the dynamic point density of the region is less than the dynamic point density threshold, the position of the rectangular bounding box corresponding to the region is kept unchanged; The target dynamic region mask image generation module: combines the irregular mask and the dynamic region corresponding to the rectangular bounding box as a target dynamic region mask image.

Images