A dynamic obstacle identification and elimination method based on SLAM research

Abstract

The application belongs to the technical field of dynamic obstacle identification and elimination. The application provides a dynamic obstacle identification and elimination method based on SLAM research. The embodiments of the disclosure greatly improve the accuracy of dynamic object detection and elimination by estimating the camera motion in real time and compensating the object prediction results. Especially in dynamic scenes, it is especially suitable for fields such as unmanned driving and robot navigation. The improved Mask R-CNN accurately segments dynamic objects, and the iterative Gaussian algorithm is used to accurately distinguish the background and dynamic objects, effectively improving the robustness of the system. Combined with semantic segmentation and depth map region growing technology, this method can accurately identify and eliminate dynamic objects, especially in complex interaction and occlusion cases, optimizing the calculation efficiency and real-time performance. According to the motion trajectory of the object, the future position can be predicted, the dynamic obstacles can be eliminated in advance, and the stability and accuracy of the static map construction are ensured.

Description

Technical Field

[0001] This disclosure relates to the field of dynamic obstacle recognition and removal technology, and in particular to a dynamic obstacle recognition and removal method based on SLAM research. Background Technology

[0002] With the rapid development of robotics technology in recent years, Simultaneous Localization and Mapping (SLAM) has become a core support for autonomous navigation and environmental perception in robots. This technology aims to enable robots to accurately locate their own position and accurately construct a map of the surrounding environment while moving in unknown environments.

[0003] Currently, SLAM-based dynamic obstacle recognition and removal methods mainly include technologies such as sensor detection, motion estimation, graph optimization, time series and data association, and semantic segmentation and target recognition.

[0004] Sensor-based detection methods include the combination of LiDAR and visual sensors. By comparing changes in LiDAR point clouds over a continuous time period, moving objects can be detected. Furthermore, deep learning methods, such as convolutional neural networks (CNNs), have also shown good performance in recognizing dynamic objects. The combination of LiDAR and visual sensors is a common technical approach, utilizing LiDAR point cloud data and image information to detect the presence of dynamic objects by comparing changes over a continuous time period.

[0005] In dynamic environments, dynamic obstacle occlusion is a key challenge when robots construct 3D maps. Dynamic obstacles not only affect map accuracy but can also lead to the misidentification of structures in the static environment or introduce map errors. This is because the presence and movement of dynamic objects can occlude some features of the static environment, preventing SLAM systems from accurately reconstructing the complete 3D environment. Although existing SLAM-based dynamic obstacle recognition and removal methods have made some progress in dynamic scenes, they still have some significant shortcomings in addressing the dynamic obstacle occlusion problem.

[0006] Specifically, these shortcomings are mainly reflected in the following aspects: 1) Due to occlusion by dynamic obstacles, some sensor data may be lost, preventing the robot from acquiring complete environmental information. This information loss affects the accuracy of dynamic obstacle recognition, and consequently, the accuracy of map construction.

[0007] 2) Data synchronization and fusion issues exacerbate the challenges posed by dynamic obstacle occlusion. When multi-sensor systems process data from different sensors, differences in time synchronization, sensing range, or update frequency may lead to errors in information fusion, further affecting the accurate identification and removal of dynamic obstacles.

[0008] 3) Inaccurate estimation of the motion trajectory of dynamic objects is also a prominent problem. Under occlusion conditions, traditional motion estimation algorithms (such as Kalman filtering, particle filtering, etc.) may fail to accurately track the motion of dynamic objects, especially under high-speed or irregular motion, causing the estimation deviation of the object's position, thus affecting the removal of dynamic obstacles and the accuracy of the 3D map.

[0009] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0010] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0011] The purpose of this disclosure is to provide a dynamic obstacle recognition and removal method based on SLAM research, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0012] According to a first aspect of the present disclosure, a method for dynamic obstacle recognition and removal based on SLAM research is provided, the method comprising: An improved Mask R-CNN network is constructed to obtain the original image and input the original image into the improved Mask R-CNN network; wherein, a mask culling branch is added to the FCN layer of the improved Mask R-CNN network; Based on IMU data and depth data, the camera's self-motion parameters are calculated, and the predicted trajectory of the object in the original image is reverse-compensated based on the camera's self-motion parameters. The original image is input into the improved Mask R-CNN network for feature extraction, and the mask is generated and updated by combining the iterative Gaussian algorithm. Dynamic obstacle culling is performed based on a mask to obtain an image of the dynamic obstacle culling effect.

[0013] Furthermore, the improved Mask R-CNN network includes: The backbone network ResNet50, FPN network, region projection network RPN, RoIAlign layer, and fully convolutional network FCN are sequentially connected. The fully convolutional network FCN includes a mask culling branch and a mask branch.

[0014] Furthermore, the loss function of the RoIAlign layer is:

[0015] in, For classifying losses, For bounding box, For mask branching loss, To mask out branch loss.

[0016] Furthermore, the step of calculating the camera's self-motion parameters based on IMU data and depth data, and then performing inverse compensation for the predicted trajectory of objects in the original image based on the camera's self-motion parameters, includes: The system acquires motion information from camera or sensor IMU data in real time and combines it with object state estimation from depth data to correct the object's trajectory; the motion information includes rotation matrices. Translation vector ; By continuously updating the motion information of the camera or sensor, the predicted position of the object is constantly adjusted, thereby eliminating errors caused by the movement of the device itself.

[0017] Furthermore, the steps of inputting the original image into the improved Mask R-CNN network for feature extraction, and combining iterative Gaussian algorithms for mask generation and updating, include: The original image is input into the backbone network ResNet50 and the FPN network for feature extraction to obtain feature maps; The feature map is input into the Region Projection Network (RPN) for anchor box determination to obtain candidate regions; The candidate regions and feature maps are input into the RoIAlign layer, and the candidate regions are mapped onto the feature maps for alignment using bilinear interpolation. The aligned feature map is input into the Mask branch of the fully convolutional network FCN to generate a binary mask; Dynamic background reconstruction is performed based on binary masks, and an iterative Gaussian algorithm is introduced for optimization to obtain masks for dynamic obstacles.

[0018] Further, the step of inputting the feature map into the Region Projection Network (RPN) for anchor box determination to obtain candidate regions includes: The feature map is input into the Region Projection Network (RPN) to generate pixel anchor boxes of different sizes and aspect ratios. The Softmax method is used to classify each pixel anchor box, and the foreground probability and background probability of each pixel anchor box are calculated. Bounding box regression is performed based on the foreground and background probabilities of each pixel anchor box to obtain the initial regression; The pixel anchor boxes are modified using regression prediction values. The foreground anchor box results are sorted according to the foreground probability and background probability. The non-maximum suppression algorithm is used to delete pixel anchor boxes that are below the threshold or have a high overlap rate to obtain candidate regions (RoIs). The candidate regions contain dynamic obstacles or static environmental features.

[0019] Furthermore, the step of inputting the candidate region and feature map into the RoIAlign layer and aligning them by mapping the candidate region onto the feature map using bilinear interpolation includes: The Region Projection Network (RPN) generates a series of target regions through an anchor frame mechanism and filters them based on foreground and background information. The input image containing the candidate region is processed by ResNet50 to extract features, resulting in a high-dimensional feature map; The RoIAlign layer transforms the coordinates of candidate regions in the original image to the feature map coordinate space and calculates the floating-point coordinates of the candidate regions; wherein, the candidate regions are divided into grid cells of fixed size; For each sampling point of the grid cell, the corresponding feature value is calculated from the feature map using bilinear interpolation. Each candidate region is mapped to a fixed-size feature map region, ensuring that candidate regions of different scales have consistent spatial alignment accuracy in subsequent classification and segmentation tasks.

[0020] Furthermore, the step of inputting the aligned feature map into the Mask branch of the fully convolutional network FCN to generate the binary mask includes: The feature map processed by the RoIAlign layer is input into the Mask branch network, and the depth features of the target region are extracted by the fully convolutional network FCN; among them, the feature map output by the RoIAlign layer is upsampled. Fully convolutional networks (FCNs) extract multi-scale features through convolutional layers of different depths, thereby ensuring that the spatial dimensions of the input and output remain consistent and avoiding the loss of spatial information due to the reduction in feature map size. Calculate the object probability for each pixel and generate a binary mask; the segmentation of the binary mask is a binary classification task, applying the sigmoid activation function to each pixel and mapping the output value to the range [0,1], representing the probability that the pixel belongs to that category of object.

[0021] Furthermore, the step of reconstructing the dynamic background based on the binary mask and optimizing iterative Gaussian algorithm to obtain the mask for the dynamic obstacle includes: Pixel-level binary masks for dynamic objects are generated using the Mask R-CNN segmentation branch; A fully convolutional network (FCN) is used to extract high-dimensional semantic information from the feature maps of candidate regions through multi-layer convolution. Combined with the classification branch of Mask R-CNN, dynamic objects are identified. Semantic segmentation is used to further verify the categories of dynamic objects in order to improve the accuracy of elimination; By performing pixel-level logical operations, a binary mask is applied to the original image to perform a region growing algorithm on the depth map, expanding the region of dynamic objects and ensuring the integrity of the culling process. The background is modeled by an iterative Gaussian algorithm, which distinguishes between dynamic objects and static backgrounds, making the background reconstruction more accurate. Fill the removed areas with background information from historical frames; Optical flow is used to predict pixel motion trends between consecutive frames in order to fill in occluded areas.

[0022] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the dynamic obstacle recognition and removal method based on SLAM research described above addresses the impact of camera or sensor self-motion on the prediction accuracy of dynamic objects through self-motion compensation. By estimating camera motion in real time and compensating for object prediction results, the accuracy of dynamic object detection and removal is greatly improved. This is particularly applicable to fields such as autonomous driving and robot navigation in dynamic scenes. Furthermore, the fusion of improved Mask R-CNN and iterative Gaussian algorithm provides a multi-algorithm collaborative solution. Improved Mask R-CNN accurately segments dynamic objects, while the iterative Gaussian algorithm accurately distinguishes between the background and dynamic objects, effectively improving the system's robustness. Combining semantic segmentation and depth map region growing techniques, this method can accurately identify and remove dynamic objects, especially in complex interactions and occlusion scenarios, optimizing computational efficiency and real-time performance. It can predict future positions based on object motion trajectories, enabling early removal of dynamic obstacles and ensuring the stability and accuracy of static map construction. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This illustration shows a dynamic obstacle identification and removal method based on SLAM research in an exemplary embodiment of this disclosure; Figure 2 This diagram shows a system corresponding to the dynamic obstacle recognition and removal method based on SLAM research in an exemplary embodiment of this disclosure. Figure 3 A schematic diagram illustrating the motion compensation principle in an exemplary embodiment of this disclosure is shown; Figure 4 This diagram illustrates the improved Mask R-CNN network architecture in an exemplary embodiment of this disclosure. Figure 5 This illustrates a dashed grid feature map in an exemplary embodiment of this disclosure; Figure 6 A flowchart illustrating the mask segmentation process in an exemplary embodiment of this disclosure is shown; Figure 7 This illustrates the specific process of the elimination algorithm in an exemplary embodiment of this disclosure; Figure 8 This illustrates the effect of dynamic feature removal in an exemplary embodiment of this disclosure. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0027] This example implementation provides a dynamic obstacle recognition and removal method based on SLAM research. (Reference) Figure 1 As shown, the dynamic obstacle recognition and removal method based on SLAM research may include steps S101 to S104.

[0028] Step S101: Construct an improved Mask R-CNN network, obtain the original image, and input the original image into the improved Mask R-CNN network; wherein, a mask culling branch is added to the FCN layer of the improved Mask R-CNN network; Step S102: Calculate the camera's self-motion parameters based on IMU data and depth data, and perform reverse compensation on the predicted trajectory of the object in the original image based on the camera's self-motion parameters. Step S103: Input the original image into the improved Mask R-CNN network for feature extraction, and combine iterative Gaussian algorithm to generate and update the mask; Step S104: Perform dynamic obstacle culling based on the mask to obtain the effect image of dynamic obstacle culling.

[0029] The dynamic obstacle recognition and removal method based on SLAM research described above addresses two key aspects. First, self-motion compensation mitigates the impact of camera or sensor motion on the prediction accuracy of dynamic objects. By estimating camera motion in real time and compensating for object predictions, the accuracy of dynamic object detection and removal is significantly improved. This is particularly applicable to autonomous driving and robot navigation in dynamic scenes. Second, the fusion of improved Mask R-CNN and iterative Gaussian algorithm provides a multi-algorithm collaborative solution. Improved Mask R-CNN accurately segments dynamic objects, while iterative Gaussian algorithm accurately distinguishes between background and dynamic objects, effectively improving the system's robustness. Combining semantic segmentation and depth map region growing techniques, this method can accurately identify and remove dynamic objects, especially in complex interactions and occlusion scenarios, optimizing computational efficiency and real-time performance. It can predict future positions based on object trajectories, enabling early removal of dynamic obstacles and ensuring the stability and accuracy of static map construction.

[0030] Below, we will refer to Figures 1 to 8 The steps of the dynamic obstacle recognition and removal method based on SLAM research described in this example implementation will be explained in more detail.

[0031] In a specific embodiment, to address the challenges posed by dynamic obstacle occlusion, insufficient accuracy and robustness of dynamic obstacle recognition, and feature matching issues in SLAM mapping during 3D map construction by robots in dynamic scenes, this research proposes an innovative dynamic obstacle recognition and removal scheme to enhance the performance of SLAM systems in complex dynamic environments, ensuring the accuracy, stability, and real-time performance of 3D maps. It improves the model's generalization ability, refines the detection and recognition of dynamic obstacles, and addresses the problem of insufficient feature matching in dynamic scenes during SLAM mapping. The main research objectives are as follows: 1. Solving the problem of dynamic obstacle occlusion: The primary objective of this application is to address the impact of dynamic obstacles in dynamic environments on robot 3D map construction, and to propose an effective method for dynamic obstacle extraction and behavior prediction. By improving data acquisition, fusion, and analysis methods, the interference of dynamic obstacle occlusion on static environmental features is reduced, thereby improving the accuracy of map construction and avoiding mislabeling or errors in static parts.

[0032] 2. Improving the Accuracy and Robustness of Dynamic Obstacle Recognition: An advanced motion compensation method is proposed to improve the efficiency and accuracy of dynamic obstacle detection. To enhance the robot's navigation and localization performance in dynamic environments, the dynamic obstacle recognition algorithm is optimized by adding a dynamic obstacle removal module to the target detection algorithm, thereby enhancing the system's accuracy and robustness. Through sensor data fusion, deep learning algorithms, and precise motion estimation, dynamic obstacles can be accurately identified and removed in real time in complex dynamic environments. Even when dynamic obstacles are occluded or moving, the recognition accuracy can be effectively maintained, reducing false positives and false negatives.

[0033] 3. Solution to Insufficient Feature Matching in SLAM Mapping: In dynamic scenes, a significant challenge for SLAM algorithms is insufficient feature matching, especially in the presence of dynamic obstacles. This paper proposes a feature matching optimization scheme based on dynamic obstacle recognition and removal. This scheme effectively addresses mismatches or omissions caused by dynamic objects during map building, improving the feature matching accuracy and map construction quality of the SLAM system.

[0034] In one specific embodiment, to address the dynamic interference and occlusion issues faced by robots when mapping in dynamic environments, a dynamic object module is added to the Mask R-CNN network. This method aims to improve the accuracy and robustness of robot mapping in dynamic scenes, especially for the detection and removal of dynamic obstacles. Traditional object detection algorithms are effective in object recognition, but they still have shortcomings in dynamic obstacle removal, especially when dealing with moving obstacles. The improved Mask R-CNN network not only performs accurate dynamic object detection but also incorporates a dynamic obstacle removal module (i.e., a mask removal branch), which can effectively remove dynamic obstacles that affect mapping. In this way, the robot can better cope with the occlusion problem of moving obstacles during real-time SLAM mapping, thereby ensuring the accuracy of map construction and the stability of the system.

[0035] Figure 2This paper presents the main framework of this application, which can be divided into two core parts: camera motion compensation and dynamic object detection and removal. For camera motion compensation, this application integrates IMU (Inertial Measurement Unit) and depth data to design an advanced motion compensation algorithm that effectively filters rotation and translation errors caused by camera self-motion, thereby improving the stability and accuracy of mapping. For handling dynamic obstacles, a Mask R-CNN network is used for dynamic object recognition and classification. In the object detection process, an iterative Gaussian fitting algorithm is introduced to more accurately extract the contours of dynamic obstacles. This process effectively avoids interference from dynamic obstacles in map construction, ensuring the real-time performance and accuracy of the SLAM system. Semantic segmentation technology is used to accurately segment and remove dynamic obstacles, further optimizing map quality and the robustness of the SLAM system. This significantly improves the performance of the SLAM system in complex dynamic environments, providing a more effective solution for real-time mapping tasks. The specific research steps are as follows: I. Extraction and Prediction of Dynamic Features Dynamic feature extraction and motion prediction are core components of dynamic object detection, especially in complex dynamic environments. Accurately identifying and predicting object trajectories is crucial for removing dynamic obstacles. This section combines an iterative Gaussian algorithm with a Mask R-CNN object detection network to effectively extract and predict the motion of dynamic objects, thereby improving the accuracy of dynamic object removal. Dynamic object extraction relies on modeling the scene background and detecting the foreground. By combining self-motion compensation, deep learning segmentation, an iterative Gaussian algorithm, and a motion prediction model, the system can accurately extract dynamic object features and predict the object's future position and trajectory, effectively reducing the interference of dynamic obstacles on static map construction. This is mainly described from the following two aspects: 1. Extraction of dynamic features The extraction of dynamic features relies first on the analysis of the object's motion patterns. In a complex dynamic environment, the motion of an object is usually affected by various factors, including the object's own motion patterns and the self-motion of the camera or sensor. To accurately extract the object's dynamic features, it is first necessary to eliminate errors caused by the camera or sensor's self-motion through self-motion compensation technology. By estimating the camera's rotation and translation in real time, the system can effectively adjust the predicted motion trajectory of the object, ensuring the accurate extraction of motion features.

[0036] Mask R-CNN plays a crucial role in the detection of dynamic objects. Through meticulous instance segmentation, Mask R-CNN generates a binary mask for each dynamic object, accurately identifying its position and shape. This mask allows for the extraction of dynamic features of the object in the image, including key parameters such as position, shape, and velocity. Furthermore, the Iterative Gaussian algorithm is applied to background modeling, helping the system distinguish between dynamic objects and static backgrounds, further improving the accuracy of dynamic feature extraction. The Iterative Gaussian algorithm is an application and extension of the Gaussian mixture model. It uses multiple Gaussian distributions to model the dynamic background and adapts to the dynamic environment by iteratively updating the parameters of the Gaussian distributions. Specifically, the Iterative Gaussian algorithm extracts dynamic objects through the following steps: 1) Initialization phase: Assign multiple Gaussian distributions to each pixel. Typically, the Gaussian distribution parameters of each pixel are initialized by learning the pixel values ​​in historical image frames, including mean, variance, and weight.

[0037] 2) Update Phase: Whenever a new image frame arrives, the algorithm calculates the degree of matching between the observed value of each pixel in the frame and its corresponding Gaussian distribution. If the current pixel value matches the mean of a Gaussian distribution well (i.e., the pixel value is close to the mean), the Gaussian distribution updates its mean and variance parameters; if the matching degree is low, a new Gaussian distribution is generated, and the observed value of the pixel is classified as foreground.

[0038] 3) Background and Foreground Separation: By setting a threshold, each pixel is determined to belong to either the background or the foreground. When the value of a pixel matches well with multiple Gaussian components of the background, the pixel is considered to belong to the background; if the pixel value does not match well with the Gaussian distribution in the background model, the pixel is considered to belong to a dynamic object (foreground).

[0039] 4) Dynamic Object Extraction: When the pixels of an object continuously change across multiple frames, the Gaussian Mixture Model (GMM) can detect these changes and classify them as dynamic objects. During this process, the iterative Gaussian algorithm continuously optimizes the background modeling, enabling the system to adapt to changes in lighting, season, or environment, thus effectively detecting real dynamic objects.

[0040] In a dynamic environment, the observed value of each pixel It can be represented by multiple Gaussian distributions, specifically the observation value of each pixel. It is composed of a weighted sum of multiple Gaussian components: (1) in, yes Pixels observed at any time The probability, It is the first The weights of a Gaussian distribution and The first The mean and variance of a Gaussian distribution. It represents the number of Gaussian components. This allows the system to adapt to different background information for each pixel based on changes in the scene, improving the accuracy of recognizing dynamic objects.

[0041] Combining the high-precision segmentation results of Mask R-CNN with the iterative Gaussian algorithm enables simultaneous accurate segmentation of dynamic objects and accurate background modeling. Mask R-CNN provides the precise contours of each dynamic object, while the iterative Gaussian algorithm is responsible for recognizing and adapting to background changes, further improving the accuracy of dynamic object extraction. This combination allows the system to accurately extract dynamic objects while maintaining the stability and consistency of the background, thereby optimizing the dynamic object removal process and ensuring the accuracy of the static map.

[0042] 2. Prediction of motor behavior Action prediction for dynamic objects plays a crucial role in preventing them from affecting static map construction, and accurate action prediction is the core of this process. Extraction of moving objects is an important preliminary step in action prediction. Generally, the contours of moving objects can be extracted completely and accurately. However, when the scale of the moving object on the image plane is too small, the region of interest (ROI) may not converge. To address this special case, a method of finding connected components is used to identify the region most likely to be the moving target within the unfused ROI. This operation, in most cases, ensures accurate extraction of the moving target.

[0043] Previous algorithms for self-motion compensation have suffered from high computational requirements or insufficient accuracy. To address this challenge, the proposed algorithm performs self-motion compensation by fusing depth and IMU data, achieving a balance between computational efficiency, accuracy, and robustness. The basic principle of self-motion compensation is to estimate the motion (such as rotation and translation) of the camera or sensor and then adjust the predicted results for the object accordingly. This includes rotation and translation, where rotation is achieved using a rotation matrix. It indicates that translation is done using a translation vector. The initial position of the object in the camera coordinate system is... The new position after the camera moves It can be derived from the formula The rotation matrix here is calculated. It is a satisfaction ( It is a 3×3 matrix (with the identity matrix as an example). During the prediction process, the system acquires motion information from the camera or sensor in real time and combines it with the object's state estimation to correct the object's trajectory. This is achieved by continuously updating the motion parameters of the camera or sensor, i.e., the rotation matrix. Translation vector It continuously adjusts the predicted position of the object, thereby eliminating errors caused by the movement of the device itself, such as... Figure 3 As shown. In The algorithm simultaneously considers the rotation and translation of the event camera. Furthermore, during the compensation process, static backgrounds and moving objects can be distinguished based on temporal distribution. Static backgrounds (blue) projected onto the same pixel have a relatively low average timestamp, while moving objects (red) cannot be compensated and have a wide timestamp distribution. Utilizing this characteristic, moving objects can be compensated and predicted more accurately, thus achieving accurate prediction of object trajectories and effectively addressing the shortcomings of previous algorithms.

[0044] II. Improved Mask R-CNN Dynamic Object Detection Dynamic object detection is achieved by using Mask R-CNN for instance segmentation of targets, combined with motion compensation and feature removal of dynamic obstacles. By integrating with IMU sensors or other motion estimation algorithms, dynamic objects are identified. These dynamic objects are then represented in the generated mask. The mask branch is a small fully convolutional network (FCN) that predicts a segmentation mask for each RoI at the pixel level. Considering the Faster R-CNN framework, Mask R-CNN is relatively simple to implement and train, and can adapt to various flexible architecture designs. Furthermore, the mask branch adds only a small amount of computational overhead, allowing the system to run quickly and conduct efficient experiments. Although Mask R-CNN is essentially an extension of Faster R-CNN, the construction of the mask branch is crucial for obtaining high-quality segmentation results. An improvement to Mask R-CNN adds a new mask removal branch. This improvement enables the effective removal of dynamic obstacle masks after generating binary masks, supporting the final frame image reconstruction and thus building a more accurate static 3D map. The improved network architecture is as follows. Figure 4 As shown, this network architecture primarily uses the ResNet50 backbone network of Mask R-CNN. It constructs hierarchical residual class connections to obtain finer-grained features. The FPN is integrated with the backbone network, which integrates three channels and extracts feature information of various sizes.

[0045] In object detection and segmentation tasks, the recognition and extraction of image features are crucial, directly impacting the accuracy of detection and segmentation. To extract richer and more expressive features, ResNet50 introduces hierarchical residual connections to obtain finer-grained features, thereby enhancing the network's ability to capture information at different scales. To further improve the processing capability of multi-scale features, the Feature Pyramid Network (FPN) is integrated with the backbone network. The backbone network extracts multi-scale feature information through multiple channels to better handle targets of different sizes. The RPN first generates anchor boxes of different sizes and aspect ratios. The softmax method is used to determine the anchor boxes. After feature extraction, the generated feature maps are input into the Region Proposal Network (RPN). The task of the RPN is to perform object detection by generating anchor boxes with different sizes and aspect ratios. The size and aspect ratio of each anchor box can be expressed as: (2) in Indicates the center coordinates of the anchor frame. and Representing the width and height of the anchor box, RPN uses the Softmax method to classify each anchor box, calculating the probability that each pixel anchor box belongs to the foreground or background, expressed as: (3) in, and These represent the scores for the foreground and background portions of the anchor box, respectively. Through this step, the RPN classifies the anchor boxes, providing input for subsequent bounding box regression. Bounding box regression is then performed using the RPN (Region Proposal Network) to adjust the positions of the generated anchor boxes. The regression objective for each anchor box is to precisely adjust it using predicted offsets (e.g., changes in position, width, and height). The regression objective for each anchor box can be represented as... This allows it to better cover the boundaries of actual objects. Regression error is smoothed. The loss function is calculated to effectively measure the difference between the predicted and ground truth boxes and helps optimize the regression process. Through regression, RPN can precisely adjust the position of anchor boxes, further improving the accuracy of object detection. Initial regression is obtained by performing bounding box regression, then the anchor boxes are modified using the regression predictions. The foreground anchor box results are sorted according to scores, and non-maximum suppression (NPMS) is used to remove boxes below a threshold or with high overlap.

[0046] After generating multiple anchor boxes, the RPN uses Non-Maximum Suppression (NMS) to remove redundant anchor boxes. First, the RPN sorts each anchor box according to its score and calculates the Intersection over Union (IoU) between them. If the IoU exceeds a set threshold, the lower-scoring anchor box is deleted. The core idea of ​​NMS is to retain anchor boxes with higher scores and less overlap with other boxes, reducing interference from redundant boxes and ensuring that the final output anchor boxes represent the optimal foreground regions.

[0047] Through the above steps, RPN generates high-quality target region proposals, providing accurate input for subsequent tasks such as target classification, segmentation, and dynamic obstacle removal. Finally, a feature map is obtained. The feature map and the remaining anchor boxes from RPN are sent to RoIAlign. RoIAlign is a key module in Mask R-CNN. By removing the quantization step and using bilinear interpolation, it accurately maps the features in each RoI onto the feature map, thus solving the accuracy loss problem caused by quantization operations in traditional convolutional neural networks and ensuring pixel-level alignment. Figure 5 The dashed grid represents the feature map processed by the Region Proposal Network (RPN), the solid lines represent RoIs (with 2×2 bins in this example), and the dots represent the 4 sampling points in each bin. RoIAlign computes the value of each sampling point from the grid points near the feature map using bilinear interpolation. No quantization is performed on any coordinates involved in the RoI, its bins, or the sampling points. RoIAlign maps each RoI to the original feature map and computes the feature value of each pixel within that RoI region. Unlike traditional pooling operations, RoIAlign does not map the input features to a fixed grid; instead, it uses bilinear interpolation to more accurately capture the features at each location. RoIAlign eliminates the quantization process and uses bilinear interpolation to map the remaining anchor boxes from the RPN to the feature map, effectively avoiding pixel mismatch issues and performing pixel-level instance segmentation.

[0048] III. RoIAlign Layer Mask Update Traditional instance segmentation tasks are highly challenging due to their high accuracy requirements. Instance segmentation not only needs to accurately detect all objects in an image, but also requires precise segmentation of each object instance, making the task even more complex. Instance segmentation combines classic computer vision techniques from object detection, with the core objective of classifying each object and locating it using bounding boxes and semantic information. This process is particularly important for handling dynamic objects, as the shape, position, and occlusion of dynamic obstacles can change over time.

[0049] To address this issue, Mask R-CNN, an extension of Faster R-CNN, introduces a new branch to predict segmentation masks for each region of interest (RoI). This masking branch works in parallel with the original classification and bounding box regression branches, enabling the model to perform pixel-level object segmentation while detecting objects, thus achieving more accurate object recognition and differentiation. This extension plays a crucial role in dynamic obstacle detection, effectively classifying dynamic objects and providing a better foundation for subsequent semantic segmentation. In this way, Mask R-CNN not only enhances object detection capabilities but also provides greater robustness and accuracy in handling dynamic obstacles and complex scenes.

[0050] In the improved Mask R-CNN, a multi-task loss is defined for each sampled RoI during training to ensure the model can effectively perform classification, bounding box regression, mask prediction, and dynamic obstacle removal. Specifically, the loss function can be expressed as: (4) Classification loss and bounding box It is determined to use the same cross-entropy loss and smoothing as the original Mask R-CNN. Loss, mask branch Optimization is achieved through binary cross-entropy to ensure that the mask for each RoI accurately represents the shape of the object. Dynamic obstacle culling loss is also employed. This is a new module designed to prevent dynamic obstacles from interfering with map building and pose estimation by identifying and culling the influence areas of dynamic objects within each mask. The mask branch of each RoI outputs a... The two-dimensional output, in which It is the number of target categories. It's the resolution of the mask; one mask corresponds to each category. A binary mask is used to represent the location and shape of the category in the image. For each pixel, the mask value is predicted using a sigmoid function, with the output value between 0 and 1, representing the probability that the pixel belongs to that object. This method provides pixel-level spatial information for each object, enabling precise segmentation of object boundaries.

[0051] By working in conjunction with the mask branch, dynamic obstacles can be accurately identified and effectively eliminated, ensuring that the robot can maintain high-precision mapping and localization in dynamic scenes. The feature maps and filtered anchor boxes processed by the Region Proposal Network (RPN) are then passed to the RoIAlign module.

[0052] This approach ensures precise alignment of feature maps for each RoI, avoiding pixel mismatch issues caused by quantization errors, thereby improving the accuracy and effectiveness of instance segmentation. Based on the precisely aligned feature maps output by RoIAlign, the network further passes the feature maps to two branches. The first branch is a mask branch network, which uses a fully convolutional network (FCN) to predict and generate segmentation masks for objects. By generating binary masks, this branch can accurately identify the shape of each object and remove masks of dynamic obstacles from the image, preventing them from interfering with the final mapping results.

[0053] IV. Mask Generation and Segmentation Removal After generating the binary mask, a series of candidate regions (RoIs) are generated by improving the RPN of Mask R-CNN. These regions typically contain the potential locations of the target object. Subsequently, the RoIAlign module precisely maps these candidate regions onto the feature map, avoiding errors introduced by traditional quantization operations. The goal of RoIAlign is to accurately align the feature maps of the RoI regions. It uses bilinear interpolation for pixel-level alignment, ensuring that fine-grained features of the target region in the feature map are preserved.

[0054] Following RoIAlign, each RoI in the image is passed to the Mask branch network for mask generation. This masking network generates binary masks based on a fully convolutional network (FCN). Each mask represents the position and shape of a target object in the image and is predicted using a pixel-level sigmoid activation function. Specifically, the mask is... For each pixel In this context, the mask value represents the probability that a pixel belongs to the target object. The mask prediction formula is as follows: (5) in, express The probability that it belongs to the target object. It is the sigmoid activation function. It is network weight. It is extracted from the feature map along with pixels. Related features It is a paranoid trait, and the generation process is as follows: Figure 6 As shown. Figure 6 The main demonstration shows two processes: outputting the original image to the network for feature extraction and dynamic object detection, and then combining iterative Gaussian algorithm to generate and update the mask.

[0055] In dynamic background modeling, an iterative Gaussian algorithm is introduced to optimize background reconstruction. It is used to model the background of a scene and effectively distinguishes between static backgrounds and dynamic objects in dynamic environments. The Gaussian mixture model, by continuously updating the pixel distribution parameters, can adapt to scene changes over time and accurately distinguish the foreground (dynamic objects) from the background. The segmentation branch of Mask R-CNN, combined with the iterative Gaussian algorithm, can accurately identify the contours of dynamic objects, thereby generating pixel-level masks to separate these dynamic objects from the background. This fine-grained segmentation mask provides a foundation for subsequent background inpainting, facilitating the accurate restoration of occluded background information after removing dynamic objects.

[0056] The dynamic background reconstruction in this study utilizes the segmentation results of Mask R-CNN for feature extraction and filtering, combined with semantic segmentation and multi-view geometric information to ensure the integrity and realism of the background after dynamic object removal. This method allows for better removal of moving objects. While segmentation networks can effectively remove prior dynamic objects, their ability to distinguish non-prior dynamic objects in the scene is limited; therefore, a multi-view geometric algorithm is used for further processing. The proposed dynamic object removal algorithm flow is as follows: Figure 7 As shown, dynamic points are detected based on the motion relationships between keyframes. After detection, dynamic points are categorized into those with semantic information and those without semantic information by determining whether they possess semantic labels. Semantic contour search is performed on the semantic map for feature points with semantic information, while region growing is performed on the depth map for dynamic points without semantic information. This fully utilizes semantic information to reduce the number of seed points for region growing, improving system efficiency. Finally, the dynamic object masks without semantic information and those with semantic information are fused to obtain a complete dynamic object mask.

[0057] The next step is to remove dynamic obstacles based on the mask. To remove dynamic obstacles, the classification branch of Mask R-CNN is first used to determine which objects are dynamic. The masks of these dynamic objects are identified and labeled. Then, using logical operations (such as binarization and multiplication), the pixels in the regions of the dynamic objects are set to zero, thus removing them from the image. Specifically, for the generated mask... and images The elimination process is as follows: (6) This is a binary mask for dynamic objects, where a value of 1 represents the region of the dynamic object. The image is then used to create this mask. and By performing pixel-by-pixel multiplication, the region of the moving object will be removed from the moving object, resulting in the final image. RGB and depth images are acquired using an RGB-D camera. The RGB images are processed by a semantic segmentation network to obtain pixel-level semantic information. This semantic information is used to remove feature points on prior dynamic objects in the image. A multi-view geometric algorithm is then used to further detect feature points corresponding to non-prior dynamic objects. The detection results from the multi-view geometry algorithm and the semantic segmentation network are cross-validated to obtain the complete dynamic region. After filtering out feature points in the dynamic region, the image is entered into the tracking thread for a more accurate pose. The dynamic feature removal effect is shown below. Figure 8 As shown.

[0058] The dynamic obstacle recognition and removal method based on SLAM research described above addresses two key aspects. First, self-motion compensation mitigates the impact of camera or sensor motion on the prediction accuracy of dynamic objects. By estimating camera motion in real time and compensating for object predictions, the accuracy of dynamic object detection and removal is significantly improved. This is particularly applicable to autonomous driving and robot navigation in dynamic scenes. Second, the fusion of improved Mask R-CNN and iterative Gaussian algorithm provides a multi-algorithm collaborative solution. Improved Mask R-CNN accurately segments dynamic objects, while iterative Gaussian algorithm accurately distinguishes between background and dynamic objects, effectively improving the system's robustness. Combining semantic segmentation and depth map region growing techniques, this method can accurately identify and remove dynamic objects, especially in complex interactions and occlusion scenarios, optimizing computational efficiency and real-time performance. It can predict future positions based on object trajectories, enabling early removal of dynamic obstacles and ensuring the stability and accuracy of static map construction.

[0059] In summary, this method has strong patentability and application prospects in dynamic object detection, background restoration, and real-time performance, and is especially suitable for complex scenes in highly dynamic environments.

[0060] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0061] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A dynamic obstacle recognition and removal method based on SLAM research, characterized in that, The method includes: An improved Mask R-CNN network is constructed to acquire the original image and input it into the improved Mask R-CNN network. The network consists of a backbone network ResNet50, an FPN network, a region projection network RPN, a RoIAlign layer, and a fully convolutional network FCN in sequence. The FCN includes a mask culling branch and a Mask branch, and the improved Mask R-CNN network adds a mask culling branch to the FCN. Based on IMU data and depth data, the camera's self-motion parameters are calculated, and the predicted trajectory of the object in the original image is reverse-compensated based on the camera's self-motion parameters. The original image is input into an improved Mask R-CNN network for feature extraction, and an iterative Gaussian algorithm is used to generate and update the mask. Specifically, this involves: inputting the original image into the backbone network ResNet50 and the FPN network for feature extraction to obtain feature maps; inputting the feature maps into the Region Projection Network (RPN) for anchor box determination to obtain candidate regions; inputting the candidate regions and feature maps into the RoIAlign layer, and using bilinear interpolation to map the candidate regions onto the feature maps for alignment; inputting the aligned feature maps into the Mask branch of the Fully Convolutional Network (FCN) to generate binary masks; using the Mask R-CNN segmentation branch to generate pixel-level binary masks for dynamic objects; and using the FCN to perform multi-layer convolution on the feature maps of the candidate regions to extract high-dimensional semantic information, combined with the Mask... The R-CNN classification branch identifies dynamic objects; semantic segmentation is used to further verify the dynamic object category to improve the accuracy of removal; through pixel-level logical operations, a binary mask is applied to the original image to perform a region growing algorithm on the depth map to expand the dynamic object region and ensure the integrity of removal; the background is modeled using an iterative Gaussian algorithm to distinguish between dynamic objects and static background, making background reconstruction more accurate; the removed region is filled with background information from historical frames; and optical flow is used to predict the pixel motion trend of previous and next frames to fill in the occluded region. Dynamic obstacle culling is performed based on a mask to obtain an image of the dynamic obstacle culling effect.

2. The dynamic obstacle recognition and removal method based on SLAM research according to claim 1, characterized in that, The loss function for the RoIAlign layer is: in, For classifying losses, For bounding box, For mask branching loss, To mask out branch loss.

3. The dynamic obstacle recognition and removal method based on SLAM research according to claim 2, characterized in that, The steps of calculating the camera's self-motion parameters based on IMU data and depth data, and then performing inverse compensation for the predicted trajectory of objects in the original image based on these parameters, include: The system acquires motion information from camera or sensor IMU data in real time and combines it with object state estimation from depth data to correct the object's trajectory; the motion information includes rotation matrices. Translation vector ; By continuously updating the motion information of the camera or sensor, the predicted position of the object is constantly adjusted, thereby eliminating errors caused by the movement of the device itself.

4. The dynamic obstacle recognition and removal method based on SLAM research according to claim 3, characterized in that, The step of inputting the feature map into the Region Projection Network (RPN) for anchor box determination to obtain candidate regions includes: The feature map is input into the Region Projection Network (RPN) to generate pixel anchor boxes of different sizes and aspect ratios. The Softmax method is used to classify each pixel anchor box, and the foreground probability and background probability of each pixel anchor box are calculated. Bounding box regression is performed based on the foreground and background probabilities of each pixel anchor box to obtain the initial regression; The pixel anchor boxes are modified using regression prediction values. The foreground anchor box results are sorted according to the foreground probability and background probability. The non-maximum suppression algorithm is used to delete pixel anchor boxes that are below the threshold or have a high overlap rate to obtain candidate regions (RoIs). The candidate regions contain dynamic obstacles or static environmental features.

5. The dynamic obstacle recognition and removal method based on SLAM research according to claim 4, characterized in that, The step of inputting candidate regions and feature maps into the RoIAlign layer and aligning them by mapping the candidate regions onto the feature maps using bilinear interpolation includes: The Region Projection Network (RPN) generates a series of target regions through an anchor frame mechanism and filters them based on foreground and background information. The input image containing the candidate region is processed by ResNet50 to extract features, resulting in a high-dimensional feature map; The RoIAlign layer transforms the coordinates of candidate regions in the original image to the feature map coordinate space and calculates the floating-point coordinates of the candidate regions; wherein, the candidate regions are divided into grid cells of fixed size; For each sampling point of the grid cell, the corresponding feature value is calculated from the feature map using bilinear interpolation. Each candidate region is mapped to a fixed-size feature map region, ensuring that candidate regions of different scales have consistent spatial alignment accuracy in subsequent classification and segmentation tasks.

6. The dynamic obstacle recognition and removal method based on SLAM research according to claim 5, characterized in that, The step of inputting the aligned feature map into the Mask branch of the fully convolutional network FCN to generate a binary mask includes: The feature map processed by the RoIAlign layer is input into the Mask branch network, and the depth features of the target region are extracted by the fully convolutional network FCN; among them, the feature map output by the RoIAlign layer is upsampled. Fully convolutional networks (FCNs) extract multi-scale features through convolutional layers of different depths, thereby ensuring that the spatial dimensions of the input and output remain consistent and avoiding the loss of spatial information due to the reduction in feature map size. Calculate the object probability for each pixel and generate a binary mask; the segmentation of the binary mask is a binary classification task, applying the sigmoid activation function to each pixel and mapping the output value to the range [0,1], representing the probability that the pixel belongs to that category of object.

Images