A SLAM method integrating deep neural network to remove purely dynamic feature points

Abstract

The present invention discloses a SLAM method that integrates a deep neural network to remove purely dynamic feature points. The method includes the following steps: Step 1: Using an RGB-D camera to collect RGB images and depth images and input them into the system; Step 2: Calculating the three-dimensional spatial coordinates corresponding to each pixel point through the principle of triangulation; Step 3: Using two different strategies to eliminate highly dynamic feature points; Step 4: Reprojecting the three-dimensional feature points of the previous frame to the current frame using the initial camera posture and calculating the reprojection offset vector; Step 5: Dividing the 3D feature points of the current frame into k clusters using the SOM-K-means algorithm; Step 6: For each cluster, calculating and determining the type of each cluster, and removing all feature points in the dynamic cluster based on the determination result. Step 7: Embedding the remaining static feature points into the SLAM system for tracking and mapping. The present invention relates to the field of synchronous positioning and mapping in dynamic environments, and specifically to a SLAM method that integrates a deep neural network to remove purely dynamic feature points.

Description

Technical Field

[0001] The present invention relates to the field of synchronous positioning and mapping in dynamic environments, and specifically to a SLAM method that integrates a deep neural network to remove purely dynamic feature points. Background Art

[0002] In recent years, with the development of neural networks, many deep learning networks have been used in the field of computer vision. Deep learning-based computer vision encompasses many different research areas, including but not limited to object detection, semantic segmentation, motion and tracking, 3D reconstruction, and action recognition. In traditional SLAM systems, pixel information such as keypoints, descriptors, and bag-of-words models are used to describe images. However, deep neural networks can provide semantic information that is more similar to the characteristics of the human visual system. Therefore, it is particularly important to study the integration of deep neural networks with visual SLAM to meet the real-time requirements of the system, minimize the influence of dynamic points, and significantly improve positioning accuracy and system robustness.

[0003] Many excellent SLAM systems have emerged, but most of them make a basic assumption that the environment is static. All of these SLAM systems have satisfactory performance in static environments. However, the assumption of a static environment is too harsh. There are a large number of dynamic objects in the real world, and feature points located on dynamic objects may cause incorrect matches, which can greatly interfere with the camera's position during tracking and cause errors in camera motion estimation. Although these traditional SLAM systems use random sampling consistency and robust cost functions to reduce the negative impact of dynamic points, these algorithms may have little or no effect when there are a large number of dynamic objects, especially when they account for a large proportion of the scene. Summary of the Invention

[0004] In view of the above situation, in order to make up for the above existing defects, this solution provides a SLAM method that integrates deep neural networks to remove pure dynamic feature points.

[0005] The technical solution adopted by the present invention is as follows: a SLAM method that integrates a deep neural network to remove purely dynamic feature points, comprising at least the following methods:

[0006] Step 1: Use an RGB-D camera to capture RGB and depth images and input them into the system. Then, use YOLOv8 to process each input image, output the coordinates and category information of each bounding box, and publish it to the SLAM through ROS2. SLAM classifies all object types detected by YOLOv8 into two categories based on the object's potential motion state: (1) static objects: such as tables, consoles, and walls; (2) dynamic objects: such as people and pets. Then, based on the coordinates and category information of the bounding box, the feature points are classified for the next step.

[0007] Step 2: Use the depth image captured by the RGB-D camera and the camera calibration parameters to calculate the 3D spatial coordinates corresponding to each pixel through the principle of triangulation. Triangulation utilizes the relationship between the distance from the camera to the object, the camera intrinsic parameters, and the pixel coordinates for calculation.

[0008] Step 3: Two different strategies are used to remove highly dynamic feature points. The remaining static feature points are then frame-matched using the RAFT method of the aggregated GMA, and then self-motion estimation is performed in conjunction with OpenCV functions to restore the initial camera pose.

[0009] Step 4: Reproject the 3D feature points of the previous frame to the current frame through the initial camera pose and calculate the reprojection offset vector, where the reprojection offset vector is used to describe the displacement of the spatial point.

[0010] Step 5: Divide the 3D feature points of the current frame into k clusters using the SOM-K-means algorithm.

[0011] Step 6: For each cluster, calculate and determine the type of each cluster, and remove all feature points in the dynamic cluster according to the determination result.

[0012] Step 7: Embed the remaining static feature points into the SLAM system for tracking and mapping.

[0013] Among them, in step 1: use YOLOv8 to process each image captured by the RGB-D camera, identify the objects in the image and generate bounding boxes and their positions and classification information, so as to pass them into the SLAM system for binary classification of the identified objects and extraction of dynamic and static feature points.

[0014] In step 2, the depth image captured by the RGB-D camera is combined with semantic information to calculate the coordinates of each pixel in the three-dimensional space using the camera's intrinsic parameters (such as focal length and principal point) and the depth value corresponding to each pixel in the depth image. The specific principles are as follows:

[0015]

[0016]

[0017] in, is the camera's focal length, which describes the camera's field of view, pixel size, and other information. (u, v) represents the coordinates of a pixel, and (x, y, z) represents the three-dimensional coordinates. A depth image is a two-dimensional image, and the grayscale value of each pixel represents the depth information at that point. By reading the depth value of each pixel in the depth image, we can determine the distance from that pixel to the camera.

[0018] Among them, in step three: Since all existing dynamic SLAM systems that use target detection directly remove all feature points in the dynamic bounding box. In fact, the bounding box occupies more area in the RGB image than the object. In particular, when the dynamic bounding box occupies a large proportion of the image area, if all feature points in the dynamic bounding box are directly removed, the remaining static feature points may not generate satisfactory point matches. In order to improve the robustness and speed of SLAM, two different strategies are used to prevent extreme cases: (1) When the number of correct point matches does not exceed 8, we retain the feature points in the dynamic bounding box but not in the static bounding box for further testing. (2) When the number of correct point matches is greater than 8, the feature points in the dynamic bounding box are directly removed. In addition, feature points that are too close to the image edge and the bounding box are also discarded. Then, the remaining feature points in the current frame are matched with the previous frame using the RAFT method of aggregated GMA. The potential static matching point set in the static area is identified by the RAFT method of aggregated GMA and semantic prior information. For the global aggregation module GMA, the query and key features are projections of the context feature map, which are used to model the appearance self-similarity in the first frame. The value feature is a projection of the motion feature, which itself encodes a four-dimensional correlation volume. The attention matrix calculated from the query and key features is used to aggregate the value features as a hidden representation of the motion. The following equation is the calculation of the global aggregate feature:

[0019]

[0020] in, It is the movement characteristic of RAFT. is a learnable parameter. , , are encoding functions for query, key, and value respectively. F is the similarity calculation function, as shown in the following formula:

[0021]

[0022]

[0023]

[0024]

[0025] The calculation of the above global aggregation features plus the position encoding information is shown in the following formula:

[0026]

[0027] in, Indicates positional encoding. and Represent the potential static matching point sets of the previous frame and the current frame respectively. Then, the most accurate matching static feature point pairs are selected using epipolar constraints and RANSAC. The epipolar constraint model can be expressed as follows:

[0028]

[0029] Where F is the fundamental matrix, and yes and Potential static feature points in .

[0030] For each , we calculate and the corresponding epipolar line The distance between them can be calculated using the following equation:

[0031]

[0032] A threshold of 0.1 is set. If D is greater than the threshold, the corresponding feature point pair is removed. E is the essential matrix, which can be calculated using the following equation:

[0033]

[0034] Where K is the camera intrinsic matrix. The feature point pair on the left and E are fed into the OpenCV function.

[0035] Among them, in step 4: Assume is a static point, is a dynamic point, { } represents the feature point in the image coordinate system at time t, { } represents the matching feature points corresponding to time t-1. Then, at time t-1, the previous frame image and reprojection { } to the camera coordinate system { }. Similarly, we can get the depth information of the camera coordinate system at time t. }. Therefore, there will be two offset vectors: , l represents the modulus of the offset vector, which can be calculated using the following equation:

[0036]

[0037] in, The angle of the offset vector can be calculated using the following equation:

[0038]

[0039] Ideally, and should overlap because is a static point, } is the corresponding feature pair. However, due to the existence of random errors, there is an offset vector. Based on their true movement in spatial coordinates, a simple and easy-to-understand method is proposed to distinguish dynamic points from static points, as shown in the following formula:

[0040]

[0041]

[0042]

[0043] in, Represents the back-projection function, which depends on the camera type, but at this time At time t-1, it is in the camera coordinate system. Projection to At time t, the initial camera pose T(R,t) is given by the equation Get the offset vector can be Calculated. When a random error occurs, if the depth d is large, the modulus of the offset vector is also large. On the contrary, the angle of the offset vector is small. A weighted average method is used to describe the offset vector: in the current frame, the set of feature points outside the dynamic bounding box is The set of feature points in the dynamic bounding box is Using the above equation, we get The corresponding offset vector is then calculated using the following equation :

[0044]

[0045] in, It is expressed in radians. Multiplying it by 0.7 is numerically similar to There may be a small number of outliers due to incorrect matching or depth information. We remove these outliers and calculate the remaining feature points The average value of , denoted as φ:

[0046]

[0047] Where: is the number of remaining inliers.

[0048] Among them, in step 5: using the SOM-K-means clustering algorithm, which is the optimization algorithm of SOM, not only solves the problem of low convergence efficiency and inaccurate classification results of the SOM neural network, but also overcomes the problem of presetting the K-means clustering center and K value. The specific steps of the algorithm are as follows: first, input the feature points on the dynamic bounding box into the SOM neural network for clustering, output the number of classifications and the preliminary classification results, and use the obtained initial classification results as the k value. Then randomly set k 3D points (j=1,2,3,…,k) as the center of these 3D feature points and calculate and The distance is as follows:

[0049]

[0050] in, are the 3D feature points that need to be classified, The number of is n, The number of is k.

[0051] For each Find the closest 3D point to it As its center. Therefore, each 3D feature point will belong to a cluster (center). Then update the new center of each cluster and iterate continuously until the cluster of each 3D feature point remains unchanged. This process can be summarized as the goal of the K-means algorithm is to minimize the loss function J:

[0052]

[0053] in, is a binary variable, , 1 represents a three-dimensional feature point belongs to the jth cluster, and 0 means it does not belong to the jth cluster.

[0054] Among them, in step six: , the corresponding three-dimensional feature point set is We use the K-means method to is divided into k clusters. Accordingly, is also divided into k clusters. For each cluster, we calculate the mean of Th. Then it is used to determine the type of each cluster. The mean of Th in K clusters is recorded as

[0055]

[0056] Where: w is the number of feature points in the kth cluster.

[0057] For each cluster, we calculated and with ( The type of the kth cluster can be defined as follows:

[0058]

[0059] If the type is determined to be dynamic, all feature points in the cluster are determined to be purely dynamic; otherwise, the cluster is static. Feature points determined to be dynamic are then removed.

[0060] In step seven: the remaining static points are embedded into the SLAM system for tracking and mapping. After the above steps, the static feature points can be retained to the greatest extent to improve the accuracy of mapping.

[0061] The beneficial effects achieved by the present invention using the above structure are as follows:

[0062] This solves the problem that traditional SLAM systems, which use random sampling consistency and robust cost functions to reduce the negative impact of dynamic points, may have little or no effect when there are a large number of dynamic objects, especially when they account for a large proportion of the scene. This approach selectively retains static feature points within the dynamic bounding box by combining semantic information, removing only purely dynamic feature points as much as possible. This provides richer information for subsequent tracking and mapping, significantly improving positioning accuracy and system robustness.

[0063] Compared to traditional SLAM methods, this method uses the YOLOv8 algorithm, which maintains high accuracy while achieving real-time detection speed. Its deeper network architecture provides stronger expressiveness and higher detection accuracy. Compared to traditional object detection algorithms, YOLOv8 is able to maintain high detection accuracy while reducing computational complexity. Furthermore, the algorithm can handle objects of various sizes and categories, demonstrating strong versatility and adaptability. RAFT, which aggregates the GMA module, performs global motion estimation for each frame in a video sequence during frame matching, rather than just local motion. This helps obtain more accurate motion information, especially in the presence of large-scale camera motion or complex backgrounds. The use of multiple frames for motion estimation makes it more robust to noise, occlusion, and incomplete motion information. By integrating information from multiple frames, it can more accurately capture the overall motion pattern. By utilizing a recurrent neural network to model the temporal dependencies between frames, it can better handle non-rigid motion in videos, improving the accuracy of frame matching and thus obtaining more precise global motion scenes.

[0064] 3. Compared to the traditional K-means algorithm, the SOM-K-means algorithm is more suitable for processing complex data sets and can better capture hidden patterns and relationships in the data. During the training process, it adjusts the node weights on the grid to adapt to the data distribution, enabling self-learning. This makes the algorithm more robust and scalable when processing large-scale data sets. The proximity relationships in the grid structure provide a certain tolerance for noise and outliers. When there is some noise or outliers in the data, the algorithm's clustering results are relatively stable and not easily affected by these interference factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 A flowchart of the steps of a SLAM method for removing pure dynamic points by integrating a deep neural network provided by the present invention;

[0067] Figure 2 The present invention provides a SOM-K-means algorithm framework for a SLAM method that integrates a deep neural network to remove pure dynamic points. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0069] like Figures 1 to 2 As shown, the technical solution adopted by the invention is as follows: a SLAM method that integrates a deep neural network to remove pure dynamic feature points, comprising at least the following steps:

[0070] Step 1: Use an RGB-D camera to capture RGB and depth images and input them into the system. Then, use YOLOv8 to process each input image, output the coordinates and category information of each bounding box, and publish it to the SLAM through ROS2. SLAM classifies all object types detected by YOLOv8 into two categories based on the object's potential motion state: (1) static objects: such as tables, consoles, and walls; (2) dynamic objects: such as people and pets. Then, based on the coordinates and category information of the bounding box, the feature points are classified for the next step.

[0071] Step 2: Use the depth image captured by the RGB-D camera and the camera calibration parameters to calculate the 3D spatial coordinates corresponding to each pixel through the principle of triangulation. Triangulation utilizes the relationship between the distance from the camera to the object, the camera intrinsic parameters, and the pixel coordinates for calculation.

[0072] Step 3: Two different strategies are used to remove highly dynamic feature points. The remaining static feature points are then frame-matched using the RAFT method of the aggregated GMA, and then self-motion estimation is performed in conjunction with OpenCV functions to restore the initial camera pose.

[0073] Step 4: Reproject the 3D feature points of the previous frame to the current frame through the initial camera pose and calculate the reprojection offset vector, where the reprojection offset vector is used to describe the displacement of the spatial point.

[0074] Step 5: Divide the 3D feature points of the current frame into k clusters using the SOM-K-means algorithm.

[0075] Step 6: For each cluster, calculate and determine the type of each cluster, and remove all feature points in the dynamic cluster according to the determination result.

[0076] Step 7: Embed the remaining static feature points into the SLAM system for tracking and mapping.

[0077] Among them, in step 1: use YOLOv8 to process each image captured by the RGB-D camera, identify the objects in the image and generate bounding boxes and their positions and classification information, so as to pass them into the SLAM system for binary classification of the identified objects and extraction of dynamic and static feature points.

[0078] In step 2, the depth image captured by the RGB-D camera is combined with semantic information to calculate the coordinates of each pixel in the three-dimensional space using the camera's intrinsic parameters (such as focal length and principal point) and the depth value corresponding to each pixel in the depth image. The specific principles are as follows:

[0079]

[0080]

[0081] in,( ) is the focal length of the camera, ( ) is the camera's principal point (same as above), which describes the camera's field of view, pixel size, and other information. (u, v) represents the pixel's coordinates, and (x, y, z) represents the three-dimensional coordinates. The depth image is a two-dimensional image, and the grayscale value of each pixel represents the depth information of that point. By reading the depth value of each pixel in the depth image, we can determine the distance from that pixel to the camera.

[0082] Among them, in step three: Since all existing dynamic SLAM systems that use target detection directly remove all feature points in the dynamic bounding box. In fact, the bounding box occupies more area in the RGB image than the object. In particular, when the dynamic bounding box occupies a large proportion of the image area, if all feature points in the dynamic bounding box are directly removed, the remaining static feature points may not generate satisfactory point matches. In order to improve the robustness and speed of SLAM, two different strategies are used to prevent extreme cases: (1) When the number of correct point matches does not exceed 8, we retain the feature points in the dynamic bounding box but not in the static bounding box for further testing. (2) When the number of correct point matches is greater than 8, the feature points in the dynamic bounding box are directly removed. In addition, feature points that are too close to the image edge and the bounding box are also discarded. Then, the remaining feature points in the current frame are matched with the previous frame using the RAFT method of aggregated GMA. The potential static matching point set in the static area is identified by the RAFT method of aggregated GMA and semantic prior information. For the global aggregation module GMA, the query and key features are projections of the context feature map, which are used to model the appearance self-similarity in the first frame. The value feature is a projection of the motion feature, which itself encodes a four-dimensional correlation volume. The attention matrix calculated from the query and key features is used to aggregate the value features as a hidden representation of the motion. The following equation is the calculation of the global aggregate feature:

[0083]

[0084] in, It is the movement characteristic of RAFT. is a learnable parameter. , , are encoding functions for query, key, and value respectively. F is the similarity calculation function, as shown in the following formula:

[0085]

[0086]

[0087]

[0088]

[0089] The calculation of the above global aggregation features plus the position encoding information is shown in the following formula:

[0090]

[0091] in, Indicates positional encoding. and Represent the potential static matching point sets of the previous frame and the current frame respectively. Then, the most accurate matching static feature point pairs are selected using epipolar constraints and RANSAC. The epipolar constraint model can be expressed as follows:

[0092]

[0093] Where F is the fundamental matrix, and yes and Potential static feature points in .

[0094] For each , we calculate and the corresponding epipolar line The distance between them can be calculated using the following equation:

[0095]

[0096] A threshold of 0.1 is set. If D is greater than the threshold, the corresponding feature point pair is removed. E is the essential matrix, which can be calculated using the following equation:

[0097]

[0098] Where K is the camera intrinsic matrix. The feature point pair on the left and E are fed into the OpenCV function.

[0099] Among them, in step 4: Assume is a static point, is a dynamic point, { } represents the feature point in the image coordinate system at time t, { } represents the matching feature points corresponding to time t-1. Then, at time t-1, the previous frame image and reprojection { } to the camera coordinate system { }. Similarly, we can get the depth information of the camera coordinate system at time t. }. Therefore, there will be two offset vectors: , l represents the modulus of the offset vector, which can be calculated using the following equation:

[0100]

[0101] in, The angle of the offset vector can be calculated using the following equation:

[0102]

[0103] Ideally, and should overlap because is a static point, } is the corresponding feature pair. However, due to the existence of random errors, there is an offset vector. Based on their true movement in spatial coordinates, a simple and easy-to-understand method is proposed to distinguish dynamic points from static points, as shown in the following formula:

[0104]

[0105]

[0106]

[0107] in, Represents the back-projection function, which depends on the camera type, but at this time At time t-1, it is in the camera coordinate system. Projection to At time t, the initial camera pose T(R,t) is given by the equation Get the offset vector can be Calculated. When a random error occurs, if the depth d is large, the modulus of the offset vector is also large. On the contrary, the angle of the offset vector is small. A weighted average method is used to describe the offset vector: in the current frame, the set of feature points outside the dynamic bounding box is The set of feature points in the dynamic bounding box is Using the above equation, we get The corresponding offset vector is then calculated using the following equation :

[0108]

[0109] in, It is expressed in radians. Multiplying it by 0.7 is numerically similar to There may be a small number of outliers due to incorrect matching or depth information. We remove these outliers and calculate the remaining feature points The average value of , denoted as φ:

[0110]

[0111] Among them, where: is the number of remaining inliers.

[0112] Among them, in step 5: using the SOM-K-means clustering algorithm, which is the optimization algorithm of SOM, not only solves the problem of low convergence efficiency and inaccurate classification results of the SOM neural network, but also overcomes the problem of presetting the K-means clustering center and K value. The specific steps of the algorithm are as follows: first, input the feature points on the dynamic bounding box into the SOM neural network for clustering, output the number of classifications and the preliminary classification results, and use the obtained initial classification results as the k value. Then randomly set k 3D points (j=1,2,3,…,k) as the center of these 3D feature points and calculate and The distance is as follows:

[0113]

[0114] in are the 3D feature points that need to be classified, The number of is n, The number of is k.

[0115] For each Find the closest 3D point to it As its center. Therefore, each 3D feature point will belong to a cluster (center). Then update the new center of each cluster and iterate continuously until the cluster of each 3D feature point remains unchanged. This process can be summarized as the goal of the K-means algorithm is to minimize the loss function J:

[0116]

[0117] in, is a binary variable, , 1 represents a three-dimensional feature point belongs to the jth cluster, and 0 means it does not belong to the jth cluster.

[0118] Among them, in step six: , the corresponding three-dimensional feature point set is We use the K-means method to is divided into k clusters. Accordingly, is also divided into k clusters. For each cluster, we calculate the mean of Th. Then it is used to determine the type of each cluster. The mean of Th in K clusters is recorded as

[0119]

[0120] Where: w is the number of feature points in the kth cluster.

[0121] For each cluster, we calculated and with ( The type of the kth cluster can be defined as follows:

[0122]

[0123] If the type is determined to be dynamic, all feature points in the cluster are determined to be purely dynamic; otherwise, the cluster is static. Feature points determined to be dynamic are then removed.

[0124] Among them, in step seven: the remaining static points are embedded into the SLAM system for tracking and mapping. After the above steps, the static feature points can be retained to the greatest extent to improve the accuracy of mapping.

[0125] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, material, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, material, or apparatus.

[0126] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A SLAM method that integrates a deep neural network to remove purely dynamic feature points, comprising at least the following steps: Step 1: Use an RGB-D camera to collect RGB images and depth images and input them into the system. Then use YOLOv8 to process each input image, output the coordinates and category information of each bounding box, and publish it to SLAM through ROS2. SLAM classifies all object types detected by YOLOv8 into two categories according to the potential motion state of the object: (1) static objects and (2) dynamic objects. Then, the feature points are classified according to the coordinates and category information of the bounding box for the next step; Step 2: Use the depth image collected by the RGB-D camera and the camera calibration parameters to calculate the three-dimensional space coordinates corresponding to each pixel through the triangulation principle. Among them, triangulation is calculated using the relationship between the distance from the camera to the object, the camera intrinsic parameters and the pixel coordinates; Step 3: Two different strategies are used to remove highly dynamic feature points: (1) When the number of correct point matches does not exceed 8, the feature points that are in the dynamic bounding box but not in the static bounding box are retained for further testing; (2) When the number of correct point matches is greater than 8, the feature points in the dynamic bounding box are directly removed; in addition, feature points that are too close to the image edge and bounding box are also discarded; the remaining static feature points are then frame matched using the RAFT method of aggregated GMA, and then self-motion estimation is performed in combination with OpenCV functions to restore the initial camera pose; Step 4: Reproject the 3D feature points of the previous frame to the current frame through the initial camera pose and calculate the reprojection offset vector. Among them, the reprojection offset vector is used to describe the displacement of the spatial point; Step 5: Divide the 3D feature points of the current frame into k clusters using the SOM-K-means algorithm; Step 6: For each cluster, calculate and determine the type of each cluster, and remove all feature points in the dynamic cluster according to the determination result; Step 7: Embed the remaining static feature points into the SLAM system for tracking and mapping.

2. A SLAM method for removing pure dynamic feature points by integrating a deep neural network according to claim 1, characterized in that: In step 1: YOLOv8 is used to process each image captured by the RGB-D camera, identify the objects in the image and generate bounding boxes and their positions and classification information, so as to be passed to the SLAM system for binary classification of the identified objects and extraction of dynamic and static feature points.

3. A SLAM method for removing pure dynamic feature points by integrating a deep neural network according to claim 1, characterized in that: In step 2, the depth image captured by the RGB-D camera is combined with semantic information to calculate the coordinates of each pixel in the 3D space using the camera's intrinsic parameters and the depth value corresponding to each pixel in the depth image. The intrinsic parameters include the focal length and the principal point. The specific principles are as follows: ； ； in,( ) is the focal length of the camera, ( ) is the camera's principal point, which describes the camera's field of view and pixel size information. (u, v) represents the coordinates of the pixel point, and (x, y, z) represents the three-dimensional coordinates. The depth image is a two-dimensional image. The grayscale value of each pixel represents the depth information of the point. By reading the depth value of each pixel in the depth image, the distance from the pixel to the camera can be known.

4. The SLAM method according to claim 1, wherein: In step 3: Since all existing dynamic SLAM systems that use target detection directly remove all feature points in the dynamic bounding box, in fact, the bounding box occupies more area in the RGB image than the object. In particular, when the dynamic bounding box occupies a large proportion of the image area, if all feature points in the dynamic bounding box are directly removed, the remaining static feature points may not generate satisfactory point matches. In order to improve the robustness and speed of SLAM, two different strategies are used to prevent extreme situations: (1) When the number of correct point matches does not exceed 8, the feature points in the dynamic bounding box but not in the static bounding box are retained for further testing; (2) When the number of correct point matches is greater than 8, the feature points in the dynamic bounding box but not in the static bounding box are directly removed. The feature points in the dynamic bounding box are used. Moreover, the feature points that are too close to the edge of the image and the bounding box are also discarded. Then, the remaining feature points in the current frame are matched with the previous frame using the RAFT method of the aggregated GMA. The potential static matching point set in the static area is identified by the RAFT method of the aggregated GMA and the semantic prior information. For the global aggregation module GMA, the query and key features are projections of the context feature map, which are used to model the appearance self-similarity in the first frame. The value feature is a projection of the motion feature, which itself is an encoding of the four-dimensional correlation volume. The attention matrix calculated from the query and key features is used to aggregate the value features as the hidden representation of the motion. The following equation is the calculation of the global aggregate feature: ； in, It is the movement characteristics of RAFT. is a learnable parameter, , , are the encoding functions for query, key, and value respectively, and F is the similarity calculation function, as shown in the following formula: ； ； ； ； The calculation of the above global aggregation features plus the position encoding information is shown in the following formula: ； in, represents the position code, and Represent the potential static matching point sets of the previous frame and the current frame respectively, and then use the epipolar constraint and RANSAC to filter out the most accurate matching static feature point pairs. The epipolar constraint model can be expressed by the following equation: ； Where F is the fundamental matrix, and yes and Potential static feature points in For each ,calculate and the corresponding epipolar line The distance between them can be calculated by the following equation: ； A threshold of 0.1 is set. If D is greater than the threshold, the corresponding feature point pair is removed. E is the essential matrix, which can be calculated by the following equation: ； Where: K is the camera intrinsic matrix, and the feature point pair on the left and E are sent to the OpenCV function.

5. The SLAM method according to claim 1, wherein: In step 4: Assume is a static point, is a dynamic point, } represents the feature point in the image coordinate system at time t, { } represents the matching feature points corresponding to time t-1, and then obtains the previous frame image and reprojection at time t-1 { } to the camera coordinate system { }, similarly, we can get the depth information of the camera coordinate system at time t. }, so there will be two offset vectors: , , l represents the modulus of the offset vector, which can be calculated using the following equation: ； in, The angle of the offset vector can be calculated using the following equation: ； Ideally, and should overlap because is a static point, } is the corresponding feature pair. However, due to the existence of random errors, there is an offset vector. Based on their true movement in the spatial coordinates, a simple and easy-to-understand method is proposed to distinguish dynamic points from static points, as shown in the following formula: ； ； ； in, Represents the back-projection function, which depends on the camera type, but at this time At time t-1, in the camera coordinate system, Projection to At time t, the initial camera pose T(R,t) is given by the equation Get, offset vector can be It is calculated that when random errors occur, if the depth d is large, the modulus of the offset vector is also large. On the contrary, the angle of the offset vector is small. A weighted average method is used to describe the offset vector: in the current frame, the set of feature points outside the dynamic bounding box is The set of feature points in the dynamic bounding box is , using the above equation, we get The corresponding offset vector is then calculated using the following equation : ； in, Expressed in radians, it multiplied by 0.7 is numerically similar to ,Due to incorrect matching or depth information, there may be a small number of outliers. Remove these outliers and calculate the remaining feature points The average value of , denoted as φ: ； Where: is the number of remaining inliers.

6. The SLAM method according to claim 1, wherein: In step five: using the SOM-K-means clustering algorithm, which is the optimization algorithm of SOM, it not only solves the problems of low convergence efficiency and inaccurate classification results of the SOM neural network, but also overcomes the preset problem of K-means clustering center and K value. The specific steps of the algorithm are as follows: first, the feature points on the dynamic bounding box are input into the SOM neural network for clustering, and the number of classifications and the preliminary classification results are output. The initial classification results are used as the k value, and then k 3D points are randomly set. (j=1,2,3,…,k) as the center of these 3D feature points and calculate and The distance is as follows: ； in, are the 3D feature points that need to be classified, The number of is n, The number of is k, For each Find the closest 3D point to it As its center, therefore, each 3D feature point will belong to the center of a cluster, and then update the new center of each cluster, and continue to iterate until the cluster of each 3D feature point remains unchanged. This process can be summarized as the purpose of the K-means algorithm is to minimize the loss function J: ； in, is a binary variable, , 1 represents a three-dimensional feature point belongs to the jth cluster, and 0 means it does not belong to the jth cluster.

7. The SLAM method according to claim 1, wherein: In step six: , the corresponding three-dimensional feature point set is , through the K-means method Divide into k clusters, accordingly, It is also divided into k clusters. For each cluster, the mean of Th is calculated and then used to determine the type of each cluster. The mean value of Th in K clusters is recorded as : ； Where: w is the number of feature points in the kth cluster; For each cluster, we calculated and with A comparison was made. for The average value of Th in , the type of the k-th cluster can be defined as follows: ； If the type is determined to be dynamic, all feature points in the cluster are determined to be purely dynamic; Otherwise, the cluster is static, and the feature points that are determined to be dynamic are then removed.

8. The SLAM method according to claim 1, wherein: In step seven: the remaining static points are embedded into the SLAM system for tracking and mapping. After the above steps, the static feature points can be retained to the greatest extent to improve the accuracy of mapping.

Images