A dynamic visual slam method

Abstract

The application discloses a dynamic visual SLAM method, comprising the following steps: 1) reading image data of a video stream; 2) inputting the image into a front end to detect ORB feature points; 3) detecting common dynamic objects by YOLO and providing minimum boundary boxes of the dynamic objects; 4) detecting dynamic objects in the boundary boxes by a dynamic ViBe and reserving static environments in the dynamic objects; 5) combining coordinate information of dynamic features, comparing the coordinate information of the dynamic features with ORB feature points in a visual odometer, detecting ORB feature points falling within a radius and eliminating the ORB feature points within the radius; 6) optimally selecting key frames without dynamic objects, establishing global constraints of time and space spans by using the key frames without dynamic objects and optimizing camera poses of each frame; and 7) judging whether a current frame is the last frame, and if not, jumping to step 2), and if yes, ending. Compared with a traditional MonoSLAM method, the application can eliminate influences of dynamic objects in an environment on a MonoSLAM system, thereby improving positioning accuracy and mapping effects.

Description

Technical Field

[0001] This invention belongs to the field of visual navigation technology, specifically relating to a dynamic visual SLAM method. Background Technology

[0002] To perform its tasks, a robot needs to perceive its surroundings and locate itself using sensors such as cameras—small, inexpensive, low-power, and information-rich sensors. Visual SLAM is a field encompassing various technologies, with monocular VSLAM, binocular VSLAM, and RGB-D VSLAM being its main branches. Binocular VSLAM, based on the principle of parallax, can accurately measure the depth information of pixels, while RGB-D VSLAM assigns a depth value to each pixel using Time-of-Flight (ToF) technology or structured light. Monocular VSLAM has limitations in acquiring pixel depth, but its computational cost is relatively low and its adaptability is wide, making it highly favored. Especially in the application of small mobile robots, monocular VSLAM plays an indispensable role and has extremely high research significance and value. Summary of the Invention

[0003] The purpose of this invention is to address the problem that the MonoSLAM algorithm is susceptible to interference from dynamic objects in the environment. It uses the MonoSLAM algorithm as the core, combines dynamic ViBe and YOLO to eliminate dynamic features in visual odometry, and uses RANSAC (Random Sample Consensus) to remove dynamic feature points in visual odometry, thereby eliminating the influence of dynamic objects in the environment on MonoSLAM and improving the positioning accuracy and mapping effect of the MonoSLAM system.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A dynamic visual SLAM method includes the following steps: Step 1: Read the image data from the video stream; Step 2: Input the image into the front end to detect ORB feature points; Step 3: Detect common dynamic objects using YOLO and provide their minimum bounding boxes; Step 4: Use the dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment within them; Step 5: Combine Ransac, YOLO, and ViBe to statistically obtain the coordinate information of dynamic features, compare the coordinates of dynamic features with ORB feature points in visual odometry; detect ORB feature points falling within the radius, and eliminate ORB feature points within this radius. Step 6: Select keyframes without dynamic objects and use these keyframes to establish global constraints in time and space to optimize the camera pose of each frame. Step 7: Determine if the current frame is the last frame. Otherwise, go back to step 2. If it is, end the process.

[0005] Specifically, in step 2, the image is input to the front end to detect ORB feature points. Based on the video stream image data read in step 1, the ORB algorithm is used to detect point features for each frame of the image. To cope with image rotation changes, the image gray-level centroid near the feature points is calculated. The image gray-level centroid is a center point with weights based on the gray-level values ​​of the image blocks. Its position can reflect the gray-level distribution characteristics of the image blocks, thereby achieving rotation-invariant feature extraction. The calculation process of the image gray-level centroid is as follows: Centroid of image patch Represented as: (1); In the above formula, x and y represent pixel coordinates; This represents the grayscale value of this coordinate; where ; The direction of the centroid can be obtained by connecting the geometric center O and the centroid C: (2); (3).

[0006] Specifically, step 3 involves detecting common dynamic objects using YOLO and providing their minimum bounding boxes. The YOLO network framework employs a deep convolutional neural network (CNN) and is pre-trained on the COCO dataset. The steps are as follows: Step 3.1: The image is input into a convolutional neural network for feature extraction. Each cell generates multiple bounding boxes based on the prediction results, and redundant bounding boxes are eliminated by Non-Maximum Suppression (NMS). Step 3.2: The input image is segmented into a fixed-size grid. Each cell is responsible for predicting a set of bounding boxes and the confidence score of the corresponding category. Each bounding box contains the object's location information and the confidence score associated with each category. The softmax function is used to calculate the probability that each bounding box belongs to a different category. The category prediction of each bounding box is calculated based on the overlap (IoU) between the bounding box and the target instance and the category confidence score. Step 3.3: Output the detected target bounding box position.

[0007] Furthermore, each bounding box contains the object's position information, including the x and y coordinates of the top-left corner, as well as its width and height.

[0008] Specifically, step 4 involves using dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment. The process is as follows: Step 4.1: Use a motion compensation algorithm to compensate for the dynamic background of the dynamic ViBe. Affine transformation or affine mapping refers to a specific mapping relationship between two vector spaces. It is composed of a linear transformation and a translation operation. This transformation can maintain the linear relationship and relative position of the vectors, but translates them as a whole, thereby achieving an effective conversion between vector spaces. The six-parameter affine model is described as follows: (4); In the above formula, It represents a specific pixel coordinate from the previous frame of the image; is the pixel corresponding to the current frame; c and f are translation parameters; a, b, d and e are motion parameters related to scaling and rotation; Step 4.2: Compensate the background image using motion model parameters and a six-parameter affine model, assuming the current frame... The coordinates of a certain pixel are Previous frame The corresponding pixel coordinates are The affine transformation based on the six-parameter affine model is obtained. ,right Each pixel is subjected to affine transformation to obtain the compensated image. ; Step 4.3: Use the compensation map to model the background, and set the radius threshold coefficient as follows. The matching threshold coefficient is The difference coefficient is used This is represented in the original image sample library. , No. OK The columns of pixels are used Represented by pixels in the second frame image To conduct a detailed analysis, this point Pixels in the sample library The pixel values ​​are subtracted and their absolute values ​​are taken in a mathematical calculation. The difference is then used... Representation; pixel If a pixel is identified as a foreground pixel, the pixel value at the corresponding position in the final value template is overwritten with the pixel value of the corresponding background image in the sample library to segment and detect the target, while saving the static environment within the bounding box.

[0009] Specifically, in step 5, Ransac, YOLO, and ViBe are combined to statistically obtain the coordinate information of dynamic features. When a dynamic object appears in an image frame, YOLO detects the image and provides the coordinate information of the minimum bounding box of the dynamic object; ViBe detects the target in the bounding box and segments the dynamic object from the static environment; Ransac detects the dynamic object in the bounding box; if a dynamic feature satisfies two of the three detection methods, it is determined to be a dynamic feature point a.

[0010] Specifically, in step 5, ORB feature points falling within the radius are detected and eliminated. A circle is drawn with the dynamic feature point a as the center and R as the radius. ORB feature points falling within the radius R are eliminated. The radius R is determined based on the movement distance between two frames.

[0011] Specifically, in step 6, the keyframes without dynamic objects are selected. Keyframes with dynamic objects are not considered because the movement of the objects causes the system to ignore more static environmental features, resulting in less environmental information.

[0012] Compared with the prior art, the present invention has the following beneficial effects: This invention uses the MonoSLAM algorithm as its core, combining dynamic ViBe and YOLO to eliminate dynamic features in visual odometry. At the same time, it utilizes RANSAC (Random Sample Consensus) to remove dynamic feature points in visual odometry, thereby eliminating the influence of dynamic objects in the environment on MonoSLAM and improving the positioning accuracy and mapping effect of the MonoSLAM system. Attached Figure Description

[0013] Figure 1 This is a flowchart of a dynamic visual SLAM method according to the present invention; Figure 2 This is a feature distribution diagram comparing the dynamic visual SLAM method of this invention with the traditional MonoSLAM method; Figure 3 The trajectory and sparse point cloud map of the dynamic visual SLAM method of this invention and the traditional MonoSLAM method; Figure 4 The ATE trajectory error between the dynamic visual SLAM method of this invention and the traditional MonoSLAM method. Detailed Implementation

[0014] To better demonstrate the advantages and innovations of this invention and to clearly illustrate its technical solution, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings.

[0015] The technical terms used in the following embodiments are understandable to those skilled in the art. Unless otherwise defined, all terms (including technical and scientific terms) herein have meanings similar to those understood by one of ordinary skill in the art. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0016] Example 1 like Figure 1 As shown, this invention provides a dynamic visual SLAM method, comprising the following steps: Step 1: Read the image data from the video stream; Step 2: Input the image into the front end to detect ORB feature points; Step 3: Detect common dynamic objects using YOLO and provide their minimum bounding boxes; Step 4: Use the dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment within them; Step 5: Combine Ransac, YOLO, and ViBe to statistically obtain the coordinate information of dynamic features, compare the coordinates of dynamic features with ORB feature points in visual odometry; detect ORB feature points falling within the radius, and eliminate ORB feature points within this radius. Step 6: Select keyframes without dynamic objects and use these keyframes to establish global constraints in time and space to optimize the camera pose of each frame. Step 7: Determine if the current frame is the last frame. Otherwise, go back to step 2. If it is, end the process.

[0017] Specifically, in step 2, the image is input to the front end to detect ORB feature points. Based on the video stream image data read in step 1, the ORB algorithm is used to detect point features for each frame of the image. To cope with image rotation changes, the image gray-level centroid near the feature points is calculated. The image gray-level centroid is a center point with weights based on the gray-level values ​​of the image blocks. Its position can reflect the gray-level distribution characteristics of the image blocks, thereby achieving rotation-invariant feature extraction. The calculation process of the image gray-level centroid is as follows: Centroid of image patch Represented as: (1); In the above formula, x and y represent pixel coordinates; This represents the grayscale value of this coordinate; where ; The direction of the centroid can be obtained by connecting the geometric center O and the centroid C: (2); (3).

[0018] Specifically, step 3 involves detecting common dynamic objects using YOLO and providing their minimum bounding boxes. The YOLO network framework employs a deep convolutional neural network (CNN) and is pre-trained on the COCO dataset. The steps are as follows: Step 3.1: The image is input into a convolutional neural network for feature extraction. Each cell generates multiple bounding boxes based on the prediction results, and redundant bounding boxes are eliminated by Non-Maximum Suppression (NMS). Step 3.2: The input image is segmented into a fixed-size grid. Each cell is responsible for predicting a set of bounding boxes and the confidence score of the corresponding category. Each bounding box contains the object's location information and the confidence score associated with each category. The softmax function is used to calculate the probability that each bounding box belongs to a different category. The category prediction of each bounding box is calculated based on the overlap (IoU) between the bounding box and the target instance and the category confidence score. Step 3.3: Output the detected target bounding box position.

[0019] Furthermore, each bounding box contains the object's position information, including the x and y coordinates of the top-left corner, as well as its width and height.

[0020] Specifically, step 4 involves using dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment. The process is as follows: Step 4.1: Use a motion compensation algorithm to compensate for the dynamic background of the dynamic ViBe. Affine transformation or affine mapping refers to a specific mapping relationship between two vector spaces. It is composed of a linear transformation and a translation operation. This transformation can maintain the linear relationship and relative position of the vectors, but translates them as a whole, thereby achieving an effective conversion between vector spaces. The six-parameter affine model is described as follows: (4); In the above formula, It represents a specific pixel coordinate from the previous frame of the image; is the pixel corresponding to the current frame; c and f are translation parameters; a, b, d and e are motion parameters related to scaling and rotation; Step 4.2: Compensate the background image using motion model parameters and a six-parameter affine model, assuming the current frame... The coordinates of a certain pixel are Previous frame The corresponding pixel coordinates are The affine transformation based on the six-parameter affine model is obtained. ,right Each pixel is subjected to affine transformation to obtain the compensated image. ; Step 4.3: Use the compensation map to model the background, and set the radius threshold coefficient as follows. The matching threshold coefficient is The difference coefficient is used This is represented in the original image sample library. , No. OK The columns of pixels are used Represented by pixels in the second frame image To conduct a detailed analysis, this point Pixels in the sample library The pixel values ​​are subtracted and their absolute values ​​are taken in a mathematical calculation. The difference is then used... Representation; pixel If a pixel is identified as a foreground pixel, the pixel value at the corresponding position in the final value template is overwritten with the pixel value of the corresponding background image in the sample library to segment and detect the target, while saving the static environment within the bounding box.

[0021] Specifically, in step 5, Ransac, YOLO, and ViBe are combined to statistically obtain the coordinate information of dynamic features. When a dynamic object appears in an image frame, YOLO detects the image and provides the coordinate information of the minimum bounding box of the dynamic object; ViBe detects the target in the bounding box and segments the dynamic object from the static environment; Ransac detects the dynamic object in the bounding box; if a dynamic feature satisfies two of the three detection methods, it is determined to be a dynamic feature point a.

[0022] Specifically, in step 5, ORB feature points falling within the radius are detected and eliminated. A circle is drawn with the dynamic feature point a as the center and R as the radius. ORB feature points falling within the radius R are eliminated. The radius R is determined based on the movement distance between two frames.

[0023] Specifically, in step 6, the keyframes without dynamic objects are selected. Keyframes with dynamic objects are not considered because the movement of the objects causes the system to ignore more static environmental features, resulting in less environmental information.

[0024] The following comparative experiments use the traditional MonoSLAM method and the dynamic visual SLAM method of this invention to demonstrate the feasibility and scientific validity of the method of this invention. The video materials involved in the experiments are from the TUM dataset.

[0025] Unlike the traditional MonoSLAM method, the dynamic visual SLAM method of this invention uses the YOLO algorithm to detect dynamic objects and provide the minimum bounding box coordinates, which includes a part of the static environment; then, the dynamic ViBe algorithm is used to detect dynamic objects within the bounding box, retaining part of the static environment; subsequently, the RANSAC algorithm is used to perform linear fitting of feature points, and the best fitting model is selected to detect feature points falling on dynamic targets.

[0026] from Figure 2 (Left: Traditional MonoSLAM method; Right: Dynamic Visual SLAM method) The yellow box clearly shows that the feature distribution of the traditional MonoSLAM method is relatively concentrated, and half of the features are dynamic. This obviously interferes with the MonoSLAM system and can even cause serious consequences in daily work. In contrast, the feature points of the dynamic visual SLAM method of this invention are mostly distributed on edges and textured objects, which can clearly distinguish between dynamic and static feature points, effectively avoiding interference from dynamic feature points on the MonoSLAM system.

[0027] exist Figure 2 Based on this, the coordinates of the dynamic features in the image are compared one by one with the ORB feature points of the corresponding frame to eliminate the interference of dynamic features in visual odometry on the system positioning, while ensuring that there are no dynamic objects in the sparse point cloud image. If the current frame is not the last frame, the previous steps are repeated until the image detection is completed.

[0028] like Figure 3 As shown (left: traditional MonoSLAM method; right: dynamic visual SLAM method), the red line represents the predicted route, the pink line represents the optimized route, and the green line represents the actual trajectory. Since dynamic features generally correspond to obstacles in real life, comparing the trajectories and sparse point cloud maps of the dynamic visual SLAM method of this invention with those of the traditional MonoSLAM method shows that the dynamic visual SLAM method provided by this invention can effectively prevent robots equipped with MonoSLAM from avoiding unnecessary damage, and it does not reduce or even improves localization performance. Furthermore, according to... Figure 4 The comparison results of absolute trajectory error (ATE) shown in the image (left: traditional MonoSLAM method; right: dynamic visual SLAM method) effectively illustrate the harm of dynamic features to MonoSLAM. Figure 4The black lines represent the true trajectory, the blue lines represent the predicted trajectory of the keyframes, and the red lines represent the error between the predicted trajectory and the true trajectory. The trajectory represented by the traditional MonoSLAM method is roughly the same as the true trajectory in shape, but there are more red lines, which indicates that there is a significant deviation in the predicted trajectory. On the other hand, the trajectory represented by the dynamic visual SLAM method of this invention maintains a better shape and is closer to the true trajectory. The red lines are close to a point, which shows that the scale shift phenomenon is significantly improved after adopting the dynamic visual SLAM method of this invention.

[0029] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A dynamic visual SLAM method, characterized in that, Includes the following steps: Step 1: Read the image data from the video stream; Step 2: Input the image into the front end to detect ORB feature points; Step 3: Detect common dynamic objects using YOLO and provide their minimum bounding boxes; Step 4: Use the dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment within them; Step 5: Combine Ransac, YOLO, and ViBe to statistically obtain the coordinate information of dynamic features, compare the coordinates of dynamic features with ORB feature points in visual odometry; detect ORB feature points falling within the radius, and eliminate ORB feature points within this radius. The method combines Ransac, YOLO, and ViBe to statistically obtain the coordinate information of dynamic features. When a dynamic object appears in an image frame, YOLO detects the image and provides the coordinate information of the minimum bounding box of the dynamic object; ViBe detects the target in the bounding box and segments the dynamic object from the static environment; Ransac detects the dynamic object in the bounding box; if a dynamic feature satisfies two of the three detection methods, it is determined to be a dynamic feature point a. Draw a circle with dynamic feature point a as the center and radius R as the radius, and eliminate ORB feature points that fall within the radius R. The radius R is determined based on the movement distance between two frames. Step 6: Select keyframes without dynamic objects and use these keyframes to establish global constraints in time and space to optimize the camera pose of each frame. Step 7: Determine if the current frame is the last frame. Otherwise, go back to step 2. If it is, end the process.

2. The dynamic visual SLAM method according to claim 1, characterized in that, In step 2, the image is input to the front end to detect ORB feature points. Based on the video stream image data read in step 1, the ORB algorithm is used to detect point features for each frame. To handle image rotation changes, the image gray-level centroid near the feature points is calculated. The image gray-level centroid is a center point weighted by the gray-level value of the image block, and its position reflects the gray-level distribution characteristics of the image block, thereby achieving rotation-invariant feature extraction. The calculation process of the image gray-level centroid is as follows: Centroid of image patch Represented as: （1）； In the above formula, x and y represent pixel coordinates; This represents the grayscale value of this coordinate; where ; The direction of the centroid can be obtained by connecting the geometric center O and the centroid C: （2）； （3）。 3. The dynamic visual SLAM method according to claim 1, characterized in that, Step 3 describes detecting common dynamic objects using YOLO and providing their minimum bounding boxes. The YOLO network framework employs a deep convolutional neural network (CNN) and is pre-trained on the COCO dataset. The specific steps are as follows: Step 3.1: The image is input into a convolutional neural network for feature extraction. Each cell generates multiple bounding boxes based on the prediction results, and redundant bounding boxes are eliminated by Non-Maximum Suppression (NMS). Step 3.2: The input image is segmented into a fixed-size grid. Each cell is responsible for predicting a set of bounding boxes and the confidence score of the corresponding category. Each bounding box contains the object's location information and the confidence score associated with each category. The softmax function is used to calculate the probability that each bounding box belongs to a different category. The category prediction of each bounding box is calculated based on the overlap (IoU) between the bounding box and the target instance and the category confidence score. Step 3.3: Output the detected target bounding box position.

4. The dynamic visual SLAM method according to claim 3, characterized in that, Each bounding box contains the object's position information, including the x and y coordinates of the top-left corner, as well as its width and height.

5. The dynamic visual SLAM method according to claim 1, characterized in that, Step 4 describes using dynamic ViBe to detect dynamic objects within the bounding box while preserving the static environment. The process is as follows: Step 4.1: Use a motion compensation algorithm to compensate for the dynamic background of the dynamic ViBe. Affine transformation or affine mapping refers to a specific mapping relationship between two vector spaces. It is composed of a linear transformation and a translation operation. This transformation can maintain the linear relationship and relative position of the vectors, but translates them as a whole, thereby achieving an effective conversion between vector spaces. The six-parameter affine model is described as follows: （4）； In the above formula, It represents a specific pixel coordinate from the previous frame of the image; is the pixel corresponding to the current frame; c and f are translation parameters; a, b, d and e are motion parameters related to scaling and rotation; Step 4.2: Compensate the background image using motion model parameters and a six-parameter affine model, assuming the current frame... The coordinates of a certain pixel are Previous frame The corresponding pixel coordinates are The affine transformation based on the six-parameter affine model is obtained. ,right Each pixel is subjected to affine transformation to obtain the compensated image. ; Step 4.3: Use the compensation map to model the background, and set the radius threshold coefficient as follows. The matching threshold coefficient is The difference coefficient is used This is represented in the original image sample library. , No. OK The columns of pixels are used Represented by pixels in the second frame image To conduct a detailed analysis, this point Pixels in the sample library The pixel values ​​are subtracted and their absolute values ​​are taken in a mathematical calculation. The difference is then used... Representation; pixel If a pixel is identified as a foreground pixel, the pixel value at the corresponding position in the final value template is overwritten with the pixel value of the corresponding background image in the sample library to segment and detect the target, while saving the static environment within the bounding box.

6. The dynamic visual SLAM method according to claim 1, characterized in that, In step 6, the keyframes without dynamic objects are selected. Keyframes with dynamic objects are not considered because the movement of the objects causes the system to ignore more static environmental features, resulting in less environmental information.

Images