A Visual SLAM Method Based on YOLOv8 Model in Dynamic Environments

Abstract

This invention presents a visual SLAM method for dynamic environments based on the YOLOv8 model. The method includes the following steps: acquiring RGB image information and depth information of the scene using an RGB-D (Red-Green-Blue Depth) camera, and matching the two in a spatial coordinate system; feeding the RGB image as input to an improved YOLOv8 model, which identifies and segments prior dynamic objects in the image; then, performing low-cost tracking based on the segmented image; next, segmenting the image using a multi-view geometric method; finally, fusing the YOLOv8 model segmentation result and the multi-view geometric segmentation result, and feeding this fused segmented image as input to the ORB (Oriented Fast and Rotated BRIEF)-SLAM module. This module achieves high-precision localization and mapping, and supports algorithms such as BA (Bundle Adjustment) optimization and loop closure detection to reduce errors. This invention provides a visual SLAM method for dynamic environments based on the YOLOv8 model, which can efficiently complete robot localization and map reconstruction in complex dynamic environments.

Description

Technical Field

[0001] This invention belongs to the field of visual spatial localization, specifically a visual SLAM method based on the YOLOv8 model in dynamic environments. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology is a prerequisite for many robotics applications, such as collision-free navigation and augmented reality. SLAM technology utilizes the robot's onboard sensors to acquire environmental information, enabling the estimation of a map of the unknown environment and the joint estimation of the robot's pose within that map. This map construction makes continuous localization of the robot possible within the same environment, effectively avoiding the cumulative drift of localization errors.

[0003] SLAM can be divided into laser SLAM and vision SLAM depending on the type of sensor used.

[0004] Many classic SLAM algorithms use LiDAR as a sensor. The robot emits a laser beam through a LiDAR sensor and receives the reflected signal to measure the distance and angle between the robot and objects in its environment. Grisetti et al. proposed Gmapping, a 2D LiDAR SLAM algorithm based on particle filtering, which uses probabilistic mapping techniques for environmental mapping. However, as the map expands, the number of particles increases significantly, so it is generally used for building maps of small to medium-sized indoor and outdoor scenes. Google's Cartographer improves upon this, achieving high-precision localization and map building in complex, large-scale map environments. It uses Ceres nonlinear optimization and builds a global map based on submaps, but its performance in 3D mapping is less than ideal. Ji Zhang's LOAM algorithm can build high-precision 3D maps. Its key idea is to distinguish between the complex problems of simultaneous localization and mapping, solving them with separate algorithms. However, LiDAR SLAM algorithms currently suffer from drawbacks such as the high cost of LiDAR equipment, excessive computational load, and poor accuracy in extreme weather conditions.

[0005] With the development of image processing, machine learning, and other technologies, visual SLAM algorithms have gradually received high attention and research in recent years. Visual SLAM algorithms capture continuous image frames using visual sensors such as cameras, extract key feature points from the images, perform feature matching and motion estimation, and thus construct a map of the surrounding environment in real time. LSD-SLAM, proposed by Jakob Engel, is a direct SLAM method that directly uses image pixel values ​​for matching and optimization. This algorithm has high computational complexity and is suitable for scenarios requiring dense mapping. The ORB-SLAM series of algorithms, proposed by Raúl Mur-Artal, is a feature-point-based SLAM algorithm that uses ORB features for keypoint detection and descriptor generation. It includes algorithms such as loop closure detection and graph optimization, exhibiting high robustness and accuracy. However, the accuracy of these algorithms is not ideal in dynamic environments. The DynaSLAM algorithm, proposed by Berta Bescos, enhances dynamic object detection and background inpainting capabilities. It segments prior dynamic images using a Mask R-CNN image segmentation network, demonstrating excellent performance in highly dynamic scenes. However, this method suffers from poor real-time performance, evident not only in the inference latency of the Mask R-CNN but also in the backend SLAM algorithm, and its accuracy has significant room for improvement. Therefore, reducing the inference latency of image segmentation models, improving their robustness and accuracy, and optimizing the backend SLAM algorithm are crucial for designing a lightweight, high-real-time visual SLAM method for dynamic environments.

[0006] The differences between this application and the prior art are as follows:

[0007] Technical comparison with patent CN116758116A "A VSLAM method based on YOLO for indoor dynamic scenes";

[0008] Patent CN116758116A is primarily based on the fundamental YOLO object detection model, outputting rectangular detection boxes rather than precise contours. The core innovation of this patent lies in its improvement of the basic YOLOv8 object segmentation model. This model, while lightweighting the backbone network and convolutional algorithms, introduces the SimAM attention mechanism and improves the segmentation head using a ResNeXt structure. This allows the object segmentation model to not only more accurately segment the specific contours of prior dynamic objects but also significantly improves real-time performance. The two patents differ fundamentally in their deep learning model structure design.

[0009] Patent CN116758116A uses a combination of object detection and geometric methods to remove dynamic points before employing the SLAM algorithm for tracking. This patent, however, introduces a low-cost tracking module after the object segmentation module. This module efficiently determines the robot's pose through rapid feature point matching, significantly improving the overall system's real-time performance. A high-precision SLAM algorithm is then used to correct the robot's pose and reconstruct the map. The two patents differ fundamentally in their tracking method design.

[0010] The SLAM method in patent CN116758116A is only deployed on a personal computer and does not implement the SLAM algorithm on a robot, thus potentially limiting its practical application. The method proposed in this patent has lower algorithmic complexity and achieves environment setup on a unitree A1 quadruped robot equipped with a Raspberry Pi 3B microcomputer. Therefore, this patent can be deployed on low-cost robots, broadening its applicability. The two methods differ fundamentally in their applicable scenarios. Summary of the Invention

[0011] To address the above problems, this invention proposes a visual SLAM method based on the YOLOv8 model in dynamic environments, which overcomes the shortcomings of traditional visual SLAM technology and improves the accuracy and real-time performance of dynamic SLAM.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A visual SLAM method based on the YOLOv8 model in dynamic environments, characterized by the following steps:

[0014] Includes the following steps:

[0015] Step 1: Acquire environmental information using an RGB-D camera:

[0016] An RGB camera and a depth camera acquire RGB images and 3D point cloud images, respectively; the image processing module processes the 3D point cloud image and performs coordinate matching with the RGB image to obtain an RGB-D image.

[0017] Step 2: Training the model and deploying the system:

[0018] The collected RGB images were converted into a YOLO format dataset to train the YOLOv8 model.

[0019] For the trained segmentation model, it is deployed on the microcomputer on the robot, and the SLAM algorithm environment is built in the system and implemented in the system;

[0020] Step 3: Image Segmentation Acquisition and Low-Cost Tracking: For the established SLAM system, the segmentation model is used to segment the prior dynamic objects in the input image to obtain a mask image, which is then tracked using the low-cost tracking module.

[0021] The mask image is then processed in parallel in two ways:

[0022] The first method involves directly inputting the mask image as a segmentation image into the ORB-SLAM tracking and mapping module;

[0023] The second method involves first inputting the mask image into a low-cost tracking module. This module searches for the correspondence between static region feature points in the image frame and local map feature points to find feature matches that minimize projection errors, thereby determining the robot's pose in real time. Subsequently, the mask image is sent to a multi-view geometry module, which compares and analyzes the current image with several historical images that have the highest overlap with it, thereby segmenting dynamic objects in the current image. Finally, the processed image is input into the ORB-SLAM module described in Method 1.

[0024] Step 4, ORB-SLAM Module: The results from the above two methods are fused into the final segmented image, which is then added to the input image sequence of the ORB-SLAM module for processing.

[0025] The tracking thread in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking camera motion, and estimating the pose of the current frame.

[0026] Subsequently, if a closed loop is detected during the robot's movement, the matching relationship between the closed loop keyframe and its corresponding reference scene will be calculated, and similarity transformation and global BA optimization will be performed to eliminate error accumulation and improve the accuracy and consistency of the map.

[0027] Meanwhile, when a new keyframe is inserted into the local map, the keyframes in the local map and the connection relationships between keyframes are updated. The poses and map points of the keyframes in the current local map are also optimized by BA. Finally, redundant keyframes and their corresponding map points are deleted to build the map.

[0028] The resulting map, robot trajectory, direction of movement, and current position will be published as a topic in the ROS system and displayed through the rviz interface.

[0029] The YOLOv8 model described in step two of this invention is a single-stage, real-time object detection framework. Its core idea is to input the entire image into the neural network at once, directly outputting the bounding box coordinates and class probabilities. This approach avoids the multi-stage processing of traditional object detection methods, thereby improving detection speed and efficiency. The model can be divided into three parts: a backbone network, a neck network, and a head network. First, the backbone network extracts multi-scale feature maps from the original image, providing rich feature information for subsequent detection and segmentation tasks. Next, the neck network's feature fusion technique fuses feature maps of different scales to improve the model's ability to detect multi-scale objects. Finally, the head network uses an anchor-free detection head to directly predict the bounding box and class of the target.

[0030] For the backbone network of the improved YOLOv8 model, EfficientNet was used to replace it, which improved the model efficiency. The SimAM attention mechanism was also used to enhance important features without increasing computational complexity, thereby improving the model accuracy.

[0031] For the neck network part of the improved YOLOv8 model, GSConv is used to replace the original convolution, dividing the input feature map into multiple groups and performing convolution operations independently on each group, thereby significantly reducing the amount of computation while maintaining good feature extraction capabilities; while in the head network part, the ResNeXt structure is used to improve the segmentation head of the segmentation model, thereby improving the information processing capability of the segmentation head.

[0032] In addition, the improved YOLOv8 model was lightweighted using pruning and other methods to adapt to the hardware environment of most robots, and was converted into an ONNX model for deployment on GPUs to achieve real-time inference and image segmentation.

[0033] The principle of the low-cost tracking module is to search for the correspondence between static region feature points in the image frame and local map feature points, find feature matching that can minimize projection error, and thus determine the camera pose. This module only tracks the robot's trajectory and does not change the map structure, and can track the robot's pose in real time.

[0034] Furthermore, in the second method of the mask image processing in step three, the multi-view geometry module will extract the current image p in advance. cur The ORB feature points x are then selected, and several historical images (p) with the highest overlap with the current image are chosen. i Where i is the image number (here, 5 historical keyframes were selected as historical images after experimentation), then, the keypoint x of each image is calculated from the historical image p.i To the current image p cur The projection key point x′ and the projection depth z proj Simultaneously, the corresponding three-dimensional spatial key point X is generated, and the angle xXx′ formed by the key point x and x′ and the three-dimensional point X is calculated and denoted as alpha. If alpha is greater than 30 degrees, the point is considered to be dynamic.

[0035] Furthermore, in the second method of the mask image processing in step three, the multi-view geometry module utilizes depth information and employs another method to jointly determine dynamic points: the depth corresponding to the key point x′ is z′, which is compared with the projection depth z. proj For comparison, if the value exceeds the threshold, it is considered a dynamic point, and the formula is: Δz = z proj -z′>τ z , where τ z After experimentation, the distance was set to 0.4m, and the dynamic points can be further marked using the two methods mentioned above.

[0036] Furthermore, in step four, the ORB-SLAM module extracts ORB feature points from the image and uses the tracking thread to determine the robot's trajectory and pose. Then, the local mapping thread is responsible for reconstructing the map. After determining the pose relationship between the current frame and historical frames using the 3D-2D pose estimation method, the thread projects the current frame onto the three-dimensional point cloud coordinate system to complete the construction of the local map. Subsequently, the loop closure detection module is introduced to correct the estimation error, thereby correcting and optimizing the global map.

[0037] Furthermore, the tracking methods in the tracking thread part of the ORB-SLAM module in step four include reference keyframe tracking, constant velocity model tracking, relocation tracking, and local map tracking.

[0038] The main advantages of the aforementioned patent for a visual SLAM method in dynamic environments based on the YOLOv8 model can be summarized as follows:

[0039] High-efficiency dynamic object recognition and segmentation:

[0040] By using an improved YOLOv8 model, which lightweights the backbone network and convolutional algorithm while introducing the SimAM attention mechanism and improving the segmentation head with a ResNeXt structure, the model can more efficiently and accurately identify and segment prior dynamic objects in RGB images. This helps remove unnecessary dynamic elements from the image, thereby improving the accuracy and stability of subsequent SLAM processing.

[0041] Integrating multi-source segmentation information:

[0042] This method not only relies on the segmentation results of the YOLOv8 model but also incorporates multi-view geometric methods for image segmentation. This fusion of multi-source information can further improve the accuracy and robustness of segmentation, especially in complex and dynamic environments.

[0043] Low-cost tracking:

[0044] After identifying and segmenting dynamic objects, the method employs a low-cost tracking technique, which helps reduce the consumption of computational resources while maintaining stable real-time tracking of the prior static environment, thus improving the real-time performance of the entire algorithm.

[0045] Low-cost deployment:

[0046] This method, due to its lightweight improvements to the YOLOv8 model and the introduction of a low-cost tracking module, has low computational complexity and low hardware requirements. Experiments have shown that it can be deployed on small robots with low cost and low computing power.

[0047] High-precision positioning and mapping:

[0048] By combining the ORB-SLAM module, this method enables high-precision localization and map reconstruction. ORB-SLAM is known for its efficiency and accuracy, making it particularly suitable for real-time applications.

[0049] Supports advanced optimization algorithms:

[0050] This method supports advanced algorithms such as Bundle Adjustment (BA) optimization and loop closure detection. These algorithms help to further reduce errors and improve the accuracy and consistency of localization and mapping.

[0051] Adapting to complex and dynamic environments:

[0052] Overall, this method is specifically designed for complex and dynamic environments. By effectively identifying and removing dynamic objects, it can significantly improve the performance and reliability of SLAM systems in real-world applications.

[0053] High degree of technology integration:

[0054] This method integrates deep learning (YOLOv8), computer vision (multi-view geometry), and SLAM technologies to form a highly integrated solution. This integration not only improves the overall performance of the system but also simplifies its deployment and maintenance.

[0055] In summary, the method proposed in this patent has significant advantages in visual SLAM applications in dynamic environments, especially in improving positioning accuracy, map reconstruction quality, and system robustness. Attached Figure Description

[0056] Figure 1 This invention relates to RGB and depth images captured by a camera when a robot moves rapidly.

[0057] Figure 2 This is a flowchart of a visual SLAM method for dynamic environments based on the YOLOv8 model, according to the present invention.

[0058] Figure 3 This is a flowchart of the sensor data acquisition and processing in this invention;

[0059] Figure 4 This is a flowchart of the multi-view geometry module in this invention;

[0060] Figure 5 This is a schematic diagram illustrating the implementation principle of the ORB-SLAM module in this invention.

[0061] Figure 6 This is a comparison chart of the method proposed in this invention and the results of comparative experiments. Detailed Implementation

[0062] To make the objectives, technical solutions, advantages and benefits of this invention clearer, the invention will be further described below in conjunction with the accompanying drawings and specific implementation examples.

[0063] This invention proposes a visual SLAM method for dynamic environments based on the YOLOv8 model. This method uses an RGB-D camera to acquire RGB image information and corresponding pixel depth information of the robot's surrounding environment, and accurately matches the two in a spatial coordinate system. Subsequently, the captured RGB image is input into an optimized YOLOv8 model to identify and segment dynamic objects in the image. For the segmented image, this study designs two processing strategies: First, a low-cost tracking algorithm is executed based on the segmented image, followed by multi-view geometry methods to further identify dynamic feature points that the YOLOv8 model failed to detect, and then an ORB-SLAM module is introduced to achieve accurate tracking and map construction; second, ORB feature points are directly extracted from the segmented image and input into the ORB-SLAM module. To improve the accuracy of localization and mapping, this study also employs advanced techniques such as graph optimization to refine the generated map and robot position estimation results, effectively reducing errors.

[0064] Figure 2 The block diagram shown illustrates the specific design flow of this invention. This invention discloses a visual SLAM method for dynamic environments based on the YOLOv8 model, with the following specific steps:

[0065] Step 1: RGB-D camera acquires environmental information: The RGB camera acquires RGB images; the depth camera sends out light, and by measuring the time it takes for the light to travel from the camera to the object and back, the distance to the object is calculated using this time difference and the speed of light, thus acquiring a depth image;

[0066] The image processing module corrects lens distortion in the depth image, converts distance data into Z coordinates, calculates point cloud based on camera intrinsics, then translates and rotates the point cloud to the RGB camera coordinate system, reprojects the point cloud onto a 2D image according to the RGB camera intrinsics, and finally matches the coordinates of the 2D point cloud image and the RGB image to obtain an RGB-D image.

[0067] Step 2, Model Training and System Deployment: Convert the collected RGB images to YOLO format, divide the prepared dataset into training, validation, and test sets, enhance the dataset images, input the training set into the improved YOLOv8 model for training, use the validation set to verify the network performance and save the best model parameters and optimizer parameters, and finally predict the segmentation results on the test set.

[0068] The trained segmentation model is deployed on the robot's onboard microcomputer (Raspberry Pi 3B, running Linux), and a SLAM algorithm environment is built in the system to implement the method proposed in this invention.

[0069] Step 3: Image Segmentation Acquisition and Low-Cost Tracking: For the established SLAM system, the ROS system is used for communication between modules. Specifically, an RGB-D camera (Intel RealSense D435i) is used to collect image information about the robot's surroundings, which is then published via ROS's `publish` function. The SLAM system receives this information via the `subscribe` function and inputs it into the segmentation model. The segmentation model segments the prior dynamic objects (i.e., people, cats, dogs, etc.) in the input image, obtaining a mask image.

[0070] The mask image is then processed in parallel in two ways:

[0071] The first method is to directly input the mask image as the segmentation image into the tracking and mapping module of ORB-SLAM. This method can better preserve the common motion objects held or carried by the prior dynamic objects.

[0072] The second approach involves first inputting the mask image into a low-cost tracking module. This module has lower computational complexity than the ORB-SLAM module, thus meeting the real-time robot localization requirements. Subsequently, the mask image is fed into a multi-view geometry module, which pre-extracts ORB feature points from the image and compares and analyzes the current image with several historical images that have the highest overlap, thereby more accurately identifying dynamic objects in the current image and generating a new segmentation image accordingly. Finally, this processed image is input into the ORB-SLAM module described in the first approach. This method can not only identify non-prior dynamic objects but also detect prior dynamic objects that the segmentation model may have missed, while ensuring the real-time operation capability of the entire system.

[0073] Step 4: ORB-SLAM Module: The tracking thread in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking camera motion, and estimating the pose of the current frame. Tracking methods include reference keyframe tracking, constant velocity model tracking, relocalization tracking, and local map tracking.

[0074] Subsequently, if a loop is detected during the robot's movement, the matching relationship between the loop keyframe and the corresponding scene referenced by the loop keyframe will be calculated, and similarity transformation and global BA optimization will be performed. Loop detection can eliminate error accumulation and improve the accuracy and consistency of the map.

[0075] Meanwhile, when a new keyframe is inserted into the local map, the keyframes in the local map and the connection relationships between keyframes are updated. At the same time, the poses and map points of the keyframes in the current local map are optimized by BA (Bundle Adjustment). Finally, redundant keyframes and their corresponding map points are deleted to build the map.

[0076] The resulting map, robot trajectory, direction of movement, and current position are published as topics in the ROS system and displayed through the rviz interface.

[0077] This invention also discloses a visual SLAM device for dynamic environments based on the YOLOv8 model, comprising:

[0078] The YOLOv8 model has been improved, resulting in increased model accuracy and inference speed:

[0079] The YOLOv8 model is a single-stage, real-time object detection framework. Its core idea is to input the entire image into the neural network at once, directly outputting the bounding box coordinates and class probabilities. This approach avoids the multi-stage processing of traditional object detection methods, thus improving detection speed and efficiency. The model can be divided into three parts: the backbone, the neck, and the head. First, the backbone extracts multi-scale feature maps from the original image, providing rich feature information for subsequent detection and segmentation tasks. Next, the neck network's feature fusion technique fuses feature maps of different scales to improve the model's ability to detect multi-scale objects. Finally, the head network uses an anchor-free detection head to directly predict the object's bounding box and class.

[0080] The improved YOLOv8 model's backbone network was replaced with EfficientNet, which improved model efficiency. Furthermore, the SimAM attention mechanism was used to enhance important features and improve model accuracy without increasing computational complexity.

[0081] The improved YOLOv8 model uses GSConv instead of the original convolution in its neck network part. It divides the input feature map into multiple groups and performs convolution operations independently on each group, thereby significantly reducing the amount of computation while maintaining good feature extraction capabilities.

[0082] The improved YOLOv8 model's head network uses the ResNeXt structure to improve the segmentation head, thereby enhancing its information processing capabilities.

[0083] To adapt to the hardware environment of most robots, the improved YOLOv8 model was lightweighted using pruning and other methods, and converted into an ONNX model for deployment on GPUs to achieve real-time inference and image segmentation.

[0084] Furthermore, the principle of the low-cost tracking module is to search for the correspondence between static region feature points in the image frame and local map feature points, find feature matching that can minimize projection error, and thus determine the camera pose. This module will only track the robot's trajectory and will not change the map structure, and can track the robot's pose in real time.

[0085] Furthermore, in the multi-view geometry module, this module will extract the current image p in advance. cur The ORB feature points x are then selected, and several historical images (p) with the highest overlap with the current image are chosen. iWhere i is the image number (here, 5 historical keyframes were selected as historical images after experimentation), then, the keypoint x of each image is calculated from the historical image p. i To the current image p cur The projection key point x′ and the projection depth z proj Simultaneously, the corresponding 3D spatial keypoint X is generated. The angle xXx′ formed by the keypoint x and x′ and the 3D point X is calculated and denoted as alpha. If alpha is greater than 30 degrees, the point is considered dynamic.

[0086] Meanwhile, the multi-view geometry module can utilize depth information to jointly determine dynamic points using another method: the depth corresponding to the key point x′ is z′, which is compared with the projection depth z. proj For comparison, if the value exceeds a threshold, it is considered a dynamic point, and the formula is: Δz = z proj -z′>τ z , where τ z After experimentation, the distance was set to 0.4m, and the dynamic points can be further marked using the two methods mentioned above.

[0087] Furthermore, the ORB-SLAM module extracts ORB feature points from the image, and the tracking thread determines the robot's trajectory and pose: feature point matching is performed by calculating the Hamming distance between pairs of identical feature points in different images. Pairs of feature points with the smallest Hamming distance and satisfying a threshold condition are matched to determine the robot's pose in the map. The local mapping thread then reconstructs the map: the pose relationship between the current frame and historical frames is determined using a 3D-2D pose estimation method.

[0088] Extract the centroid coordinates of feature points in the world coordinate system. and the centroid coordinates in the camera coordinate system Take control points P i C ,P i W These are the coordinates in each dimension of the world coordinate system and the camera coordinate system, respectively. Then, for P... i C ,P i W Remove the centroid to obtain matrix P. C ,P W

[0089] Then for matrix P C Perform SVD decomposition to obtain At this point, four control points are obtained. These four points can be used to represent other points through a weighted combination. Furthermore, according to the imaging formula, the control points in the camera coordinate system can be obtained. Combining this with the projection model, we can obtain the formula:

[0090]

[0091] Where P i C =[u i ,v i ,1] T , Solving the equations yields the rotation matrix R and the displacement matrix t, completing pose estimation. Local mapping is then implemented based on these results.

[0092] The current frame is projected onto the 3D point cloud coordinate system to complete local mapping. Afterwards, the loop closure detection module can be used to correct the estimation error and modify the global map.

[0093] This invention includes five sets of comparative experiments:

[0094] (1) Simulation conditions

[0095] This invention was simulated on an Nvidia DGX GPU Tesla P100-SXM2 (16GB of video memory) with PyTorch 1.11.0 and Python 3.8 as the software environment.

[0096] (2) Simulation content

[0097] Test dataset: TUM's fr3 / walking_xyz dataset;

[0098] Comparative Experiment 1: ORB-SLAM2 performed tracking and mapping on the test dataset;

[0099] Comparative Experiment 2: DynaSLAM performed tracking and mapping on the test dataset;

[0100] Comparative Experiment 3: Tracking and mapping were performed on the test dataset using the original YOLOv8 model combined with the SLAM algorithm proposed in this invention;

[0101] (3) Simulation results

[0102] Table 1 shows the segmentation performance of different experimental methods on the fr3 / walking_xyz dataset.

[0103]

[0104] The simulation experiments provide evaluation metrics for the three experimental methods and the method proposed in this invention on the same dataset for tracking and mapping. The smaller the value, the more accurate the robot's trajectory tracking in the reconstructed map, that is, the higher the accuracy and robustness of the algorithm. Another evaluation metric is the average tracking delay. The smaller the value, the higher the real-time performance of the algorithm. The specific results are shown in Table 1.

[0105] Combination Figure 6 The comparison between the prediction results and the evaluation metrics in Table 1 shows that the visual SLAM method based on the YOLOv8 model proposed in this invention has the highest accuracy and best performance in dynamic environments. This also verifies the effectiveness of introducing deep learning and multi-view geometry to segment dynamic points in dynamic environments. Furthermore, the improved YOLOv8 model described in this invention achieves high accuracy while significantly reducing computational complexity, resulting in an overall FPS of 16.7, which meets the requirement of FPS > 15 in most scenarios.

[0106] In summary, this invention discloses a visual SLAM method for dynamic environments based on the YOLOv8 model. Firstly, it overcomes the shortcomings of traditional SLAM methods in dynamic environments, effectively segmenting dynamic objects and eliminating their negative impact on localization and mapping. Secondly, it improves the YOLOv8 model, significantly enhancing its segmentation accuracy and speed for prior dynamic objects. Finally, this method has low hardware requirements, achieving high real-time performance even with lower-end configurations. Experiments show that this invention can achieve real-time and accurate localization and mapping in complex indoor environments with high-speed moving objects. It can be applied to scenarios such as autonomous robot patrolling, object retrieval, and object transport in high-traffic public places like airports and shopping malls, demonstrating significant application value.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A visual SLAM method based on the YOLOv8 model in a dynamic environment, characterized by: Includes the following steps: Includes the following steps: Step 1: Acquire environmental information using an RGB-D camera: An RGB camera and a depth camera acquire RGB images and 3D point cloud images, respectively; the image processing module processes the 3D point cloud image and performs coordinate matching with the RGB image to obtain an RGB-D image. Step 2: Training the model and deploying the system: The collected RGB images were converted into a YOLO format dataset to train the YOLOv8 model. For the trained segmentation model, it is deployed on the microcomputer on the robot, and the SLAM algorithm environment is built in the system and implemented in the system; Step 3: Image Segmentation Acquisition and Low-Cost Tracking: For the established SLAM system, the segmentation model is used to segment the prior dynamic objects in the input image to obtain a mask image, which is then tracked using the low-cost tracking module. The mask image is then processed in parallel in two ways: The first method involves directly inputting the mask image as a segmentation image into the ORB-SLAM tracking and mapping module; The second method involves first inputting the mask image into a low-cost tracking module. This module searches for the correspondence between static region feature points in the image frame and local map feature points to find feature matches that minimize projection errors, thereby determining the robot's pose in real time. Subsequently, the mask image is sent to a multi-view geometry module, which compares and analyzes the current image with several historical images that have the highest overlap with it, thereby segmenting dynamic objects in the current image. Finally, the processed image is input into the ORB-SLAM module described in Method 1. Step 4, ORB-SLAM Module: The results from the above two methods are fused into the final segmented image, which is then added to the input image sequence of the ORB-SLAM module for processing. The tracking thread in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking camera motion, and estimating the pose of the current frame. Subsequently, if a closed loop is detected during the robot's movement, the matching relationship between the closed loop keyframe and its corresponding reference scene will be calculated, and similarity transformation and global BA optimization will be performed to eliminate error accumulation and improve the accuracy and consistency of the map. Meanwhile, when a new keyframe is inserted into the local map, the keyframes in the local map and the connection relationships between keyframes are updated. The poses and map points of the keyframes in the current local map are also optimized by BA. Finally, redundant keyframes and their corresponding map points are deleted to build the map. The resulting map, robot trajectory, direction of movement, and current position will be published as a topic in the ROS system and displayed through the rviz interface.

2. The visual SLAM method based on the YOLOv8 model in a dynamic environment according to claim 1, characterized in that: In the second method of the mask image processing in step three, the multi-view geometry module will extract the current image in advance. ORB feature points Then select several historical images that have the highest overlap with the current image. ,in These are the image sequence numbers. After experimentation, five historical keyframes were selected as historical images. Then, each keypoint was calculated. From historical images To the current image Key points of projection and projection depth At the same time, corresponding 3D spatial key points are generated. Calculate key points , With three-dimensional points The angle formed , denoted as ,like If the temperature is greater than 30 degrees, the point is considered dynamic.

3. The visual SLAM method based on the YOLOv8 model in a dynamic environment according to claim 1, characterized in that: In the second method of mask image processing in step three, the multi-view geometry module utilizes depth information and uses the first method to jointly determine dynamic points: key points. The corresponding depth is Combine it with the projection depth For comparison, if the value exceeds a threshold, it is considered a dynamic point, and the formula is: ,in After experimentation, the distance was set to 0.4m, and the dynamic points were further marked using the two methods mentioned above.

4. The visual SLAM method based on the YOLOv8 model in a dynamic environment according to claim 1, characterized in that: In step four, the ORB-SLAM module extracts ORB feature points from the image and uses the tracking thread to determine the robot's trajectory and pose. Then, the local mapping thread is responsible for reconstructing the map. After determining the pose relationship between the current frame and historical frames using the 3D-2D pose estimation method, the thread projects the current frame onto the three-dimensional point cloud coordinate system to complete the construction of the local map. Subsequently, the loop closure detection module is introduced to correct the estimation error, thereby correcting and optimizing the global map.

5. The visual SLAM method based on the YOLOv8 model in a dynamic environment according to claim 1, characterized in that: The tracking methods in the tracking thread part of the ORB-SLAM module in step four include reference keyframe tracking, constant velocity model tracking, relocation tracking, and local map tracking.

Images