Visual SLAM method in dynamic environment based on YOLOv8 model

Abstract

The invention relates to a visual SLAM (Simultaneous Localization and Mapping) method in a dynamic environment based on a YOLOv8 model, which comprises the following steps of: acquiring RGB (Red-Green-Blue Depth) image information and depth information of a scene by using an RGB-D (Red-Green-Blue Depth) camera, and matching the RGB image information and the depth information on a space coordinate system; the RGB image is used as input to be sent to the improved YOLOv8 model, and the model recognizes a prior dynamic object in the image and carries out segmentation; then, low-cost tracking is carried out according to the segmented image; secondly, segmenting the image by using a multi-view geometric method; finally, a YOLOv8 model segmentation result and a multi-view geometric segmentation result are fused, the fused result serves as a segmentation image to be input to an ORB (Oriented FAST and Rotated BRIEF)-SLAM module, and the module can achieve high-precision positioning and mapping and supports BA (Bundle Advanced) optimization, loopback detection and other algorithms to reduce errors. According to the visual SLAM method in the dynamic environment based on the YOLOv8 model, robot positioning and map reconstruction in the complex dynamic environment can be efficiently completed.

Description

Technical Field

[0001] The present invention belongs to the field of visual space positioning, and specifically relates to a visual SLAM method in a dynamic environment based on the YOLOv8 model. Background Art

[0002] The SLAM technology is a prerequisite for many robot applications, such as collision-free navigation, augmented reality, etc. The SLAM technology realizes the estimation of the unknown environment map and the joint estimation of the robot's pose in this map by using the sensors carried by the robot itself. The construction of this map makes it possible for the robot to continuously locate in the same environment, effectively avoiding the cumulative drift phenomenon of positioning errors.

[0003] SLAM can be divided into laser SLAM and visual SLAM according to different sensors.

[0004] Many classic SLAM algorithms use lidar as a sensor. The robot emits laser beams through the laser sensor and receives the reflected signals, thereby measuring the distance and angle information between the robot and the objects in the surrounding environment. Grisetti et al. proposed that Gmapping is a 2D laser SLAM algorithm based on particle filtering, which uses probability map technology for environment mapping. However, when the map expands, the number of particles will also increase significantly, so it is generally used for constructing indoor and small outdoor scenes maps. Cartographer launched by Google has improved this situation and can achieve high-precision positioning and map construction in a complex large map environment. It uses Ceres nonlinear optimization and constructs a global map based on submaps. However, the performance of this algorithm in 3D mapping is not ideal. The LOAM algorithm proposed by Ji Zhang can construct a high-precision three-dimensional map. The key idea of this algorithm is to distinguish the two complex problems of simultaneous localization and mapping and solve them through two algorithms. However, the laser SLAM algorithm currently still has disadvantages such as expensive radar equipment, high computational complexity, and poor accuracy in extreme weather.

[0005] With the development of technologies such as image processing and machine learning, in recent years, visual SLAM algorithms have gradually received high attention and research. Visual SLAM algorithms capture continuous image frames through visual sensors such as cameras, extract key feature points from the images, perform feature matching and motion estimation, and thus construct a map of the surrounding environment in real time. LSD-SLAM proposed by Jakob Engel is a direct method SLAM. This method directly uses image pixel values for matching and optimization. This algorithm has a relatively high computational complexity and is suitable for scenarios that require dense mapping. The ORB-SLAM series of algorithms proposed by Raúl Mur-Artal is a feature-point-based SLAM algorithm. It uses ORB features for key-point detection and descriptor generation, and has algorithms such as loop detection and graph optimization, with high robustness and accuracy. However, the accuracy of the above algorithms is not ideal in dynamic environments. The DynaSLAM algorithm proposed by Berta Bescos adds the ability to detect dynamic objects and repair the background. It segments prior dynamic images through the MASK R-CNN image segmentation network and has good results in highly dynamic scenarios. However, the real-time performance of this method is poor, which is not only reflected in the inference latency of MASK R-CNN, but also in the backend SLAM algorithm, and there is also a large room for improvement in accuracy. Therefore, it is extremely important to reduce the inference latency of the image segmentation model, improve the robustness and accuracy of the model, and optimize the backend SLAM algorithm to design a visual SLAM method based on a lightweight and highly real-time segmentation model in dynamic environments.

[0006] The differences between this application and the prior art are as follows:

[0007] Technical comparison with the patent CN116758116A "A VSLAM Method Based on YOLO for Indoor Dynamic Scenarios";

[0008] The structure of the patent CN116758116A is mainly based on the basic YOLO object detection model, and the output result of the image is a rectangular detection box rather than an accurate contour. The core innovation of this patent lies in the improvement of the basic YOLOv8 object segmentation model. While lightweighting the backbone network and convolutional algorithm of this model, the SimAM attention mechanism is introduced, and the ResNeXt structure is used to improve the segmentation head. This enables the object segmentation model to not only more accurately segment the specific contours of prior dynamic objects, but also greatly improve the real-time performance. There are essential differences in the structural design of the deep learning models between the two.

[0009] Patent CN116758116A uses a target detection method combined with a geometric method to eliminate dynamic points before using the SLAM algorithm for tracking. This patent introduces a low-cost tracking module after the target segmentation module. This module efficiently determines the pose of the robot through fast feature point matching, greatly improving the real-time performance of the overall system. Subsequently, a high-precision SLAM algorithm is used to correct the pose of the robot and reconstruct the map. There are essential differences in the design of the tracking methods between the two.

[0010] The SLAM method of patent CN116758116A is only deployed on a personal computer, and the SLAM algorithm is not implemented on the robot. Therefore, the actual application scenario may be limited. The method proposed in this patent has a low algorithm complexity and realizes the construction of the environment on a quadruped robot of the unitree A1 model. The microcomputer carried by this robot is a Raspberry Pi 3B. Therefore, this patent can be deployed on low-cost robots and has a wide range of applications. There are essential differences in the applicable scenarios between the two. Summary of the Invention

[0011] In view of the above problems, the present invention proposes a visual SLAM method based on the YOLOv8 model in a dynamic environment, overcoming the defects of traditional visual SLAM technology to improve the accuracy and real-time performance of dynamic SLAM.

[0012] To achieve the above object, the technical solution adopted by the present invention is:

[0013] A visual SLAM method based on the YOLOv8 model in a dynamic environment, characterized in that it includes the following steps:

[0014] It includes the following steps:

[0015] Step 1: The RGB-D camera obtains environmental information:

[0016] The RGB camera and the depth camera respectively obtain an RGB image and a 3D point cloud image; the image processing module processes the 3D point cloud image and matches its coordinates with the RGB image to obtain an RGB-D image;

[0017] Step 2: Train the model and deploy the system:

[0018] Convert the collected RGB images into a yolo format dataset to train the YOLOv8 model;

[0019] For the trained segmentation model, deploy it on the microcomputer carried by the robot, build a SLAM algorithm environment in the system, and implement it in the system;

[0020] Step 3. Image segmentation acquisition and low-cost tracking: For the established SLAM system, use a segmentation model to segment the prior dynamic objects in the input image to obtain a mask image, and then perform low-cost tracking through a low-cost tracking module:

[0021] There are two parallel processing methods for the obtained mask image:

[0022] The first method is to directly use the mask image as the segmented image and input it into the tracking and mapping module of ORB-SLAM;

[0023] The second method is to first input the mask image into the low-cost tracking module. This module finds the feature matching that can minimize the projection error by searching for the correspondence between the feature points in the static area of the image frame and the feature points in the local map, so as to determine the pose of the robot in real time; Subsequently, the mask image will be sent to a multi-view geometry module, which will compare, analyze and calculate the current image with several historical images with the highest overlap degree, so as to segment the dynamic objects in the current image; Finally, the processed image will be input into the ORB-SLAM module described in Method 1;

[0024] Step 4. ORB-SLAM module: The results of the above two methods are fused into the final segmented image and added to the input image sequence of the ORB-SLAM module for processing by this module:

[0025] The tracking thread part in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking the movement of the camera, and estimating the pose of the current frame;

[0026] Subsequently, if a loop closure is detected during the movement of the robot, the matching relationship between the loop closure key frame and its corresponding reference scene will be calculated, and similarity transformation and global BA optimization will be performed to eliminate error accumulation and improve the accuracy and consistency of the map;

[0027] At the same time, when a new key frame is inserted into the local map, the key frames in the local map and the connection relationship between the key frames will be updated, and the BA optimization will also be performed on the poses of the key frames and the map points in the current local map. Finally, the redundant key frames and their corresponding map points will be deleted to build the map;

[0028] Finally, the established map, the movement trajectory of the robot, the movement direction of the robot, and the current position of the robot are published as topics in the ROS system and displayed through the rviz interface in the ROS system.

[0029] The YOLOv8 model mentioned in step two of the present invention is a single-stage and real-time object detection framework. Its core idea is to input the entire image into the neural network at once and directly output the bounding box coordinates and class probabilities. This approach avoids the multi-stage processing in traditional object detection methods, thereby improving the detection speed and efficiency. The model can be divided into three parts: the backbone network, the neck network, and the head network. First, the backbone network can extract multi-scale feature maps from the original image, providing rich feature information for subsequent detection and segmentation tasks. Then, the feature fusion technology of the neck network fuses feature maps of different scales to improve the model's detection ability for multi-scale objects. Finally, the head network uses an Anchor-Free detection head to directly predict the bounding boxes and classes of the objects.

[0030] For the backbone network part of the improved YOLOv8 model, EfficientNet is used for replacement, which improves the model efficiency. And the SimAM attention mechanism is used to enhance important features without increasing the computational complexity, thus improving the model accuracy.

[0031] For the neck network part of the improved YOLOv8 model, GSConv is used to replace the original convolution. The input feature map is divided into multiple groups, and convolution operations are independently performed on each group, thereby significantly reducing the computational amount while maintaining good feature extraction ability. In the head network part, the ResNeXt structure is used to improve the segmentation head of the segmentation model, enhancing the information processing ability of the segmentation head.

[0032] In addition, in order to adapt to the hardware environment of most robots, the improved YOLOv8 model is lightweighted using methods such as pruning, and is converted into an ONNX model for deployment on the GPU to achieve real-time inference and image segmentation.

[0033] The principle of the low-cost tracking module is to search for the correspondence between the feature points in the static area of the image frame and the feature points in the local map, find the feature matching that can minimize the projection error, thereby determining the pose of the camera. This module only tracks the trajectory of the robot and will not change the map structure, and can track the pose of the robot in real time.

[0034] Furthermore, in the second method of the mask image processing in step three, the multi-view geometry module extracts the ORB feature points x in the current image p cur first, and then selects several historical images (p i where i is the image serial number, and 5 historical key frames are selected as historical images after experiments) that have the highest overlap with the current image. Then, for each key point x, it calculates from the historical image pi to the current image p cur projected key point x′, and projected depth z proj Meanwhile, generate the corresponding 3D space key point X, calculate the angle xXx′ formed by key points x, x′ and 3D point X, denoted as alpha. If alpha is greater than 30 degrees, then this point is considered dynamic.

[0035] Furthermore, in the second method of the mask image processing in step three, the multi-view geometry module uses depth information and uses another method to jointly determine dynamic points: the depth corresponding to key point x′ is z′, compare it with the projected depth z proj for comparison. If it exceeds the threshold, it is a dynamic point. The formula is: Δz = z proj -z′ > τ z where τ z is set to 0.4m through experiments. Through the above two methods, dynamic points can be further marked.

[0036] Furthermore, in step four, the ORB-SLAM module extracts ORB feature points in the image and uses the tracking thread to determine the trajectory and pose of the robot. Then, the local mapping thread is responsible for reconstructing the map. After determining the pose relationship between the current frame and the historical frame through the 3D-2D pose estimation method, project the current frame into the 3D point cloud coordinate system to complete the construction of the local map. Subsequently, the loop detection module is introduced to correct the estimation error, and then the global map is corrected and optimized.

[0037] Furthermore, the tracking methods in the tracking thread part of the ORB-SLAM module in step four include reference key frame tracking, constant velocity model tracking, relocalization tracking, and local map tracking.

[0038] The patent for the above visual SLAM method in a dynamic environment based on the YOLOv8 model, its main advantages can be summarized as the following points:

[0039] Efficient dynamic object recognition and segmentation:

[0040] By using the improved YOLOv8 model, while lightweighting the backbone network and convolution algorithm of the model, introducing the SimAM attention mechanism, and using the ResNeXt structure to improve the segmentation head, it can more efficiently and accurately recognize and segment the prior dynamic objects in the RGB image. This helps to remove unnecessary dynamic elements from the image, thereby improving the accuracy and stability of subsequent SLAM processing.

[0041] Fusion of multi-source segmentation information:

[0042] This method not only relies on the segmentation results of the YOLOv8 model but also combines multi-view geometry methods for image segmentation. The fusion of such multi-source information can further improve the accuracy and robustness of segmentation, especially in complex and dynamic environments.

[0043] Low-cost tracking:

[0044] After identifying and segmenting dynamic objects, this method adopts low-cost tracking technology, which helps reduce the consumption of computing resources while maintaining stable real-time tracking of the prior static environment and improving the real-time performance of the entire algorithm.

[0045] Low-cost deployment:

[0046] Due to the lightweight improvement of the YOLOv8 model and the introduction of a low-cost tracking module, this method has a low computational complexity and low hardware requirements. Experiments have proven that it can be deployed on small robots with low cost and low computing power.

[0047] High-precision positioning and mapping:

[0048] Combined with the ORB-SLAM module, this method can achieve high-precision positioning and map reconstruction. ORB-SLAM is known for its high efficiency and accuracy and is especially suitable for real-time applications.

[0049] Support for advanced optimization algorithms:

[0050] This method supports advanced algorithms such as BA (Bundle Adjustment) optimization and loop closure detection. These algorithms help further reduce errors and improve the accuracy and consistency of positioning and mapping.

[0051] Adaptation to complex dynamic environments:

[0052] Overall, this method is designed for complex and dynamic environments. By effectively identifying and removing dynamic objects, it can significantly improve the performance and reliability of the SLAM system in real-world applications.

[0053] High degree of technology integration:

[0054] This method integrates deep learning (YOLOv8), computer vision (multi-view geometry), and SLAM technologies to form a highly integrated solution. This integration not only improves the overall performance of the system but also simplifies the deployment and maintenance of the system.

[0055] In summary, the method proposed in this patent has significant advantages in visual SLAM applications in dynamic environments, especially in improving positioning accuracy, map reconstruction quality, and system robustness. Description of the Drawings

[0056] Figure 1 The RGB images and depth images collected by the camera when the robot moves quickly in the present invention;

[0057] Figure 2 The flowchart of a visual SLAM method based on the YOLOv8 model in the dynamic environment of the present invention;

[0058] Figure 3 The flowchart of data collection and processing by sensors in the present invention;

[0059] Figure 4 The flowchart of the multi-view geometry module in the present invention;

[0060] Figure 5 The implementation schematic diagram of the ORB-SLAM module in the present invention;

[0061] Figure 6 The result comparison chart of the method proposed in the present invention and the comparative experiment. Detailed implementation manners

[0062] In order to make the purpose, technical solution, and advantages of the present invention clearer, the present invention will be further described below in conjunction with the drawings and specific implementation examples.

[0063] The present invention proposes a visual SLAM method based on the YOLOv8 model in a dynamic environment. This method uses an RGB-D camera to obtain the RGB image information of the robot's surrounding environment and the depth information of the corresponding pixel points, and precisely matches the two in the spatial coordinate system. Subsequently, the captured RGB images are input into the optimized YOLOv8 model to identify and segment the dynamic objects in the images. For the segmented images, two processing strategies are designed in this study: First, a low-cost tracking algorithm is executed based on the segmented images, and then the multi-view geometry method is used to further identify the dynamic feature points that the YOLOv8 model fails to detect. On this basis, the ORB-SLAM module is introduced to achieve precise tracking and map construction; Second, the ORB feature points are directly extracted from the segmented images and input into the ORB-SLAM module. To improve the accuracy of positioning and mapping, this study also adopts advanced technologies such as graph optimization to refine the generated map and the robot position estimation results to effectively reduce errors.

[0064] Figure 2 The block diagram shown presents the specific design process of the present invention. The present invention discloses a visual SLAM method based on the YOLOv8 model in a dynamic environment. The specific steps are as follows:

[0065] Step 1. The RGB-D camera acquires environmental information: The RGB camera acquires RGB images; the depth camera emits light, measures the time it takes for the light to travel from the camera to the object and back, and calculates the distance of the object using this time difference and the speed of light, thereby acquiring depth images;

[0066] The image processing module corrects the lens distortion of the depth image, converts the distance data into Z coordinates, calculates the point cloud based on the camera internal parameters, then translates and rotates the point cloud to the RGB camera coordinate system, projects the point cloud onto a 2D image according to the RGB camera internal parameters, and finally matches the coordinates of the 2D point cloud image and the RGB image to obtain the RGB-D image.

[0067] Step 2. Training the model and deploying the system: The collected RGB images are converted into the yolo format. The prepared dataset is divided into a training set, a validation set, and a test set. After enhancing the dataset images, the training set is input into the improved YOLOv8 model for training. The validation set is used to verify the network performance and save the best model parameters and optimizer parameters. Finally, the segmentation results are predicted on the test set;

[0068] For the trained segmentation model, it is deployed on the microcomputer (Raspberry Pi 3B with Linux system) carried by the robot, and the SLAM algorithm environment is built in the system to implement the method proposed in the present invention.

[0069] Step 3. Obtaining the segmented image and low-cost tracking: For the built SLAM system, the ROS system is used for communication between various modules, that is, the RGB-D camera (Intel RealSense D435i) is used to collect the image information around the robot, which is published through the publish function of ROS. The SLAM system receives it through the subscribe function and inputs it into the segmentation model. The segmentation model segments the prior dynamic objects (i.e., people, cats, dogs, etc.) in the input image to obtain the mask image:

[0070] There are two parallel processing methods for the obtained mask image:

[0071] The first method is to directly use the mask image as the segmented image and input it into the tracking and mapping module of ORB-SLAM. This method can better retain the objects with common motion held or carried by the prior dynamic objects;

[0072] The second method is to first input the mask image into a low-cost tracking module, which has a lower computational complexity compared to the ORB-SLAM module and can thus meet the requirements of real-time robot positioning. Subsequently, the mask image is fed into a multi-view geometry module, which pre-extracts the ORB feature points in the image and compares and operates on the current image with several historical images that have the highest overlap with it, so as to more accurately determine the dynamic objects in the current image and generate a new segmented image accordingly. Finally, this processed image is input into the ORB-SLAM module described in Method 1. This method can not only identify non-priori dynamic objects but also detect priori dynamic objects that the segmentation model may miss, while ensuring the real-time operation ability of the entire system.

[0073] Step 4: ORB-SLAM module: The tracking thread in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking the movement of the camera, and estimating the pose of the current frame. The tracking methods include reference keyframe tracking, constant velocity model tracking, relocalization tracking, and local map tracking.

[0074] Subsequently, during the movement of the robot, if a loop closure is detected, the matching relationship between the loop closure keyframe and the corresponding scene referenced by the loop closure keyframe is calculated, and similarity transformation and global BA optimization are performed. Loop closure detection can eliminate error accumulation and improve the accuracy and consistency of the map.

[0075] At the same time, when a new keyframe is inserted into the local map, the keyframes in the local map and the connection relationships between the keyframes are updated. At the same time, BA (Bundle Adjustment) optimization is performed on the poses of the keyframes and the map points in the current local map. Finally, redundant keyframes and their corresponding map points are deleted to build the map.

[0076] Finally, the established map, the movement trajectory of the robot, the movement direction of the robot, and the current position of the robot are published as topics in the ROS system and displayed through the rviz interface in the ROS system.

[0077] The present invention also discloses a visual SLAM device in a dynamic environment based on the YOLOv8 model, including:

[0078] The YOLOv8 model is improved to improve the model accuracy and inference speed:

[0079] The YOLOv8 model is a single-stage, real-time object detection framework. Its core idea is to input the entire image into the neural network at once and directly output the bounding box coordinates and class probabilities. This approach avoids multi-stage processing in traditional object detection methods, thus improving the detection speed and efficiency. The model can be divided into three parts: the backbone network, the neck network, and the head network. First, the backbone network can extract multi-scale feature maps from the original image, providing rich feature information for subsequent detection and segmentation tasks. Then, the feature fusion technology of the neck network fuses feature maps of different scales to improve the model's detection ability for multi-scale objects. Finally, the head network uses an Anchor-Free detection head to directly predict the bounding boxes and classes of the objects.

[0080] In the backbone network part of the improved YOLOv8 model, EfficientNet is used for replacement, which improves the model efficiency. And the SimAM attention mechanism is used to enhance important features without increasing the computational complexity, thus improving the model accuracy.

[0081] In the neck network part of the improved YOLOv8 model, GSConv is used to replace the original convolution. The input feature map is divided into multiple groups, and convolution operations are independently performed on each group, thus significantly reducing the computational amount while maintaining good feature extraction ability.

[0082] In the head network part of the improved YOLOv8 model, the ResNeXt structure is used to improve the segmentation head of the segmentation model, enhancing the information processing ability of the segmentation head.

[0083] In order to adapt to the hardware environment of most robots, the improved YOLOv8 model is lightweighted using methods such as pruning, and is converted into an ONNX model for deployment on the GPU to achieve real-time inference and image segmentation.

[0084] Furthermore, the principle of the low-cost tracking module is to search for the correspondence between the feature points in the static region of the image frame and the feature points in the local map, find the feature matching that can minimize the projection error, thereby determining the pose of the camera. This module only tracks the trajectory of the robot and will not change the map structure, and can track the pose of the robot in real time.

[0085] Furthermore, in the multi-view geometry module, this module will extract the ORB feature points x in the current image p cur first, and then select several historical images with the highest overlap with the current image (p i, where i is the serial number of the image, and 5 historical key frames are selected as historical images after experiments. Then, calculate the projected key point x' of each key point x from the historical image p i to the current image p cur , and the projected depth z proj . At the same time, generate the corresponding 3D space key point X. Calculate the included angle xXx' formed by the key points x, x' and the 3D point X, denoted as alpha. If alpha is greater than 30 degrees, then this point is considered dynamic.

[0086] At the same time, the multi-view geometry module can use the depth information and use another method to jointly judge the dynamic points: the depth corresponding to the key point x' is z'. Compare it with the projected depth z proj . If it exceeds the threshold, it is a dynamic point. The formula is: Δz = z proj -z' > τ z , where τ z is set to 0.4m through experiments. The dynamic points can be further marked through the above two methods.

[0087] Furthermore, the ORB-SLAM module extracts the ORB feature points in the image, and the tracking thread determines the trajectory and pose of the robot: by solving the Hamming distance between the same feature points in different images pairwise for feature point matching, a pair of feature points with the smallest Hamming distance and meeting the threshold conditions can be matched to determine the pose of the robot in the map. The local mapping thread reconstructs the map: by using the 3D-2D pose estimation method to determine the pose relationship between the current frame and the historical frame:

[0088] Extract the centroid coordinates of the feature points in the world coordinate system and the centroid coordinates in the camera coordinate system Take the control points P i C , P i W are the coordinates of each dimension in the world coordinate system and the camera coordinate system respectively. Then, subtract the centroid from P i C , P i W respectively to obtain the matrices P C , P W

[0089] Then perform SVD decomposition on the matrix P C to obtain At this time, four control points are obtained. These four points can represent other points through weighted combination. According to the imaging formula, the control points in the camera coordinate system can be obtained Combined with the projection model, the formula can be obtained:

[0090]

[0091] where P i C = [u i , v i , 1] T , After solving the equation, the rotation matrix R and the displacement matrix t can be obtained to complete the pose estimation. Then, local mapping is realized according to this result.

[0092] Local mapping is completed by projecting the current frame into the 3D point cloud coordinate system. Then, the loop detection module can be used to correct the estimation error and revise the global map.

[0093] The present invention has designed a total of five groups of comparative experiments:

[0094] (1) Simulation conditions

[0095] The simulation of the present invention is completed on the GPU Tesla P100 - SXM2 (with a video memory of 16GB) of Nvidia DGX, and the software environment is Pytorch 1.11.0 and Python 3.8.

[0096] (2) Simulation content

[0097] Test dataset: fr3 / walking_xyz dataset of TUM;

[0098] Comparative experiment 1: ORB - SLAM2 performs tracking and mapping on the test dataset;

[0099] Comparative experiment 2: DynaSLAM performs tracking and mapping on the test dataset;

[0100] Comparative experiment 3: The original YOLOv8 model combined with the SLAM algorithm proposed by the present invention performs tracking and mapping on the test dataset;

[0101] (3) Simulation results

[0102] Table 1 Segmentation performance of different experimental methods under the fr3 / walking_xyz dataset

[0103]

[0104] In the simulation experiments, the evaluation metrics - APE (Absolute Pose Error) for tracking and mapping of the above three experimental methods and the method proposed in the present invention on the same dataset are given. The smaller the value, the more accurate the trajectory tracking of the robot in the reconstructed map, that is, the higher the accuracy and robustness of the algorithm; and another evaluation metric - average tracking delay, the smaller the value, the higher the real-time performance of the algorithm. The specific results are shown in Table 1.

[0105] Combined with Figure 6 the prediction results and the comparison results of the evaluation metrics in Table 1, it can be seen that the visual SLAM method based on the YOLOv8 model proposed in the present invention has the highest accuracy and the best performance, and at the same time verifies the effectiveness of the scheme of introducing deep learning and multi-view geometry to segment dynamic points in a dynamic environment. In addition, the improved YOLOv8 model described in the present invention has a greatly reduced computational complexity while having a high accuracy, making the overall algorithm fps reach 16.7, meeting the requirement of fps>15 in most scenarios.

[0106] In summary, a visual SLAM method based on the YOLOv8 model disclosed in the present invention, first, overcomes the defects of traditional SLAM methods in a dynamic environment, can segment dynamic objects, and eliminate their negative impacts on positioning and mapping; second, improves the YOLOv8 model, making its segmentation accuracy and speed for prior dynamic objects significantly improved; finally, this method has low requirements for hardware and has high real-time performance under low configurations. Through experiments, the present invention can perform real-time and accurate positioning and mapping in an indoor environment with complex and relatively fast-moving dynamic objects, and can be applied to scenarios such as robot autonomous patrol, object searching, and object transportation in public places with high traffic such as airports and shopping malls, and has high application value.

[0107] The above is only a preferred embodiment of the present invention, and it is not any other form of limitation to the present invention. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope protected by the present invention.

Claims

1. A visual SLAM method in a dynamic environment based on the YOLOv8 model, characterized by: The following steps are involved: The following steps are involved: Step 1: RGB-D camera obtains environmental information: The RGB camera and the depth camera obtain RGB images and 3D point cloud images respectively; the image processing module processes the 3D point cloud image and matches its coordinates with the RGB image to obtain an RGB-D image; Step 2: Train the model and deploy the system: Convert the collected RGB images into YOLO format datasets to train the YOLOv8 model; For the trained segmentation model, deploy it on the robot's onboard microcomputer, build a SLAM algorithm environment in the system, and implement it in the system; Step 3: Segmentation image acquisition and low-cost tracking: For the built SLAM system, use the segmentation model to segment the prior dynamic objects in the input image, obtain the mask image, and then perform low-cost tracking through the low-cost tracking module: There are two ways to process the mask image in parallel: The first method is to directly input the mask image as a segmented image into the tracking and mapping module of ORB-SLAM; The second method is to first input the mask image into a low-cost tracking module, which searches for the correspondence between the feature points in the static area of ​​the image frame and the feature points in the local map to find the feature match that can minimize the projection error, thereby determining the robot's position in real time; then, the mask image will be sent to a multi-view geometry module, which will compare and analyze and calculate the current image with several historical images with the highest overlap, thereby segmenting the dynamic objects in the current image; finally, the processed image will be input into the ORB-SLAM module described in method one; Step 4, ORB-SLAM module: The results of the above two methods are fused into the final segmented image, added to the input image sequence of the ORB-SLAM module, and processed by the module: The tracking thread in the ORB-SLAM module is responsible for processing the input image sequence, extracting ORB features, tracking the camera's motion, and estimating the pose of the current frame; Subsequently, if a closed loop is detected during the robot's motion, the matching relationship between the closed-loop keyframe and the corresponding scene it refers to is calculated, and similarity transformation and global BA optimization are performed to eliminate error accumulation and improve the accuracy and consistency of the map; At the same time, when a new keyframe is inserted into the local map, the keyframes in the local map and the connection relationship between the keyframes will be updated, and the poses and map points of the keyframes in the current local map will be optimized by BA. Finally, the redundant keyframes and their corresponding map points will be deleted to build the map. Finally, the established map, the robot's motion trajectory, the robot's motion direction, and the robot's current position are published as topics of the ROS system and displayed through the rviz interface in the ROS system.

2. The visual SLAM method based on the YOLOv8 model dynamic environment according to claim 1, characterized in that: The multi-view geometry module in the second method of the mask image processing method in step 3 will extract the current image p in advance. cur Then select several historical images with the highest overlap with the current image (p i , where i is the sequence number of the image. After the experiment, 5 historical key frames are selected as historical images. Then, each key point x is calculated from the historical image p i To the current image p cur The projected key point x′ and the projected depth z proj , and generate the corresponding three-dimensional key point X at the same time, calculate the angle xXx′ formed by the key point x, x′ and the three-dimensional point X, recorded as alpha, if alpha is greater than 30 degrees, the point is considered dynamic.

3. according to claim 1 based on the visual SLAM method under the YOLOv8 model dynamic environment, it is characterized in that: In the second method of the step 3 mask image processing method, the multi-view geometry module uses depth information and uses another method to jointly determine the dynamic point: the depth corresponding to the key point x' is z', which is compared with the projection depth z proj For comparison, if it exceeds the threshold, it is a dynamic point, and the formula is: Δz = z proj -z′>τ z , where τ z After experiments, it is set to 0.4m. The dynamic points can be further marked by the above two methods.

4. according to claim 1 based on the visual SLAM method under the YOLOv8 model dynamic environment, it is characterized in that: In step 4, the ORB-SLAM module extracts ORB feature points from the image and uses the tracking thread to determine the trajectory and posture of the robot. Then, the local mapping thread is responsible for reconstructing the map. After determining the posture relationship between the current frame and the historical frame through the 3D-2D posture estimation method, the current frame is projected into the three-dimensional point cloud coordinate system to complete the construction of the local map. Subsequently, the loop detection module is introduced to correct the estimation error, thereby correcting and optimizing the global map.

5. according to claim 1 based on the visual SLAM method under the YOLOv8 model dynamic environment, it is characterized in that: The tracking method of the tracking thread part in the ORB-SLAM module in step 4 includes reference key frame tracking, constant speed model tracking, relocation tracking and local map tracking.

Images