Deep learning based multi-mobile robot visual slam method

Abstract

This invention discloses a multi-robot visual simultaneous localization and mapping (SLAM) method based on deep learning. First, multiple individual wheeled mobile robots construct an environment map using an improved visual SLAM algorithm based on deep learning. Second, a map fusion algorithm based on Roust ICP is used to fuse the dense point cloud maps constructed by the multiple mobile robots, resulting in a complete and accurate environment map. The environment map constructed by the multi-robot visual SLAM method proposed in this invention is relatively accurate and can effectively remove interference from dynamic objects in the environment.

Description

TECHNICAL FIELD

[0001] The present application relates to a visual SLAM technology optimized using a target detection algorithm, belonging to the field of robot perception technology. BACKGROUND

[0002] As one of the important methods for mobile robots to perceive autonomously, visual SLAM technology has gradually shown its importance in recent years. This technology uses cameras to capture environmental images and analyzes these image data to infer the robot's position in the environment and construct a detailed three-dimensional map of the surrounding environment. The main challenge of visual SLAM is to effectively extract continuous visual information to obtain accurate spatial positioning data and accurately reconstruct the environment map. Its development has a profound impact on the field of robotics, especially in the fields of autonomous driving, unmanned aerial vehicle navigation, and indoor robots. This technology enables robots to navigate effectively in indoor environments without GPS signals or complex urban landscapes. Compared to other navigation technologies, visual SLAM has the advantages of high cost-effectiveness and strong environmental adaptability, as it uses relatively inexpensive and widespread cameras rather than relying on devices such as laser radar (LIDAR), making it an economically practical choice. However, early visual SLAM still faces many challenges, such as low feature extraction efficiency, sensitivity to dynamic objects, and high error in map generation. With the continuous innovation of SLAM technology and the development of deep learning technology, these challenges have gradually been overcome.

[0003] Deep learning has the potential to improve the accuracy and robustness of visual SLAM systems. Through neural networks, machines can more effectively extract features and maintain stable performance under a wider range of environmental conditions. The important advantage that deep learning technology brings to visual SLAM is its high adaptability to various environmental conditions. It can work stably under different lighting conditions, and even effectively operate in environments with insufficient or excessive light. In addition, it can also adapt to visual disturbances caused by weather changes and scene dynamics, such as rain, snow, or the presence of moving objects. This adaptability makes visual SLAM more reliable in practical applications, especially in situations where environmental changes are frequent or difficult to predict.

[0004] However, most current research focuses on single robot visual SLAM systems, while research on multi-mobile robot systems is relatively scarce. Multi-mobile robot systems, especially in terms of collaboration, are of great significance in advancing robotics. These systems exhibit higher efficiency and adaptability when performing complex tasks, especially in scenarios requiring wide coverage or multi-angle observation. For example, multi-robot systems can play an important role in disaster response, environmental monitoring, or large-scale industrial applications. However, to achieve this goal, robots need to effectively share information and coordinate actions, which is the core of deep learning-based multi-mobile robot visual SLAM research.

[0005] In addition, the application of deep learning in visual SLAM also brings new challenges. For example, the dependence of deep learning models on data, the demand for computing resources, and how to effectively combine deep learning techniques with traditional SLAM algorithms. In the face of these challenges, this patent aims to explore and develop a new deep learning-based multi-mobile robot visual SLAM system to improve its positioning accuracy and map construction capabilities in various environments.

[0006] In summary, the research on deep learning-based multi-mobile robot visual SLAM algorithms not only expands existing technology but also provides a glimpse into the possible future direction of intelligent systems. The success of this research will have a profound impact on the practical application of mobile robots, providing more effective tools for solving complex real-world problems. SUMMARY

[0007] TECHNICAL PROBLEM

[0008] To address the problem of traditional visual SLAM algorithms being heavily disturbed in dynamic environments and being unable to construct dense point cloud maps, a deep learning-based multi-mobile robot visual SLAM method is proposed. The method uses a target detection network to eliminate the interference of dynamic objects in the environment, adds a dense point cloud map construction module, and performs multi-map fusion through a map fusion algorithm to obtain a complete visual SLAM method.

[0009] TECHNICAL SCHEME

[0010] To achieve the purpose of the present application, the technical scheme adopted by the present application is: a deep learning-based multi-mobile robot visual SLAM method, the steps of which include:

[0011] First, the mobile robot carries a camera to collect images into the tracking (Tracking) thread and the target detection thread to detect dynamic objects and eliminate dynamic feature points.

[0012] Secondly, the tracking thread generates key frames and sends them to the local mapping thread and the dense point cloud map construction module, respectively, to optimize the local map and construct the dense point cloud map.

[0013] Thirdly, the map is optimized and the repeated point cloud points are removed to obtain a complete map of the environment constructed by a single mobile robot.

[0014] Fourthly, the environment maps constructed by multiple mobile robots are uploaded to a server, and a map fusion algorithm based on the Robust ICP algorithm is used for fusion to obtain a more complete and accurate environment map.

[0015] Further, in the first step, an Intel Realsense D435i camera is used for image acquisition, which has a resolution of 1920 pixels x 1080 pixels and a maximum frame rate of 90 FPS. A fixed bracket is used to install the camera on the mobile robot.

[0016] Further, in the first step, the Tracking thread and the target detection thread respectively acquire the images collected by the camera for feature point extraction and feature point screening. In the Tracking thread, the method extracts ORB (Oriented FAST and Rotated BRIEF) feature points. ORB features are composed of Fast key points and Brief descriptors. Fast is a corner point that mainly detects places where local pixel gray changes are obvious, and the detection speed is very fast. The main idea is to determine whether it can be a Fast corner point by detecting the difference between the pixel and the surrounding pixels. Brief descriptor is a binary type descriptor commonly used in feature point method SLAM. Its main advantage is the simple storage format and fast calculation speed. In practical applications, Brief descriptor selects two pixel points p and q near the corner point at random and compares them. If the gray value of p is greater than q, the point is recorded as 1; otherwise, it is recorded as 0. In order to obtain sufficient information, usually 128 such pixel points are selected. In the target detection thread, the method uses YOLOv8s as the basic target detection model to realize the detection and screening of dynamic objects in the environment. On the basis of YOLOv8s model, the SlimNeck structure composed of Coordinate Attention mechanism and GSConv module is added. Coordinate Attention is a special attention mechanism, and its design purpose is to embed the exact spatial position information in the neural network architecture, so as to effectively capture the correlation between features and the dependence relationship across a long distance. First, the design of Coordinate Attention aims to consider both position information and direction-related information, avoiding the simple compression of spatial information to the channel dimension. In order to achieve this goal, it adopts average pooling operations in the width and height dimensions when processing feature maps. Specifically, the input feature map with size C·H·W is pooled to generate two feature maps with size C·H·1 and C·1·W:

[0017]

[0018] where C, H, W are the length, height and width of the image respectively, x c (i,j) is the value of the input feature map at coordinate (i,j), and H

[0019] Next, the two pooled feature maps are further processed. The C·1·W feature map is subjected to a specific transpose operation to facilitate subsequent processing steps. After that, the transformed feature map is spliced with the original C·H·1 feature map. After splicing, a 1·1 size convolution kernel is used to process the spliced feature map. Finally, the convolutional feature map is activated by the Sigmoid activation function.

[0020] f = δ (F1 ([z h ,z w ]))

[0021] wherein z h , z w are height and width feature maps, F1 is a 1·1 size convolution function, and δ is a Sigmoid activation function. In the final stage of the Coordinate Attention mechanism, a segmentation operation is performed to finely divide the activated output function f into two parts, f h and f w . Subsequently, a 1·1 size convolution kernel is applied to each of the two parts to perform dimensionality raising operation. Finally, the Sigmoid function activation is performed to obtain the attention vectors g h and g w :

[0022]

[0023] The output formula of the Coordinate Attention module can be expressed as:

[0024]

[0025] wherein x c (i,j) is the value of the input feature map at coordinate (i,j), y c (i,j) is the value of the output feature map at coordinate (i,j), a is the attention weight in the height direction, and b is the attention weight in the width direction.

[0026] After obtaining the improved target detection algorithm, the method selects the ORB-SLAM3 system as the basis for further improvement. In this patent, the front-end visual odometry module of ORB-SLAM3 is significantly improved, mainly embodied in the integration of an independent target detection thread, which is complementary to the tracking thread, and its activation or not is controlled by the tracking thread. The specific implementation process is as follows: the input RGB image is first sent to the feature extraction module and the optimized YOLOv8s target detection thread. In this process, the target detection thread is responsible for identifying dynamic objects in the image and marking the feature points on these objects. Then, the system will perform matching and corresponding operations between the image and the feature points to identify the feature points belonging to high dynamic, low dynamic or static objects. In order to optimize system performance, feature points from high dynamic objects will be excluded from the feature matching process, while other feature points will be retained.

[0027] After that, considering that the motion trajectory of high dynamic objects is difficult to predict, and these objects may occupy a large area of the image, which may cause the ORB-SLAM3 system to experience tracking loss, thereby affecting the overall accuracy. To solve this problem, this method introduces trajectory tracking and feature point judgment logic. The Byte-track tracker is used to track all objects in the image, and each object is assigned a unique number. In the logical judgment of feature points, if the detected label is a human or other mobile robot, and the detection box area of these objects exceeds half of the total image area, even if these objects belong to the high dynamic category, the system will still retain their feature points. This strategy aims to avoid too frequent tracking loss events, reduce the number of inactive maps generated, and thus reduce the consumption of computing resources.

[0028] And in this method, the dense map construction is designed as an independent module, whose activation or not depends on the setting of the input parameters. This module is mainly composed of two parallel threads, which are responsible for the construction of dense point cloud maps and semantic segmentation respectively. The input data of the module comes from the key frame images selected by the ORB-SLAM3 system. In the dense point cloud map construction thread, the first step is to perform data fusion using the RGB information and depth information in the key frame, realize the projection from two-dimensional image points to three-dimensional space points, and generate initial point cloud data. Then, the system converts these preliminary constructed point cloud data to the global coordinate system, integrates and splices the point cloud data, and the process is described as:

[0029]

[0030] Finally, the method proposes a map fusion strategy aimed at integrating the dense point cloud environment maps constructed by multiple mobile robots, generating a more accurate and higher density point cloud comprehensive map. The map fusion process mainly consists of three key steps: point cloud registration, point cloud fusion and fusion map optimization. Point cloud registration is the first step of the fusion process, aiming to accurately align the point cloud data from different robots into a unified coordinate system. Next, the point cloud fusion step is committed to merging these aligned point clouds into a continuous and comprehensive environment representation. Finally, in the fusion map optimization stage, various algorithms are used to further improve the accuracy and quality of the fusion map, ensuring the practicality and reliability of the map.

[0031] The beneficial effects of the present application are that the proposed visual SLAM system has better robustness in complex and dynamic environments, and can construct a high-precision dense point cloud map without losing real-time performance.

[0032] Beneficial effects:

[0033] By using the improved YOLOv8 model for dynamic object detection and feature point elimination, the interference of dynamic objects on positioning is reduced, and the positioning accuracy of the system in dynamic environments is improved.

[0034] Based on ORB-SLAM3, multiple parallel threads are added, including target detection threads and dense point cloud map construction modules, so that the system can more efficiently process data in complex environments and enhance the robustness of the system.

[0035] The lightweight YOLOv8 model and GSConv structure are used to optimize the target detection and feature extraction process, improving the real-time performance of the system and making it suitable for resource-constrained mobile devices.

[0036] The dense point cloud map construction module is introduced, which uses deep learning algorithms to generate dense point cloud data and perform optimization and noise elimination to construct a high-precision environment map.

[0037] The Robust ICP algorithm is used to realize the fusion of environment maps constructed by multiple mobile robots, generating a more complete and accurate environment map suitable for multi-robot collaborative navigation and task execution. The system can adaptively adjust the feature points and pose estimation according to changes in the dynamic environment, improving the adaptability of the system in different environments.

[0038] The pose detection method of the present application can achieve an average trajectory error of 0.0163 in multiple groups of experiments on multiple sequences in the TUM dataset, and the accuracy is improved by 95% compared with mainstream visual SLAM algorithms in dynamic environments. In addition, the detection speed of the overall SLAM system is improved from 37FPS to 45FPS, with a large improvement. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the visual SLAM system structure;

[0040] Figure 2 Here is a diagram of the target detection algorithm model structure;

[0041] Figure 3 This is a schematic diagram of visual SLAM for multiple mobile robots. Detailed Implementation

[0042] Example:

[0043] like Figure 1 As shown, the overall process of a deep learning-based visual SLAM method for multiple mobile robots is as follows: first, target images are acquired; then, a target detection algorithm is used to remove dynamic objects; next, keyframes are generated to construct a dense point cloud map; finally, the dense point cloud maps of multiple mobile robots are fused to generate a more accurate and complete dense point cloud map. In the image acquisition stage, a Hikvision Intel Realsense D435i camera is used for image acquisition. This camera has a resolution of 1920 pixels × 1080 pixels and a maximum frame rate of 90 FPS. A fixed bracket is used to mount the camera on the mobile robot.

[0044] like Figure 2 As shown, in the object detection algorithm, after the image passes through the Coordinate Attention mechanism of the backbone network, a feature map with richer information and more reasonable weight allocation is obtained. Then, the feature map undergoes further feature fusion through a SlimNeck structure composed of GSConv, obtaining richer features and more detailed and correlated information. Finally, a more accurate detection result is obtained in the Head part. This object detection algorithm is integrated into... Figure 1 In the feature point selection section, feature points generated by the SLAM system for dynamic objects are removed, thus eliminating interference from dynamic information during camera pose calculation. Then, keyframes generated by the SLAM system without dynamic interference are transmitted to the local mapping thread and the dense point cloud map construction module, respectively. In the local mapping thread, map points are generated and local map optimization is performed before being passed to the loop closure detection thread for error correction. The dense point cloud map construction module performs semantic segmentation and point cloud reconstruction on the input keyframes. By segmenting low-dynamic objects and fusing semantic labels with the dense point cloud, dynamic interference is further reduced. Finally, point cloud stitching and voxel-based raster filtering for duplicate point removal are performed to obtain a complete and accurate dense point cloud map, as well as camera pose and trajectory information with smaller errors.

[0045] like Figure 3As shown, multiple mobile robots perform synchronized SLAM in the environment, and transmit the obtained dense point cloud map to the server side for fusion. During the fusion process, first, the point cloud registration is performed using the Robust ICP algorithm, and each point cloud dataset is aligned to a unified coordinate system. This operation ensures that the three-dimensional coordinates of all point clouds are in the same coordinate space, thereby laying the foundation for finding the most suitable Euclidean transformation. The purpose of this Euclidean transformation is to minimize the error in spatial position and direction between the two point cloud datasets, laying the foundation for further point cloud data fusion steps. After the point cloud registration by the Robust-ICP algorithm, it is necessary to fuse multiple point cloud data. Specifically, first, the entire point cloud space is divided into a regular, fixed-size cubic lattice, and each lattice is called a voxel. Secondly, the in-voxel point simplification is performed. For each non-empty voxel, i.e., a voxel containing at least one point, the filter simplifies all points in the voxel to a single point. The position of this representative point can be the centroid, geometric center, or other specifically calculated points of all points in the voxel. Finally, these representative points constitute the new point cloud after downsampling. The purpose of downsampling is to reduce the amount of calculation and storage requirements, while simplifying the processing of the point cloud without significantly losing the detailed information of the point cloud. Finally, the fused map optimization is performed, and the quality of the fused point cloud map can be effectively improved through statistical outlier removal, radius outlier removal, and Gaussian filter smoothing.

[0046] The above implementation is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. It is worth noting that any improvement to the technical solution on the technical idea of the present application belongs to the protection scope of the present application.

Claims

1. A deep learning-based visual SLAM method for multiple mobile robots, characterized in that, Includes the following steps: The first step involves the mobile robot carrying a camera to capture images, which are then fed into the Tracking thread and the target detection thread for the detection of dynamic objects and the removal of dynamic feature points. In the first step, the Tracking thread and the object detection thread respectively acquire the images captured by the camera to extract and filter feature points. In the Tracking thread, ORB, or Oriented Fast and Rotated BRIEF feature points, are extracted. ORB features are composed of Fast keypoints and Brief descriptors. In the object detection thread, YOLOv8s is used as the basic object detection model to detect and filter dynamic objects in the environment. The SlimNeck structure, consisting of the Coordinate Attention mechanism and the GSConv module, is added to the YOLOv8s model. The second step involves the tracking thread generating keyframes, which are then passed to the Local Mapping thread and the dense point cloud map construction module to optimize the local map construction and construct the dense point cloud map, respectively. The target detection algorithm is integrated into the feature point selection part to remove feature points generated by the SLAM system for dynamic objects, thereby eliminating the interference of dynamic information when calculating camera pose. Then, the key frames generated by the SLAM system without dynamic information interference are transmitted to the local mapping thread and the dense point cloud map construction module respectively. In the local mapping thread, map points are generated and local map optimization is performed before being passed to the loop closure detection thread for error correction. The dense point cloud map construction module performs semantic segmentation and point cloud restoration on the input key frames. By segmenting low-dynamic objects and fusing semantic labels with dense point clouds, dynamic information interference is further reduced. The third step is to optimize the map and remove duplicate point cloud points to obtain a complete map of the environment constructed by a single mobile robot. The fourth step involves uploading the environmental maps built by multiple mobile robots to the server and fusing them using a map fusion algorithm based on the Robust ICP algorithm to obtain a more complete and accurate environmental map.

2. The deep learning-based visual SLAM method for multiple mobile robots according to claim 1, characterized in that, The first step, image acquisition, dynamic object detection, and dynamic feature point removal, specifically includes the following steps: Each mobile robot is equipped with a high-resolution camera to capture images of the surrounding environment in real time. The captured image data is then fed into the tracking thread and the target detection thread. Dynamic object detection is performed using an improved YOLOv8 model to mark dynamic object regions in the image, and image feature points are extracted using the ORB feature extraction algorithm. Feature points located within the area of ​​dynamic objects are removed, and only static feature points are retained, thereby reducing the positioning error caused by dynamic objects.

3. The deep learning-based visual SLAM method for multiple mobile robots according to claim 1, characterized in that, The second step, keyframe generation, local mapping, and dense point cloud map construction, includes the following specific steps: The tracking thread generates keyframes based on image data during the robot's movement, with selection criteria including the motion amplitude of the image and scene changes. Keyframes are passed to the local mapping thread to construct a local map, and the Bundle Adjustment (BA) algorithm is used to optimize the local map to ensure its accuracy and consistency. The keyframes are simultaneously fed into the dense point cloud map construction module, which uses deep learning algorithms to generate dense point cloud data, uses a depth estimation network to extract depth information from the image, and combines the position information of the keyframes to generate a dense point cloud map.

4. The deep learning-based visual SLAM method for multiple mobile robots according to claim 1, characterized in that, The third step, global map optimization and duplicate point cloud point removal, includes the following specific steps: The global optimization algorithm is used to optimize the entire map, ensuring the consistency of the global map, and taking into account the relative poses and mutual observation relationships between robots to improve the accuracy of the global map. The dense point cloud generated from multiple keyframes is processed, and the iterative nearest point algorithm is used to register the point cloud, remove duplicate and redundant points, and improve the quality of the point cloud. By integrating optimized and deduplicated point cloud data, a complete map of the environment for a single mobile robot is constructed, including dense point cloud data and corresponding location information, providing reliable data support for navigation and path planning.

5. The deep learning-based visual SLAM method for multiple mobile robots according to claim 1, characterized in that, The fourth step, map uploading, map fusion, and complete map construction, includes the following specific steps: Each mobile robot uploads its completed map data to the server. The server receives map data from multiple mobile robots, stores and processes it uniformly. The complete map data includes dense point clouds and location information. The Robust ICP (Iterative Closest Point) algorithm is used for map fusion. Based on the relative pose of the mobile robot, the uploaded map is initially registered, and a robust processing mechanism is introduced for multiple iterations to optimize and reduce registration errors, thereby ensuring accurate map fusion. After fusion processing, the server generates a complete and accurate environmental map. The complete map includes the fusion result of local maps built by multiple mobile robots, which has high accuracy and high consistency. It can be used for multi-mobile robot collaborative navigation, path planning and task execution application scenarios.

Images