A mobile robot based on dynamic real-time visual SLAM method

Abstract

The present invention provides a mobile robot based on a dynamic real-time visual SLAM method. The mobile robot provides an image segmentation module and a visual SLAM module, and inputs environmental image data acquired by an image acquisition module into the image segmentation module and the visual SLAM module respectively, and processes image segmentation processing and posture estimation operations in parallel. By removing abnormal points in a Lucas-Kanade sparse optical flow module and a secondary removal module respectively, a dynamic image frame output by the image segmentation module participates in a second abnormal point removal operation in the secondary removal module, which not only improves the recognition accuracy, but also greatly improves the processing efficiency on the basis of ensuring the accuracy of posture estimation, ensures that the technical solution of the present invention can perform real-time posture estimation in a dynamic environment, and enables the mobile robot to be more flexibly applicable to different application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, in particular to a mobile robot based on a dynamic real-time visual SLAM method. Background Art

[0002] In the existing mobile robot technology, in order to build a high-precision map of the working environment, the simultaneous localization and mapping (SLAM) technology is usually adopted. By collecting external sensor data, the robot's posture state is estimated and the surrounding environment is mapped. However, the traditional dynamic visual SLAM algorithm can be roughly divided into two categories, one is based on geometric detection (multi-view geometry, geometric clustering, optical flow, etc.), and the other is based on image segmentation (Mask R-CNN, SegNet, etc.). Although the dynamic visual SLAM algorithm based on geometric detection has advantages in operation efficiency, it cannot accurately separate dynamic objects in high-dynamic scenes, and the positioning accuracy is not ideal; the dynamic visual SLAM algorithm based on deep learning is the mainstream direction of the current dynamic SLAM algorithm. It is based on image segmentation algorithm and supplemented by geometric detection. However, since the image segmentation network itself is very time-consuming, it cannot complete the robot's posture estimation in real time in a dynamic environment. Therefore, it cannot be applied in scenes with complex environments and real-time positioning, which limits the application of mobile robots in real life. Summary of the invention

[0003] In order to solve the problem that the existing mobile robots based on SLAM technology cannot accurately and real-timely complete the robot's posture estimation in a dynamic environment, the present invention provides a mobile robot based on a dynamic real-time visual SLAM method, which can not only perceive and understand the unknown environment, but also perform real-time posture estimation in a dynamic environment, so that the mobile robot can be more flexibly applied to different application scenarios.

[0004] The technical solution of the present invention is as follows: a mobile robot based on a dynamic real-time visual SLAM method, comprising: an image acquisition module, a dynamic visual SLAM module and a controller, characterized in that:

[0005] The dynamic visual SLAM module includes: a visual SLAM module and an image segmentation module;

[0006] The visual SLAM module and the image segmentation module are respectively connected to the image acquisition module for communication;

[0007] The controller is connected to the dynamic vision SLAM module via serial port communication;

[0008] After the image acquisition module acquires the RGB image information and the depth information of the environment, the RGB image information and the depth information are transmitted in the form of an RGB image sequence and a depth image sequence, which are recorded as data to be processed; the image acquisition module sends the data to be processed to the visual SLAM module and the image segmentation module respectively;

[0009] The data to be processed is sent to the visual SLAM module for recognition operation; the processing operation flow of the data to be processed in the visual SLAM module is carried out in parallel with the operation flow of the data to be processed in the image segmentation module;

[0010] The visual SLAM module includes: a Lucas-Kanade sparse optical flow module, a secondary elimination module, and a feature matching pose estimation module connected in sequence;

[0011] In the image segmentation module, the RGB image frame in the data to be processed is subjected to target detection through a semantic segmentation network, and after obtaining a semantic segmentation image of a dynamic object, a threshold segmentation process is performed to finally obtain a dynamic image frame containing only dynamic objects, and the dynamic image frame is sent to the secondary elimination operation in the visual SLAM module;

[0012] After the Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module and extracts the ORB feature points, it removes abnormal feature points based on the Lucas-Kanade sparse optical flow method, and then sends the ORB feature points of the data to be processed to the secondary elimination module for secondary elimination operation;

[0013] A dynamic feature point container is set in the secondary elimination module, and before performing the secondary elimination operation, it is first confirmed whether a new dynamic image frame sent by the image segmentation module is received;

[0014] If received, the newly received dynamic image frame is recorded as: NewImg, NewImg is used to update the dynamic feature point container, and then the dynamic feature point container is used to perform a secondary elimination operation on the ORB feature points to be processed;

[0015] If no dynamic image frame update is received, directly using the existing feature points in the dynamic feature point container to perform the secondary elimination operation on the ORB feature points of the data to be processed;

[0016] The secondary elimination module sends the ORB feature points of the data to be processed after the secondary elimination operation to the feature matching pose estimation module;

[0017] In the feature matching pose estimation module, feature matching is performed on the ORB feature points of the data to be processed, the pose of the robot is estimated to obtain a pose estimate, and a three-dimensional point cloud map of the environment is constructed;

[0018] The pose estimation result and the three-dimensional point cloud map are sent to the controller, and the controller constructs a two-dimensional occupancy grid map based on the pose estimation result and the three-dimensional point cloud map, completes the robot's trajectory planning according to the desired target position, and controls the robot's movement.

[0019] It is further characterized by:

[0020] The process of removing abnormal feature points in the Lucas-Kanade sparse optical flow module specifically includes the following steps:

[0021] a1: the Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module, which is recorded as prevImg;

[0022] a2: The Lucas-Kanade sparse optical flow module detects ORB feature points on the image frame prevImg to be processed, extracts the ORB feature points, completes the initialization of the sparse optical flow points, and saves the image frame prevImg and the coordinate data of the feature points;

[0023] a3: receiving the next set of RGB image frames of prevImg, denoted as: nextImg;

[0024] a4: Based on the Lucas-Kanade sparse optical flow method, calculate the optical flow of prevImg and nextImg, perform ORB feature point tracking, and obtain the tracking status of each ORB feature point;

[0025] a5: Calculate the error value errors of each ORB feature point;

[0026] Compare each error value errors with the preset error value threshold errors_value;

[0027] If the ORB feature point is tracked successfully and the error value errors is greater than the error value threshold errors_value, the ORB feature point is set as an abnormal feature point; delete the abnormal feature point;

[0028] Until all ORB feature points of nextImg have participated in the calculation, execute step a6;

[0029] a6: Compare the number of the remaining ORB feature points in nextImg with the preset feature point number threshold,

[0030] The feature point quantity threshold represents the number of feature points that must be met when performing ORB feature point tracking;

[0031] When the number of the remaining ORB feature points of nextImg is less than the feature point number threshold, performing ORB feature point detection on nextImg, and putting the obtained ORB feature points into the feature point set of nextImg;

[0032] a7: Send the ORB feature points corresponding to nextImg to the secondary elimination module;

[0033] a8: Define nextImg as prevImg, and execute steps a3 to a7 in a loop;

[0034] The secondary elimination operation performed in the secondary elimination module includes the following steps:

[0035] b1: receiving the ORB feature points of the data to be processed sent by the Lucas-Kanade sparse optical flow module;

[0036] b2: confirm whether a new dynamic image frame sent by the image segmentation module has been received;

[0037] If received, then the newly received dynamic image frame NewImg is subjected to ORB feature extraction, and the extracted feature points are recorded as: NewORB, and step b3 is executed;

[0038] Otherwise, directly execute step b6;

[0039] b3: Read the existing dynamic feature points in the secondary elimination module, denoted as CurORB;

[0040] Calculate the Hamming distance d between NewORB and CurORB:

[0041]

[0042] Among them, d(A,B) represents the Hamming distance between the two feature points A and B; A i and B i are the binary descriptors of the i-th pair of feature points A and B respectively;

[0043] b4: Compare the Hamming distance d corresponding to each feature point in NewORB with the preset distance threshold d_value;

[0044] When d < d_value, it is determined that the corresponding feature point duplicates the existing dynamic feature points and is deleted from NewORB;

[0045] b5: Put the remaining NewORB into CurORB;

[0046] b6: Use the dynamic feature point container to perform secondary elimination on the ORB feature points of the data to be processed;

[0047] During the secondary elimination process, calculate the Hamming distance between the feature points in the dynamic feature point container and the ORB feature points of the data to be processed, denoted as d_C, and compare d_C with the preset distance threshold d_cValue;

[0048] Among them, the distance threshold d_cValue is set to the value that eliminates 20% of the ORB feature points of the data to be processed in the secondary elimination operation;

[0049] b7: Send the ORB feature points of the data to be processed after secondary elimination into the feature matching pose estimation module;

[0050] The process of segmenting the RGB image frame in the data to be processed in the image segmentation module includes the following steps:

[0051] c1: Build a semantic segmentation network model based on the Mask R-CNN model;

[0052] c2: Receive the RGB image frame in the data to be processed sent by the image acquisition module, denoted as: the image frame to be processed;

[0053] c3: Send the image frame to be processed into the semantic segmentation network model for object detection to obtain the semantic segmentation image of the dynamic object;

[0054] c4: Obtain the threshold segmentation map of the semantic segmentation image;

[0055] c5: Perform an AND operation on the threshold segmentation map and the corresponding image frame to be processed to mask and cover the non-dynamic area, that is, obtain the dynamic image frame with only dynamic objects.

[0056] The present invention provides a mobile robot based on a dynamic real-time visual SLAM method. The mobile robot provides an image segmentation module and a visual SLAM module, and inputs environmental image data acquired by an image acquisition module into the image segmentation module and the visual SLAM module respectively, and processes image segmentation processing and posture estimation operations in parallel. Outliers are eliminated in a Lucas-Kanade sparse optical flow module and a secondary elimination module respectively, and a dynamic image frame output by the image segmentation module participates in a second outlier elimination operation in the secondary elimination module, which not only improves the recognition accuracy, but also greatly improves the processing efficiency on the basis of ensuring the accuracy of posture estimation, and ensures that the technical solution of the present invention can perform real-time posture estimation in a dynamic environment, so that the mobile robot can be more flexibly applied to different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the modules of the mobile robot of the present invention;

[0058] Figure 2 This is a framework diagram of the dynamic real-time visual SLAM algorithm;

[0059] Figure 3 It is a schematic diagram of threshold segmentation of dynamic image frames;

[0060] Figure 4 It is a schematic diagram of the secondary rejection module process;

[0061] Figure 5 This is a single-frame timing graph of the tracking thread in the algorithm timing experiment. DETAILED DESCRIPTION

[0062] The present invention provides a mobile robot based on a dynamic real-time visual SLAM method, such as Figure 1 As shown, it includes: an image acquisition module, a dynamic visual SLAM module, a controller and a power supply module. The dynamic visual SLAM module includes: a visual SLAM module and an image segmentation module. The visual SLAM module and the image segmentation module are connected to the image acquisition module through ROS topic communication; the controller is connected to the dynamic visual SLAM module through serial port communication. The power supply module supplies power to all other modules. Other robot components are implemented based on modules in the prior art.

[0063] like Figure 2 As shown, after the image acquisition module obtains the RGB image information and depth information of the environment, the RGB image information and the depth information are transmitted in the form of RGB image sequence and depth image sequence, which are recorded as data to be processed; the image acquisition module sends the data to be processed to the visual SLAM module and the image segmentation module respectively.

[0064] In the visual SLAM module, the main working steps of the dynamic real-time visual SLAM algorithm include: tracking thread, local mapping, loop detection and construction of three-dimensional point cloud map in sequence.

[0065] The data to be processed is sent to the visual SLAM module for recognition operation; the processing operation flow of the data to be processed in the visual SLAM module is carried out in parallel with the processing operation flow of the data to be processed in the image segmentation module; the visual SLAM module includes: a sequentially connected Lucas-Kanade sparse optical flow module, a secondary elimination module, and a feature matching pose estimation module. Figure 2 As shown in the figure, the operations of the image acquisition module, the sequentially connected Lucas-Kanade sparse optical flow module, the secondary elimination module and the feature matching pose estimation module constitute a complete tracking thread, which makes the image segmentation module independent of the tracking thread. The tracking thread does not need to wait for the segmentation result of the image segmentation module for the current frame, but directly uses the existing dynamic feature points for matching, which greatly reduces the time consumption of the tracking thread and improves the real-time performance of the system.

[0066] In the image segmentation module, the RGB image frames in the processed data are subjected to target detection through the semantic segmentation network. After obtaining the semantic segmentation image of the dynamic object, threshold segmentation processing is performed, and finally a dynamic image frame with only dynamic objects is obtained. The dynamic image frame is sent to the secondary elimination operation in the tracking thread.

[0067] In the tracking thread, the Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module to extract the ORB feature points, and then removes abnormal feature points based on the Lucas-Kanade sparse optical flow method, and then sends the ORB feature points of the data to be processed to the secondary elimination module for secondary elimination operation.

[0068] A dynamic feature point container is set in the secondary elimination module. Before performing the secondary elimination operation, it is first confirmed whether a new dynamic image frame sent by the image segmentation module is received;

[0069] If received, the newly received dynamic image frame is recorded as: NewImg, and the dynamic feature point container is updated using NewImg, and then the dynamic feature point container is used to perform a secondary elimination operation on the ORB feature points to be processed;

[0070] If no dynamic image frame update is received, the existing feature points in the dynamic feature point container are directly used to perform a secondary elimination operation on the ORB feature points of the data to be processed;

[0071] The secondary elimination module sends the ORB feature points of the data to be processed after the secondary elimination operation to the feature matching pose estimation module.

[0072] In the feature matching pose estimation module, feature matching is performed on the ORB feature points of the data to be processed, the robot's pose is estimated, and a three-dimensional point cloud map of the environment is constructed; the pose estimation results and the three-dimensional point cloud map are sent to the controller, and the controller constructs a two-dimensional occupancy grid map based on the pose estimation results, completes the robot's trajectory planning according to the expected target position, and controls the robot's movement.

[0073] The Lucas-Kanade sparse optical flow method is based on three assumptions: first, constant brightness, that is, the brightness of the same target in different frames does not change; second, small motion, that is, the motion of adjacent frames is small; third, spatial consistency, that is, the pixels in the target area have similar motion. The Lucas-Kanade sparse optical flow method tracks feature points in continuous frame images and calculates the optical flow vector of each feature point to determine whether there are abnormal feature points.

[0074] Based on the two assumptions of constant brightness and small motion, the constraint equation of the image is:

[0075] I x u+I y v+I t =0

[0076] Where u and v are the velocity vectors of the pixel along the x and y axes respectively; I x ,I y are the gradients of the image in the x and y axis directions respectively; I t is the gradient in the time direction.

[0077] According to the spatial consistency assumption, that is, the optical flow in the neighborhood is a fixed value, the pixels in the neighborhood have similar motion, and the selected neighborhood range, all the pixels in the neighborhood can be expressed by the following formula:

[0078]

[0079] Where P is the pixel point in the neighborhood;

[0080] The tracking points selected in the technical solution of the present invention are ORB feature points. ORB feature points are corner points with obvious changes, which can effectively avoid the aperture problem and thus avoid the problem of being unable to determine the direction of the displacement point. The fitting optimization is performed by the least squares method, and the velocity vector can be finally solved as:

[0081]

[0082] The process of removing abnormal feature points in the Lucas-Kanade sparse optical flow module specifically includes the following steps:

[0083] a1: The Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module, denoted as prevImg;

[0084] a2: The Lucas-Kanade sparse optical flow module detects ORB feature points on the image frame prevImg to be processed, extracts the ORB feature points, completes the initialization of the sparse optical flow points, and saves the image frame prevImg and the coordinate data of the feature points;

[0085] a3: Receive the next set of RGB image frames from prevImg, denoted as nextImg;

[0086] a4: Based on the Lucas-Kanade sparse optical flow method, call the calcOpticalFlowPyrLK() function to calculate the optical flow of prevImg and nextImg, perform ORB feature point tracking, and obtain the tracking status of each ORB feature point;

[0087] a5: Calculate the error value errors of each ORB feature point;

[0088] Compare each error value errors with the preset error value threshold errors_value;

[0089] If the ORB feature point is tracked successfully and the error value errors is greater than the error value threshold errors_value, the ORB feature point is set as an abnormal feature point; delete the abnormal feature point;

[0090] Until all ORB feature points of nextImg have participated in the calculation, execute step a6;

[0091] a6: Compare the number of ORB feature points remaining in nextImg with the preset feature point number threshold.

[0092] The feature point number threshold indicates the number of feature points that must be met when performing ORB feature point tracking;

[0093] When the number of remaining ORB feature points of nextImg is less than the feature point number threshold, ORB feature point detection is performed on nextImg, and the obtained ORB feature points are put into the feature point set of nextImg;

[0094] a7: Send the ORB feature points corresponding to nextImg to the secondary elimination module;

[0095] a8: Define nextImg as prevImg, and execute steps a3 to a7 in a loop.

[0096] In the Lucas-Kanade sparse optical flow module, each time the input image frame is subjected to the removal of abnormal feature points, the image will lose some ORB feature points, so that the number of feature points gradually decreases with the tracking time, and eventually the tracking is lost and the path is incomplete due to too few feature points. For this reason, this technical solution sets a feature point number threshold after the abnormal feature points are removed. When the number of feature points in the image frame with abnormal feature points is lower than the feature point number threshold, the current image frame is detected again with ORB feature points, and the feature point set of the current frame image is updated to ensure that there are sufficient feature points for tracking.

[0097] In the technical solution of the present invention, based on ORB feature points and Lucas-Kanade sparse optical flow method, the calcOpticalFlowPyrLK() function of the OpenCV library is used to realize the elimination of abnormal feature points, and integrate it into the tracking thread to realize the preliminary elimination of potential dynamic feature points.

[0098] In the technical solution of the present invention, the secondary elimination module uses the semantic segmentation result of the image segmentation module to perform secondary elimination on the potential dynamic feature points of the input image frame; Figure 4 As shown, the secondary elimination module first determines whether the dynamic image frame of the image segmentation module is received. If there is no dynamic image frame update, the existing dynamic image frame is directly used for secondary elimination; if there is a dynamic image frame update, the received dynamic image frame is processed before subsequent secondary elimination.

[0099] The secondary elimination operation performed in the secondary elimination module specifically includes the following steps:

[0100] b1: receives the ORB feature points of the data to be processed sent by the Lucas-Kanade sparse optical flow module;

[0101] b2: confirm whether a new dynamic image frame sent by the image segmentation module has been received;

[0102] If received, then the newly received dynamic image frame NewImg is subjected to ORB feature extraction, and the extracted feature points are recorded as: NewORB, and step b3 is executed;

[0103] Otherwise, directly execute step b6;

[0104] b3: Read the existing dynamic feature points in the secondary elimination module, denoted as CurORB;

[0105] Calculate the Hamming distance d between NewORB and CurORB:

[0106]

[0107] Among them, d(A,B) represents the Hamming distance between two feature points A and B; A i and B i are respectively the binary descriptors of the i-th pair of points of two feature points A and B;

[0108] b4: Compare the Hamming distance d corresponding to each feature point in NewORB with a preset distance threshold d_value;

[0109] When d < d_value, it is determined that the corresponding feature point duplicates the existing dynamic feature points and is deleted from NewORB;

[0110] By means of the distance threshold d_value, feature points with too small distances, that is, similar dynamic feature points, are eliminated to reduce the overall calculation amount;

[0111] b5: Put the remaining NewORB into CurORB;

[0112] b6: Use the dynamic feature point container to perform secondary elimination on the ORB feature points of the data to be processed;

[0113] During the secondary elimination process, calculate the Hamming distance between the feature points in the dynamic feature point container and the ORB feature points of the data to be processed, denoted as d_C, and compare d_C with a preset distance threshold d_cValue;

[0114] Among them, the distance threshold d_cValue is set to the value that eliminates 20% of the ORB feature points of the data to be processed in the secondary elimination operation, ensuring that abnormal feature points are eliminated and there are enough feature points to participate in the subsequent pose estimation calculation;

[0115] b7: Send the ORB feature points of the data to be processed after secondary elimination into the feature matching pose estimation module.

[0116] As Figure 3 shown, the process of segmenting the RGB image frame in the data to be processed in the image segmentation module includes the following steps:

[0117] c1: Build a semantic segmentation network model based on the Mask R-CNN model;

[0118] c2: Receive the RGB image frame in the data to be processed sent by the image acquisition module, denoted as: the image frame to be processed;

[0119] c3: Send the image frame to be processed into the semantic segmentation network model for target detection to obtain the semantic segmentation image of the dynamic object;

[0120] c4: Obtain the threshold segmentation map of the semantic segmentation image;

[0121] The pixel values ​​of the dynamic area of ​​the semantic segmentation image are set to 0, and the pixel values ​​of the remaining areas are set to 255, so as to obtain a threshold segmentation image in which the dynamic area is white and the remaining areas are black;

[0122] c5: Perform an AND operation on the threshold segmentation map and the corresponding image frame to be processed, and mask the non-dynamic area, so as to obtain a dynamic image frame with only dynamic objects;

[0123] Call the cv2.bitwise_and() function of the OpenCV library to perform an AND operation on the threshold segmentation map and the input image frame, mask the non-dynamic area, and obtain a dynamic image frame with only dynamic objects. The image binarization operation is as follows:

[0124]

[0125] Among them, I input ,I output and I mask are the pixel values ​​of the target position in the input image frame, dynamic image frame and threshold segmentation map respectively.

[0126] The semantic segmentation network model constructed in the image segmentation module in the technical solution of the present invention is based on the Mask R-CNN instance segmentation network, ensuring that the semantic segmentation model has the advantages of high accuracy, simplicity, intuitiveness, and ease of use. In specific implementation, the Mask R-CNN network is pre-trained on the COCO dataset, which contains more than 80 categories of dynamic objects and meets the experimental requirements. Instance segmentation based on the semantic segmentation network model in the technical solution of the present invention can predict pixel-level semantic labels and generate masks for dynamic objects in the input image.

[0127] In order to verify the effectiveness and feasibility of the technical solution of the present invention, the TUM dataset was selected for experiments, and compared with several visual recognition methods such as ORB-SLAM2, ReFusion, and Dyna-SLAM in the prior art, and a quantitative analysis was performed in terms of accuracy and running frame rate.

[0128] The experimental platform is a laptop with Ubuntu 16.04 operating system, 8GB running memory, processor model: i7-8550U, main frequency 1.8GHz, 64-bit operating system, and an NVIDIA GeForce MX 150 graphics card. The TUM dataset provides sequence-aligned RGB images and depth images, which can directly perform operations such as point cloud segmentation, pose estimation and 3D reconstruction. The image resolution of this data is 640×480. The experiment selected six sub-datasets of walking_static, walking_xyz, walking_rpy, walking_halfsphere, sitting_xyz, and sitting_halfsphere under the TUM RGB-D freiburg3 dataset, where walking and sitting represent the datasets of high-dynamic and low-dynamic scenes respectively. The experiment selected absolute trajectory error as the evaluation criterion, and the specific results of the comparative experiment are shown in Table 1 below.

[0129] Table 1 Absolute trajectory error (unit: m)

[0130]

[0131] It can be seen from Table 1 that compared with the ORB-SLAM2 algorithm, the camera positioning error of the technical solution of the present invention is reduced by 22.14% and 95.14% in low-dynamic and high-dynamic environments respectively; compared with the ReFusion algorithm, the positioning error is reduced by 74.32% and 78.14% in low-dynamic and high-dynamic environments respectively; compared with the Dyna-SLAM algorithm, the experimental results are similar; the absolute trajectory error of the camera of the ORB-SLAM2 and ReFusion algorithms is small in low-dynamic scenes, and the error is large in high-dynamic environments, and even frame loss occurs; Dyna-SLAM and the technical solution of the present invention can retain static feature points as much as possible in two dynamic environments, ensure the accuracy of camera positioning, and keep the estimated trajectory with high consistency. The data set experiment shows that the technical solution of the present invention has good positioning accuracy and robustness in dynamic environments.

[0132] In addition to positioning accuracy, the real-time performance of the SLAM algorithm is also an important performance indicator for evaluating the quality of the system. Figure 5 It shows the time consumption of each frame of the tracking thread of the four algorithms under each sub-dataset of TUM RGB-D. Figure 5 It can be seen that the ORB-SLAM2 algorithm has the lowest time consumption, the Dyna-SLAM algorithm based on deep learning has the highest time consumption, and the time consumption fluctuates greatly in dynamic scenes. The algorithm of the present invention is close to ORB-SLAM2 in terms of time consumption. Table 2 below records the average time consumption of the tracking thread under the four algorithms in each sub-dataset of TUM RGB-D.

[0133] Table 2 Average time spent on tracking threads (unit: ms)

[0134]

[0135] It can be seen from Table 2 that the technical solution of the present invention improves the tracking thread so that the time consumption of the tracking thread is less affected by the image segmentation algorithm, which can effectively reduce the time consumption of the algorithm tracking thread and improve the real-time performance of the algorithm while ensuring the positioning accuracy.

[0136] The dynamic real-time visual SLAM method in the technical solution of the present invention is based on the ORB-SLAM2 algorithm, combined with the Lucas-Kanade sparse optical flow method and the Mask R-CNN target detection algorithm, to detect dynamic potential objects in the environment, to achieve perception of dynamic objects in the environment, and to ensure that the robot can perceive and understand the environment more accurately; to improve the tracking thread algorithm, to remove dynamic feature points in real time, to accurately and efficiently complete the robot pose estimation and the construction of the three-dimensional point cloud map; the controller can more accurately complete the path planning based on the robot pose estimation information and the three-dimensional point cloud map; the technical solution of this patent can ensure that the robot can be used more flexibly in various complex scenes with dynamic objects.

Claims

1. A mobile robot based on a dynamic real-time visual SLAM method, comprising: The image acquisition module, dynamic visual SLAM module and controller are characterized by: The dynamic visual SLAM module includes: a visual SLAM module and an image segmentation module; The visual SLAM module and the image segmentation module are respectively connected to the image acquisition module for communication; The controller is connected to the dynamic vision SLAM module via serial port communication; After the image acquisition module acquires the RGB image information and the depth information of the environment, the RGB image information and the depth information are transmitted in the form of an RGB image sequence and a depth image sequence, which are recorded as data to be processed; the image acquisition module sends the data to be processed to the visual SLAM module and the image segmentation module respectively; The data to be processed is sent to the visual SLAM module for recognition operation; the processing operation flow of the data to be processed in the visual SLAM module is carried out in parallel with the operation flow of the data to be processed in the image segmentation module; The visual SLAM module includes: a Lucas-Kanade sparse optical flow module, a secondary elimination module, and a feature matching pose estimation module connected in sequence; In the image segmentation module, the RGB image frame in the data to be processed is subjected to target detection through a semantic segmentation network, and after obtaining a semantic segmentation image of a dynamic object, a threshold segmentation process is performed to finally obtain a dynamic image frame containing only dynamic objects, and the dynamic image frame is sent to the secondary elimination module in the visual SLAM module; After the Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module and extracts the ORB feature points, it removes abnormal feature points based on the Lucas-Kanade sparse optical flow method, and then sends the ORB feature points of the data to be processed to the secondary elimination module for secondary elimination operation; A dynamic feature point container is set in the secondary elimination module, and before performing the secondary elimination operation, it is first confirmed whether a new dynamic image frame sent by the image segmentation module is received; If received, the newly received dynamic image frame is recorded as: NewImg, NewImg is used to update the dynamic feature point container, and then the dynamic feature point container is used to perform a secondary elimination operation on the ORB feature points of the data to be processed; If no dynamic image frame update is received, directly using the existing feature points in the dynamic feature point container to perform the secondary elimination operation on the ORB feature points of the data to be processed; The secondary elimination module sends the ORB feature points of the data to be processed after the secondary elimination operation to the feature matching pose estimation module; In the feature matching pose estimation module, feature matching is performed on the ORB feature points of the data to be processed, the pose of the robot is estimated to obtain a pose estimate, and a three-dimensional point cloud map of the environment is constructed; The pose estimation result and the three-dimensional point cloud map are sent to the controller, and the controller constructs a two-dimensional occupancy grid map based on the pose estimation result and the three-dimensional point cloud map, completes the robot's trajectory planning according to the desired target position, and controls the robot's movement.

2. A mobile robot based on a dynamic real-time visual SLAM method according to claim 1, characterized in that: The process of removing abnormal feature points in the Lucas-Kanade sparse optical flow module specifically includes the following steps: a1: the Lucas-Kanade sparse optical flow module receives the current RGB image frame input by the image acquisition module, which is recorded as prevImg; a2: The Lucas-Kanade sparse optical flow module detects ORB feature points on the image frame prevImg to be processed, extracts the ORB feature points, completes the initialization of the sparse optical flow points, and saves the image frame prevImg and the coordinate data of the feature points; a3: receiving the next set of RGB image frames of prevImg, denoted as: nextImg; a4: Based on the Lucas-Kanade sparse optical flow method, calculate the optical flow of prevImg and nextImg, perform ORB feature point tracking, and obtain the tracking status of each ORB feature point; a5: Calculate the error value errors of each ORB feature point; Compare each error value errors with the preset error value threshold errors_value; If the ORB feature point is tracked successfully and the error value errors is greater than the error value threshold errors_value, the ORB feature point is set as an abnormal feature point; delete the abnormal feature point; Until all ORB feature points of nextImg have participated in the calculation, execute step a6; a6: Compare the number of the remaining ORB feature points in nextImg with the preset feature point number threshold, The feature point quantity threshold represents the number of feature points that must be met when performing ORB feature point tracking; When the number of the remaining ORB feature points of nextImg is less than the feature point number threshold, performing ORB feature point detection on nextImg, and putting the obtained ORB feature points into the feature point set of nextImg; a7: Send the ORB feature points corresponding to nextImg to the secondary elimination module; a8: Define nextImg as prevImg, and execute steps a3 to a7 in a loop.

3. A mobile robot based on a dynamic real-time visual SLAM method according to claim 1, characterized in that: The secondary elimination operation performed in the secondary elimination module includes the following steps: b1: receiving the ORB feature points of the data to be processed sent by the Lucas-Kanade sparse optical flow module; b2: confirm whether a new dynamic image frame sent by the image segmentation module has been received; If received, then the newly received dynamic image frame NewImg is subjected to ORB feature extraction, and the extracted feature points are recorded as: NewORB, and step b3 is executed; Otherwise, directly execute step b6; b3: Read the existing dynamic feature points in the secondary rejection module, denoted as: CurORB; Calculate the Hamming distance d between NewORB and CurORB: Among them, d(A,B) represents the Hamming distance between the two feature points A and B; A i and B i are the binary descriptors of the i-th pair of feature points A and B respectively; b4: Compare the Hamming distance d corresponding to each feature point in NewORB with a preset distance threshold d_value; When d < d_value, determine that the corresponding feature point duplicates the existing dynamic feature points and delete it from NewORB; b5: Put the remaining NewORB into CurORB; b6: Use the dynamic feature point container to perform secondary rejection on the ORB feature points of the data to be processed; During the secondary rejection process, calculate the Hamming distance between the feature points in the dynamic feature point container and the ORB feature points of the data to be processed, denoted as d_C, and compare d_C with a preset distance threshold d_cValue; Among them, the distance threshold d_cValue is set to the value for rejecting 20% of the ORB feature points of the data to be processed in the secondary rejection operation; b7: Send the ORB feature points of the data to be processed after secondary rejection into the feature matching pose estimation module.

4. A mobile robot based on a dynamic real-time visual SLAM method according to claim 1, characterized in that: The process of segmenting the RGB image frame in the data to be processed in the image segmentation module includes the following steps: c1: Build a semantic segmentation network model based on the Mask R-CNN model; c2: Receive the RGB image frame in the data to be processed sent by the image acquisition module, denoted as: the image frame to be processed; c3: Send the image frame to be processed into the semantic segmentation network model for object detection to obtain a semantic segmentation image of the dynamic object; c4: Obtain the threshold segmentation map of the semantic segmentation image; c5: Perform an AND operation on the threshold segmentation map and the corresponding image frame to be processed to mask and cover the non-dynamic area, that is, obtain the dynamic image frame with only the dynamic object remaining.

Images