Visual SLAM method and system of mobile robot and readable storage medium

Abstract

The invention provides a visual SLAM method and system of a mobile robot and a readable storage medium, and the method comprises the steps: S1, collecting original image data, and carrying out the classification and marking of the image data; s2, training the original image data set to obtain a migration training image data set; s3, inputting the original image data into the migration training image data set, and obtaining a coordinate rectangular frame of the dynamic object; s4, acquiring a depth map of the original image data, performing image segmentation on the dynamic object through the coordinate rectangular frame of the dynamic object, and acquiring a mask of the moving object; and S5, feature point extraction is carried out on the original image through the moving object mask, dynamic feature points are removed, and static feature points are reserved. The method disclosed by the invention has the capability of stably working in an indoor dynamic scene, and is small in calculation amount, simple to use and relatively high in real-time performance.

Description

[Technical field]

[0001] The present invention relates to the field of image detection technology, and in particular to a visual SLAM method, system and readable storage medium of a mobile robot, which are suitable for indoor dynamic environments. [Background technology]

[0002] In the field of autonomous navigation of mobile robots, simultaneous localization and mapping (SLAM) technology is the key to achieve accurate positioning and environmental mapping. Visual SLAM, especially feature point-based visual SLAM methods, has become the mainstream choice due to its cost-effectiveness and information richness. However, dynamic indoor environments, such as warehouses and shopping malls, are full of walking people and moving objects, which makes traditional visual SLAM systems based on static environment assumptions face challenges because they easily misidentify dynamic objects as static feature points, resulting in inaccurate positioning.

[0003] The feature point-based visual SLAM system uses multi-view geometric constraints for feature point matching. This method can treat dynamic feature points as bad points and eliminate them, but it is more suitable for low-dynamic scenes, that is, when there are fewer dynamic objects in the scene, the system can usually obtain more accurate positioning accuracy and build an environmental map, but when there are more dynamic objects in the scene and they account for a large proportion, the robustness of the system will be significantly reduced. In order to cope with the challenges brought by high-dynamic environments, researchers have proposed a variety of improvement schemes, including detection techniques based on frame difference method, optical flow method and image segmentation technology based on deep learning. These methods attempt to identify and exclude dynamic objects in the image to maintain the stability of feature points. However, these methods still face the dual challenges of real-time and accuracy when dealing with complex dynamic scenes.

[0004] Among them, the scheme based on the frame difference method is to identify moving objects by comparing and analyzing the differences between consecutive image frames. This type of algorithm performs differential operations on two or three consecutive frames of images in time, subtracts the pixels corresponding to different frames, and determines the absolute value of the grayscale difference. When the absolute value exceeds a certain threshold, it can be determined as a moving target, thereby realizing the detection function of moving targets. Different from conventional moving target segmentation, since the camera and the moving targets in the environment move at the same time, motion compensation is required to eliminate the impact of camera motion, resulting in a causal paradox problem, and subsequent processing steps such as threshold processing and morphological algorithms may not be able to completely eliminate errors.

[0005] Another solution is the optical flow method, which detects dynamic objects by analyzing the movement of pixels in an image sequence. This method distinguishes its own dynamics by comparing the motion vectors of feature points or combines them with clustering algorithms and other methods for detection. Literature review and actual operation have shown that the pure optical flow method has low accuracy, and its real-time performance is reduced when combined with various algorithms. The optical flow method works based on the assumption of constant brightness, so it is easily affected by lighting. In addition, this method is affected by changes in the optical flow vector and noise interference when the camera moves, making it difficult to distinguish between dynamic and static backgrounds.

[0006] Currently, deep learning-based object detection technology has become one of the reliable solutions for handling dynamic objects in SLAM systems. These technologies improve the stability of feature points by training deep neural networks to identify and segment dynamic objects in images. However, deep learning models usually contain a large number of parameters, which significantly increases the demand for computing resources, especially in real-time processing. Even high-performance GPU acceleration can hardly meet the real-time requirements in all application scenarios, and deep learning's reliance on high-end hardware limits the portability of these systems.

[0007] Therefore, it is necessary to study a visual SLAM method, system and readable storage medium for a mobile robot to address the deficiencies of the prior art and to solve or alleviate one or more of the above problems. [Summary of the invention]

[0008] In view of this, the present invention provides a visual SLAM method, system and readable storage medium for a mobile robot. The method has the ability to work stably in indoor dynamic scenes, has small computational complexity, is easy to use, and has high real-time performance.

[0009] In one aspect, the present invention provides a visual SLAM method for a mobile robot, the method comprising the following steps:

[0010] S1: Collect original image data, classify and annotate the image data, and obtain the original image data set;

[0011] S2: Train the original image dataset through deep learning to obtain a migration training image dataset;

[0012] S3: Based on the migration training dataset for pedestrians in indoor dynamic environments, a lightweight dynamic object target recognition algorithm framework is constructed to obtain the coordinate rectangle of the dynamic object;

[0013] The target loss function of the matching strategy during the target recognition algorithm framework transfer training process is:

[0014]

[0015] x is the target indicator, c is the center of the target rectangular box, l is the target predicted rectangular box, g is the target real rectangular box, N is the number of target matching bounding boxes, L conf is the confidence loss, L loc is the positioning loss, α is the formaldehyde parameter;

[0016] S4: Obtain the depth map of the original image data, design a region growing algorithm based on depth information, and use the dynamic object coordinate rectangle to build a dynamic object mask generation and image segmentation module:

[0017]

[0018] D x,y is the depth information at (x, y) in the image, is the depth information at (x0, y0) in the image, (x, y) and (x0, y0) are adjacent pixels

[0019] S5: Extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

[0020]

[0021] Among them, P ORB is the ORB feature point, N is the number of ORB feature points, P MASK is any pixel point of the dynamic object mask, P dynamic is a dynamic ORB feature point, P static is a static ORB feature point, D T is the distance threshold.

[0022] According to the above aspects and any possible implementation, an implementation is further provided, wherein the original image data in S1 is collected by a mobile robot performing multiple data collections in a working scene.

[0023] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the classification and labeling process in S1 is specifically: labeling and classifying objects in the image through a graphical interface labeling tool, generating category labels and labeling labels, wherein the labeling labels include but are not limited to coordinate information and coordinate rectangular boxes.

[0024] According to the aspects and any possible implementation methods described above, an implementation method is further provided, wherein S2 specifically comprises: performing migration training on the Mobile Net-SSD model according to preset training parameters through a deep learning framework, wherein the deep learning framework includes but is not limited to a caffe deep learning framework.

[0025] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein the deep learning training process is specifically: downloading the model and weights and writing their addresses in the command of the training file, setting a series of deep learning parameters for migration training, and some parameters and their values ​​are as follows: setting the learning rate base_lr in the range of 0.0001-0.005, and setting the weight decay parameter weight_decay to 0.00005.

[0026] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein S3 specifically: inputs the original image data into the Mobile Net-SSD neural network, performs image detection through the model weights trained by S2, outputs the type, confidence and rectangular frame coordinate information of the detected target, further filters and obtains the coordinates of the prior moving object through the confidence level, and designs the growth point coordinates according to the type of moving object.

[0027] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein S4 is specifically: based on the depth map corresponding to the original image, regional growth is performed within the same target rectangular frame coordinate range through the growth points in S3, the threshold of the regional growing algorithm is set, the contour of the moving object is segmented, and the segmentation result of each frame of the image is made into a binary mask, wherein the pixels with a value of 1 in the binary mask are semantically static object pixels, and the pixels with a value of 0 are semantically dynamic object pixels.

[0028] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein S5 is specifically: inputting the generated moving object mask into the feature point extraction, using the moving object mask to eliminate the feature points belonging to the dynamic object, retaining the feature points of the static object, and completing the precise positioning and map construction of the visual SLAM.

[0029] According to the above aspects and any possible implementation, a visual SLAM system for a mobile robot is further provided, the system comprising:

[0030] The data acquisition and processing module is used to collect original image data, classify and annotate the image data, and obtain the original image data set;

[0031] A data learning training module is used to train the original image data set through deep learning to obtain a migration training image data set;

[0032] A dynamic object acquisition module is used to input the original image data into the migration training image data set to obtain the coordinate rectangular frame of the dynamic object;

[0033] The image segmentation processing module is used to obtain the depth map of the original image data, perform image segmentation on the dynamic object through the coordinate rectangular frame of the dynamic object, and obtain the moving object mask;

[0034] The feature point extraction module is used to extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

[0035] According to the above aspects and any possible implementation, a computer-readable storage medium is further provided, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method.

[0036] Compared with the prior art, the present invention can achieve the following technical effects:

[0037] Compared with the prior art, the present invention focuses on the SLAM system in a dynamic environment, comprehensively analyzes the advantages and disadvantages of the lightweight model Mobile Net-SSD, performs migration training on the lightweight model Mobile Net-SSD, and improves the accuracy of dynamic object recognition. In addition, the present invention proposes to complete segmentation along the contour of dynamic objects based on depth map and region growing algorithm, while retaining enough static feature points while eliminating dynamic feature points.

[0038] Of course, any product implementing the present invention does not necessarily need to achieve all of the above-mentioned technical effects at the same time.

Brief Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a flow chart of a dynamic SLAM method based on a lightweight target detection model disclosed in the present invention;

[0041] Figure 2 A schematic diagram of the SSD architecture of a visual SLAM method for a mobile robot disclosed in the present invention;

[0042] Figure 3 A target detection data set preparation process for a visual SLAM method for a mobile robot disclosed in the present invention;

[0043] Figure 4The dynamic object segmentation result of the visual SLAM method of a mobile robot disclosed in the present invention;

[0044] Figure 5 The figure is a feature point extraction result diagram of a visual SLAM method for a mobile robot disclosed in the present invention, wherein (a) is a feature point extraction diagram of the system of the present invention, and (b) is a feature point extraction diagram of the ORB-SLAM2 system. [Specific implementation method]

[0045] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0046] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0048] The present invention provides a visual SLAM method, system and readable storage medium for a mobile robot, which selects a lightweight network architecture, such as YOLO (You Only Look Once) or Mobile Net-SSD (Single Shot MultiBox Detector), and combines an optimized compensation algorithm to design a moving object rejection module, so as to significantly reduce the computational burden of the model without sacrificing too much accuracy. This optimized deep learning model not only improves the ability of real-time processing, but also reduces the dependence on high-end hardware, making it easier to deploy the SLAM system on resource-constrained terminal devices.

[0049] The present invention provides a visual SLAM method, system and readable storage medium for a mobile robot, which has the ability to work stably in indoor dynamic scenes, has a small amount of calculation, is easy to use and has high real-time performance. The method comprises the following steps:

[0050] S1: Collect original image data, classify and annotate the image data, and obtain the original image data set;

[0051] S2: Train the original image dataset through deep learning to obtain a migration training image dataset;

[0052] S3: Based on the migration training dataset for pedestrians in indoor dynamic environments, a lightweight dynamic object target recognition algorithm framework is constructed to obtain the coordinate rectangle of the dynamic object;

[0053] The target loss function of the matching strategy during the target recognition algorithm framework transfer training process is:

[0054]

[0055] x is the target indicator, c is the center of the target rectangular box, l is the target predicted rectangular box, g is the target real rectangular box, N is the number of target matching bounding boxes, L conf is the confidence loss, L loc is the positioning loss, α is the formaldehyde parameter;

[0056] S4: Obtain the depth map of the original image data, design a region growing algorithm based on depth information, and use the dynamic object coordinate rectangle to build a dynamic object mask generation and image segmentation module:

[0057]

[0058] D x,y is the depth information at (x, y) in the image, is the depth information at (x0, y0) in the image, (x, y) and (x0, y0) are adjacent pixels

[0059] S5: Extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

[0060]

[0061] Among them, P ORB is the ORB feature point, N is the number of ORB feature points, P MASK is any pixel point of the dynamic object mask, P dynamic is a dynamic ORB feature point, P static is a static ORB feature point, D T is the distance threshold.

[0062] The original image data in S1 is collected by the mobile robot for multiple data collections in the working scene.

[0063] The classification and labeling process in S1 is specifically as follows: labeling and classifying the objects in the image through a graphical interface labeling tool to generate category labels and labeling labels, wherein the labeling labels include but are not limited to coordinate information and coordinate rectangular boxes.

[0064] Specifically, S2 includes: performing migration training on the Mobile Net-SSD model according to preset training parameters through a deep learning framework, wherein the deep learning framework includes but is not limited to a caffe deep learning framework.

[0065] The deep learning training process is specifically as follows: download the model and weights and write their addresses in the command of the training file, set a series of deep learning parameters for migration training, some parameters and their values ​​are as follows: set the learning rate base_lr in the range of 0.0001-0.005, and set the weight decay parameter weight_decay to 0.00005.

[0066] Specifically, S3 inputs the original image data into the Mobile Net-SSD neural network, performs image detection through the model weights trained by S2, outputs the type, confidence and rectangular frame coordinate information of the detected target, further filters and obtains the coordinates of the prior moving object through the confidence level, and designs the growth point coordinates according to the type of the moving object.

[0067] Specifically, S4 includes: based on the depth map corresponding to the original image, performing regional growth within the same target rectangular frame coordinate range through the growth points in S3, setting the threshold of the regional growing algorithm, performing contour segmentation on the moving object, and making the segmentation result of each frame image into a binary mask, wherein the pixels with a value of 1 in the binary mask are semantically static object pixels, and the pixels with a value of 0 are semantically dynamic object pixels.

[0068] The S5 is specifically as follows: inputting the generated moving object mask into feature point extraction, using the moving object mask to remove feature points belonging to dynamic objects, retaining feature points of static objects, and completing accurate positioning and map construction of visual SLAM.

[0069] Specifically, S5 is as follows: image feature matching is performed based on ORB-SLAM2 by using information of instance segmentation mask, wherein ORB feature points are mainly matched by similarity of feature point descriptors of two adjacent frames of images, and multiple sets of matched feature point pairs are obtained through feature matching.

[0070] The feature point matching requirements are:

[0071]

[0072] Where I(x) is the grayscale of any point on the circumference, I(p) is the grayscale of the center of the circle, and ε d is the threshold of gray value difference. If N is greater than the given threshold, generally three quarters of the surrounding circle points, then p is considered to be a feature point.

[0073] The present invention also provides a visual SLAM system for a mobile robot, the system comprising:

[0074] The data acquisition and processing module is used to collect original image data, classify and annotate the image data, and obtain the original image data set;

[0075] A data learning training module is used to train the original image data set through deep learning to obtain a migration training image data set;

[0076] A dynamic object acquisition module is used to input the original image data into the migration training image data set to obtain the coordinate rectangular frame of the dynamic object;

[0077] The image segmentation processing module is used to obtain the depth map of the original image data, perform image segmentation on the dynamic object through the coordinate rectangular frame of the dynamic object, and obtain the moving object mask;

[0078] The feature point extraction module is used to extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

[0079] The present invention also provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method.

[0080] Embodiment 1:

[0081] The present invention provides a visual SLAM method for a mobile robot in an indoor dynamic environment, comprising:

[0082] S1. Create an indoor moving object image detection dataset and perform migration training on the lightweight model Mobile Net-SSD;

[0083] S2, performing target detection on the original image of the mobile robot's surrounding environment collected by the positioning system, and obtaining the coordinates of the dynamic objects in the image, which are expressed in the form of a rectangular frame;

[0084] S3, based on the depth map information corresponding to the original image, combined with the rectangular frame coordinates, image segmentation is performed along the contour of the dynamic object to obtain a moving object mask;

[0085] S4, passing the mask of the moving object into the feature point extraction module, after extracting the feature points in the original image, removing the dynamic feature points based on the mask and retaining the static feature points;

[0086] Create an indoor moving object image detection dataset and perform migration training on the lightweight model Mobile Net-SSD; the moving object image detection dataset contains moving objects in the working scene of the mobile robot, including people, cats and dogs. Step S1 includes:

[0087] In step S1 of making an indoor moving object image detection dataset, it is first necessary to collect moving object images in the working scene of the mobile robot. These images should cover various dynamic objects that may affect the robot's navigation, such as pedestrians, cats, dogs and other mobile robots. In order to efficiently annotate these images, a graphical interface annotation tool such as labelimg can be used. This tool allows users to mark objects in the image through simple interactions and generate annotation information, such as rectangular box coordinates and category labels. After the annotation is completed, these images and annotation information will be stored in different folders, and together constitute a dataset for training deep learning models.

[0088] Furthermore, the collected images can be manually annotated. Manually screen the collected data, select the target in the image and use the Labelimg image annotation tool to annotate the location of the target object in the image, including the object category. <class>and location<x_center> ,<y_center> , <width>and <height>, and place the images and annotation results in the VOC (Visual Object Classes) dataset format.

[0089] Next, use a deep learning framework such as Caffe to perform transfer training on the MobileNet-SSD model according to the preset training parameters. Transfer training refers to using a model pre-trained on a similar task and fine-tuning it to adapt to the new dataset.

[0090] Furthermore, the caffe (Convolution Architecture For Feature Extraction) deep learning framework is used for training. The model and weights are downloaded and their addresses are written in the command of the training file. A series of deep learning parameters are set for migration training. Some parameters and their values ​​are as follows: the learning rate (base_lr) is set in the range of 0.0001-0.005, the weight decay parameter (weight_decay) is set to 0.00005, etc. The parameter settings are mainly based on the default parameters, which can be fine-tuned according to needs and the actual situation of the device to speed up the training.

[0091] Furthermore, target detection is performed on the original image of the mobile robot's surrounding environment collected by the system to obtain the coordinates of dynamic objects in the image, which are expressed in the form of a rectangular frame; the original image is an RGB image, and step S2 includes:

[0092] The original image is input into the transfer-trained Mobile Net-SSD neural network. The network uses the model weights trained in step S1 to perform image detection, and outputs information such as the category, confidence, and rectangular frame coordinates of each detected moving object. By setting a threshold, detection results with confidence levels higher than the threshold can be screened out to obtain the coordinates of dynamic objects.

[0093] Furthermore, an image of the surrounding environment is obtained from the visual sensor and input into the Mobile Net-SSD neural network. The network uses the model weights trained in step 1 to perform image detection and outputs information such as the type, confidence level, and rectangular frame coordinates of the detected target. The coordinates of the prior moving objects are further screened and obtained by the confidence level. The growth point coordinates are designed according to the type of moving objects to prepare for step 3. The growth point coordinates of humans are one quarter below the center of the rectangular frame, the growth point coordinates of cats and small dogs are the center of the rectangular frame, and the growth point coordinates of large dogs can be designed to be one quarter above the center of the rectangular frame.

[0094] Based on the depth map information corresponding to the original image, image segmentation is performed along the contour of the dynamic object in combination with the rectangular frame coordinates to obtain a moving object mask. Step S3 includes:

[0095] Get a depth map that is synchronized with the original RGB image. This can be achieved by using a depth camera mounted on a mobile robot. Then, use the rectangular frame coordinates obtained from Mobile Net-SSD in combination with the region growing algorithm to segment the contours of dynamic objects from the depth map. The region growing algorithm is a seed point-based image segmentation technique that starts from a predefined seed point and gradually expands to the neighborhood until specific conditions are met. In this process, the seed point is obtained using the rectangular frame coordinates. Taking pedestrians as an example, the seed point is one quarter of the height below the center of the rectangular frame. The algorithm identifies and segments dynamic objects based on local features in the depth map, such as the continuity of depth values. After the segmentation is completed, the generated binary mask mask will be used to distinguish between static and dynamic pixels in the image.

[0096] Furthermore, based on the depth map corresponding to the original image, the growth points in step 2 are used to perform regional growth within the coordinate range of the same target rectangular frame. The threshold size of the region growing algorithm is set to a depth information difference between 8 and 15, which can be adjusted according to the actual system operation situation. The two algorithms are combined to complete the moving object contour segmentation, and the segmentation result of each frame of the image is made into a binary mask. The pixels with a value of 1 in the binary mask are semantically static object pixels, and the pixels with a value of 0 are semantically dynamic object pixels.

[0097] Furthermore, the generated moving object mask is input into the feature point extraction module. In the SLAM system, feature point extraction is a key step to identify salient points from the image that can be used for tracking and matching. After extracting the feature points in the original image, the moving object mask is used to remove the feature points belonging to dynamic objects and only retain the feature points of static objects. This step is crucial because the SLAM system relies on stable feature points for accurate positioning and map construction.

[0098] In general, the present invention first creates an indoor moving object image detection data set for training the target detection algorithm; then performs target detection on the RGB original image obtained by the positioning system to obtain the prior dynamic object coordinates; then, based on the depth map corresponding to the original image and combined with the prior dynamic object coordinates, segmentation is performed along the dynamic object contour to create a binary mask of the moving object; finally, image feature points are extracted, and the dynamic feature points are eliminated based on the mask, while the static feature points are retained.

[0099] like Figure 1 As shown, the present invention is a dynamic SLAM method based on joint optimization of semantics and image segmentation, comprising:

[0100] In the flowchart, the lightweight model Mobile Net-SSD that has completed migration training is used to obtain the coordinates of dynamic objects, which are passed into the region growing algorithm to complete segmentation along the contours of the dynamic objects. Combined with morphological processing, the mask is ensured to accurately remove dynamic objects. The mask is passed to the feature point extraction module, and the numerical value on the binary mask is used to verify whether the feature points are retained to complete the dynamic feature point removal.

[0101] The key to the dynamic SLAM problem is how to accurately detect dynamic objects and properly handle dynamic objects. Image semantic segmentation takes image streams as input, outputs classified semantic information of objects in the scene, and detects the edges of objects at the same time. However, semantic segmentation is not perfect. First, semantic segmentation cannot run in real time; second, the edge detection of objects is inaccurate, and some edges will be regarded as background. Therefore, in response to these three problems, the lightweight model MobileNet-SSD is used to replace image semantic segmentation to obtain the prior coordinates of dynamic objects in real time, and segmentation is completed along the contours of dynamic objects based on depth maps and region growing algorithms. Combined with morphological processing, it can ensure that dynamic objects are removed from the mask.

[0102] The present invention adopts a tightly coupled approach of target detection and region growing algorithm, takes the result of target detection as prior information, and integrates it into the operation process of region growing algorithm, so as to further improve the robustness and accuracy of positioning.

[0103] To obtain better target detection results, the lightweight model Mobile Net-SSD needs to be migrated and trained before using it:

[0104] First, collect images containing moving objects in the mobile robot working scene. In this example, the moving objects are people. Second, complete the labeling of the data set through labeling tools such as labelimg. Figure 3 As shown in the figure, the format of the dataset is slightly different according to the deep learning framework you actually use. Finally, use the completed dataset to perform migration training on the lightweight model Mobile Net-SSD. The training parameters can use the default parameters of the model. If you want to achieve better training results, you can adjust the parameters yourself.

[0105] Embodiment 2:

[0106] 1. Target detection dataset production:

[0107] The system was tested using the TUM (Technische München) dynamic dataset, so this target detection dataset uses images containing people selected from the TUM dataset, annotated using the labelimg annotation tool, and produces a VOC format dataset.

[0108] 2. Use the caffe framework to complete migration training

[0109] Put the dataset in the Caffe framework and train it using default parameters.

[0110] 3. Put the trained model into the modified dynamic SLAM directory and wait for it to be called.

[0111] 4. Use the experimental platform to run the test. The platform hardware configuration is: AMD Ryzen 7 5800h CPU, 8 cores and 16 logical processors, frequency is 3.20GHz, 16GB memory, and the operating system is Ubuntu 20.04 distribution.

[0112] 5. The present invention uses ORB-SLAM2 as the comparison benchmark, and also uses the following dynamic SLAM algorithms for comparative experiments on the TUM dataset:

[0113] Dyna-SLAM: In the RGB-D dataset, geometric segmentation is used to supplement the mask produced by semantic segmentation, and the union of the two segmentation results is taken as the final segmentation result.

[0114] YOLO-SLAM: Use YOLOv7 for target detection and mask creation in RGB-D dataset.

[0115] The high-dynamic sequences of the TUM dataset are mainly fr3_walking_xyz, fr3_walking_static, fr3_walking_rpy, and fr3_walking_halfsphere, which represent four camera movement modes: (1) xyz means the camera moves along the xyz axis, (2) the camera is manually kept static, (3) rpy means the camera rotates on the xyz axis, and (4) halfsphere means the camera moves on a hemisphere with a diameter of 1 meter.

[0116] Evaluation criteria:

[0117] Absolute Trajectory Error (ATE) and Relative Pose Error (RPE), where the absolute trajectory error is called the absolute pose error (APE) in the EVO evaluation tool. The absolute trajectory error is used to describe the error between the estimated value and the true trajectory value of the pose, and can be used to evaluate the global consistency between the estimated pose and the true pose; the relative pose error is used to describe the difference between the true pose change and the estimated pose change within two identical timestamps (i.e., the same period of time), reflecting the drift of the visual SLAM system.

[0118] The calculation formula for improving system accuracy is:

[0119]

[0120] Among them, Imp represents the degree of improvement, O represents the error of ORB-SLAM2, and M represents the error of MSSD-SLAM.

[0121]

[0122] Table 1

[0123]

[0124] Table 2

[0125]

[0126] Table 3

[0127]

[0128] Table 4

[0129]

[0130] Table 5

[0131] In the above Tables 1 to 5, Table 1 is a comparison of the absolute trajectory error ATE of ORB-SLAM2, YOLO-SLAM and this system, Table 2 is a comparison of the relative trajectory error RPE of ORB-SLAM2, YOLO-SLAM and this system, and Table 3 is a comparison of the absolute trajectory error ATE of ORB-SLAM2, YOLO-SLAM, this system and Dyna SLAM system. Table 4 is a comparison of the relative trajectory error RPE of ORB-SLAM2, YOLO-SLAM, this system and Dyna SLAM system. Table 5 is the average time required for each frame when ORB-SLAM2, YOLO-SLAM, this system and Dyna SLAM system are running, in seconds. It can be seen from Tables 1 to 4 that the system proposed by the present invention has greatly improved the original system ORB-SLAM2 and can work stably in dynamic scenes. In the comparative experiment with advanced dynamic SLAM, the positioning accuracy of the system of the present invention is slightly worse, but it is sufficient to work in dynamic scenes.

[0132] The absolute trajectory error of the MSSD-SLAM system in the two sequences fr3_walking_xyz and fr3_walking_static also improved by more than 90%, showing a similar improvement effect to YOLO-SLAM.

[0133] In the fr3_walking_half sequence, the standard deviation is improved by 83.74% and the root mean square error is improved by 90.52%.

[0134] In the fr3_walking_rpy sequence, the standard deviation is improved by 49.83% and the root mean square error is improved by 56.92%.

[0135] It can be seen from Table 5 that the real-time performance of the system of the present invention is the highest in the dynamic SLAM system.

[0136] In summary, the present invention takes into account both positioning accuracy and operating speed, and can ensure stable operation in dynamic scenes while having high real-time performance. The present invention uses migration training to improve the recognition accuracy of the lightweight model Mobile Net-SSD for moving objects in the image; uses the region growing algorithm to complete the contour segmentation of moving objects in the image; the present invention improves the positioning accuracy and operating stability of the mobile robot visual SLAM system in indoor dynamic environments; the present invention can run at a speed of 65 milliseconds per frame of image only on the CPU.

[0137] The above is a detailed introduction to a visual SLAM method, system and readable storage medium for a mobile robot provided in the embodiments of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0138] For example, certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that hardware manufacturers may use different nouns to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but use differences in the functions of components as the criteria for distinction. As mentioned throughout the specification and claims, "including" and "comprising" are open-ended terms, so they should be interpreted as "including / including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect. The subsequent description of the specification is a preferred embodiment of the present application, but the description is for the purpose of illustrating the general principles of the present application, and is not used to limit the scope of the present application. The scope of protection of the present application shall be determined by the definition of the attached claims.

[0139] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or system. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0140] It should be understood that the term "and / or" used in the present invention is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0141] The above description shows and describes several preferred embodiments of the present application, but as mentioned above, it should be understood that the present application is not limited to the form disclosed in the present invention, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the application concept described in the present invention through the above teachings or the technology or knowledge in the relevant field. The changes and modifications made by those skilled in the art do not deviate from the spirit and scope of the present application, and should be within the scope of protection of the claims attached to the present application.< / height> < / width> < / class>

Claims

1. A visual SLAM method for a mobile robot, characterized in that: The method comprises the following steps: S1: Collect original image data, classify and annotate the image data, and obtain the original image data set; S2: Train the original image dataset through deep learning to obtain a migration training image dataset; S3: Based on the migration training dataset for pedestrians in indoor dynamic environments, a lightweight dynamic object target recognition algorithm framework is constructed to obtain the coordinate rectangle of the dynamic object; S4: Obtain the depth map of the original image data, design a region growing algorithm based on depth information, and use the dynamic object coordinate rectangle to build a dynamic object mask generation and image segmentation module: S5: Extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

2. The method according to claim 1, characterized in that The original image data acquisition in S1 is performed by a mobile robot to collect data multiple times in the working scene. The classification and labeling process in S1 is specifically: labeling and classifying the objects in the image through a graphical interface labeling tool, generating category labels and labeling labels, wherein the labeling labels include but are not limited to coordinate information and coordinate rectangular boxes.

3. The method according to claim 1, characterized in that Specifically, S2 includes: performing migration training on the Mobile Net-SSD model according to preset training parameters through a deep learning framework, wherein the deep learning framework includes but is not limited to a caffe deep learning framework.

4. The method according to claim 3, characterized in that The deep learning training process is specifically as follows: download the model and weights and write their addresses in the command of the training file, set a series of deep learning parameters for migration training, some parameters and their values ​​are as follows: set the learning rate base_lr in the range of 0.0001-0.005, and set the weight decay parameter weight_decay to 0.00005.

5. The method according to claim 4, characterized in that Specifically, S3 inputs the original image data into the Mobile Net-SSD neural network, performs image detection through the model weights trained by S2, outputs the type, confidence and rectangular frame coordinate information of the detected target, further filters and obtains the coordinates of the prior moving object through the confidence level, and designs the growth point coordinates according to the type of the moving object.

6. The method according to claim 5, characterized in that Specifically, S4 includes: based on the depth map corresponding to the original image, performing regional growth within the same target rectangular frame coordinate range through the growth points in S3, setting the threshold of the regional growing algorithm, performing contour segmentation on the moving object, and making the segmentation result of each frame image into a binary mask, wherein the pixels with a value of 1 in the binary mask are semantically static object pixels, and the pixels with a value of 0 are semantically dynamic object pixels.

7. The method according to claim 1, characterized in that The S5 is specifically as follows: inputting the generated moving object mask into the ORB feature point extraction module, using the moving object mask to remove the feature points belonging to the dynamic object, retaining the feature points of the static object, and completing the precise positioning and map construction of the visual SLAM.

8. The method according to claim 1, characterized in that The S5 is specifically as follows: image feature matching is performed based on ORB-SLAM2 by using information of instance segmentation mask, ORB feature points are matched by similarity of feature point descriptors of two adjacent frames of images, and multiple sets of matched feature point pairs are obtained through feature matching; The feature point matching requirements are: Where I(x) is the grayscale of any point on the circumference, I(p) is the grayscale of the center of the circle, and ε d is the threshold of gray value difference. If N is greater than the given threshold, generally three quarters of the surrounding circle points, then p is considered to be a feature point.

9. A visual SLAM system for a mobile robot, based on the method according to any one of claims 1 to 8, characterized in that: The system comprises: The data acquisition and processing module is used to collect original image data, classify and annotate the image data, and obtain the original image data set; A data learning training module is used to train the original image data set through deep learning to obtain a migration training image data set; A dynamic object acquisition module is used to input the original image data into the migration training image data set to obtain the coordinate rectangular frame of the dynamic object; The image segmentation processing module is used to obtain the depth map of the original image data, perform image segmentation on the dynamic object through the coordinate rectangular frame of the dynamic object, and obtain the moving object mask; The feature point extraction module is used to extract feature points from the original image through the moving object mask, remove dynamic feature points, retain static feature points, and build a map with precise positioning.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

Images