Semantic vision SLAM (Simultaneous Localization and Mapping) method based on depth mask segmentation in dynamic environment

Abstract

The invention provides a semantic vision SLAM method based on depth mask segmentation in a dynamic environment. The semantic vision SLAM method comprises the following steps: inputting an RGB image and a depth image into a tracking thread for preprocessing; inputting the preprocessed RGB image into a semantic segmentation thread for precise detection and segmentation of a scene target, obtaining a semantic tag of each pixel, and generating a segmentation mask; in combination with the depth information consistency of the preprocessed depth image, correcting and optimizing the mask to obtain a mask image; inputting each frame of corrected mask image into a dynamic feature screening thread, and determining dynamic feature points of potential dynamic objects; epipolar constraints are constructed through matching points in a background region of the RGB image, and dynamic feature points in a high dynamic object and a potential dynamic object in the RGB image are eliminated; and merging and splicing local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated through a point cloud mapping thread, and constructing a global three-dimensional point cloud map without the dynamic objects. According to the method, the mapping precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing visual SLAM positioning technology, and in particular to a semantic visual SLAM method based on depth mask segmentation in a dynamic environment. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is the process of using mobile carriers equipped with sensors such as cameras and lidar to locate and map themselves in unknown environments. In recent years, visual SLAM (VSLAM) technology using cameras as sensors has become a research hotspot in the fields of intelligent mobile robots, unmanned driving, and virtual reality because it provides rich environmental information and has low costs.

[0003] At present, some advanced visual SLAM algorithms have achieved satisfactory results, such as MonoSLAM, ORB-SLAM series, SVO, etc. Traditional visual SLAM algorithms are mainly based on the assumption of static environment and only allow a few dynamic points, but there are complex dynamic objects in the real environment, which brings challenges to positioning and mapping: dynamic objects introduce a large number of abnormal points, destroy the corresponding matching relationship, and make it difficult to locate the camera during tracking; these dynamic object information will be retained in the constructed map, and due to their position changes, the map ghosting will increase, affecting subsequent navigation and other tasks.

[0004] In the prior art, dynamic points are mainly removed by two methods: traditional geometry-based methods and deep learning-based methods. Traditional geometry methods use optical flow, RANSAC algorithm, triangulation or epipolar geometry constraints to remove dynamic feature points. However, the visual SLAM system based on traditional geometry methods has low hardware requirements, but poor stability, and it is difficult to identify dynamic objects and completely remove dynamic points, resulting in large errors. The effectiveness is limited and can only handle situations where there are a small number of dynamic points in the field of view. The deep learning-based method mainly extracts semantic information from the input image and generates pixel-by-pixel labels and masks for each identified object. It performs well in feature extraction, semantic map construction and dynamic object detection, but the mask boundaries of the segmented dynamic objects are not clear, resulting in some dynamic feature points being used for camera pose calculations, affecting positioning accuracy.

[0005] YOLOv8s-seg is a semantic segmentation network that takes into account both accuracy and real-time performance. However, there are still some problems when it is applied to dynamic visual SLAM algorithms: the mask boundaries of dynamic objects are unclear during semantic segmentation, and ghost images are easily generated during the mapping process; although the YOLO algorithm is fast, it is easy to miss objects when detecting them in the image, affecting the subsequent matching accuracy; semantic segmentation can only determine the category of the target, but cannot determine the motion state of the object. Summary of the invention

[0006] In view of this, the present invention provides a semantic visual SLAM method based on depth mask segmentation in a dynamic environment to solve the above problems.

[0007] The present invention provides a semantic visual SLAM method based on depth mask segmentation in a dynamic environment, comprising: inputting an RGB image and a depth image into a tracking thread for preprocessing; inputting the preprocessed RGB image into a semantic segmentation thread for accurate detection and segmentation of scene targets, obtaining a semantic label for each pixel, and generating a segmentation mask, wherein the scene targets can be divided into three categories: static objects, high-dynamic objects, and potential dynamic objects; combining the depth information consistency of the preprocessed depth image, correcting and optimizing the mask to obtain a mask image; inputting each frame of the corrected mask image into a dynamic feature screening thread to determine the dynamic feature points in the potential dynamic objects; constructing epipolar constraints through matching points in the background area of ​​the RGB image, eliminating the dynamic feature points in the high-dynamic objects and potential dynamic objects in the RGB image, and using the static matching points for camera positioning; merging and splicing the local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated through a point cloud mapping thread to obtain a new point cloud, and constructing a global three-dimensional point cloud map without dynamic objects.

[0008] In another implementation of the present invention, the inputting the RGB image and the depth image into the tracking thread for preprocessing includes: performing ORB feature point extraction preprocessing on the RGB image; and performing depth value filtering preprocessing on the depth image.

[0009] In another implementation of the present invention, the result of the accurate detection and segmentation is expressed as:

[0010]

[0011] Among them, S k represents the instance segmentation result of the kth frame, They respectively represent the category and mask of the i-th prior object segmented in the current k-th frame of RGB image.

[0012] In another implementation of the present invention, the mask is corrected and optimized in combination with the depth information consistency of the preprocessed depth image to obtain the mask image, including: in the current frame, for each prior object in the RGB image, its mask is combined with the preprocessed depth image, and the average depth in each mask area is calculated; each category in the RGB image is traversed to obtain the current pixel value and its corresponding depth information, if the difference between the depth of the current pixel and the average depth of the area exceeds a preset threshold, the part is removed from the mask; if there is an inaccuracy in the edge of the depth image mask, a two-pixel dilation operation is performed to generate a processed mask image.

[0013] In another implementation of the present invention, the average depth is expressed as:

[0014]

[0015] Where N represents the number of all pixels of object i in the mask area, p j Indicates the depth value corresponding to each pixel in the area.

[0016] In another implementation of the present invention, it also includes: setting a similarity threshold based on the distance information between the center point of the historical frame and the current frame; performing missed detection judgment based on the similarity threshold to obtain a judgment result; if the judgment result is that a missed detection has occurred, projecting the object of the previous frame into the current frame for compensation processing.

[0017] In another implementation of the present invention, the step of inputting each corrected mask image frame into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic object includes: inputting each corrected mask image frame into the dynamic feature screening thread; traversing all pixels of the current frame mask image; traversing the neighboring pixels around the current pixel and calculating their distances from the current pixel; if the neighboring pixel is an edge pixel of the potential dynamic object mask and does not belong to the same dynamic object as the current pixel, then updating and calculating the minimum value of the edge pixel distances between the two; if the minimum value is less than a preset edge distance threshold and the depth difference between the two pixels is less than a preset pixel depth threshold, then it indicates that interference has occurred and the feature points in the potential dynamic object are regarded as dynamic feature points.

[0018] On the other hand, the present invention provides a semantic visual SLAM system based on depth mask segmentation in a dynamic environment, including: an image preprocessing module: used to input RGB images and depth images into a tracking thread for preprocessing; a mask image generation module: used to input the preprocessed RGB images into a semantic segmentation thread for accurate detection and segmentation of scene targets, obtain a semantic label for each pixel, and generate a segmentation mask, wherein the scene targets can be divided into three categories: static objects, high dynamic objects, and potential dynamic objects; in combination with the depth information consistency of the preprocessed depth image, the mask is corrected and optimized to obtain a mask image; a feature point classification module: used to input each frame of the corrected mask image into a dynamic feature screening thread to determine the dynamic feature points in the potential dynamic objects; epipolar constraints are constructed through matching points in the background area of ​​the RGB image to eliminate the dynamic feature points in the high dynamic objects and potential dynamic objects in the RGB image, and the static matching points are used for camera positioning; a map construction module: through a point cloud mapping thread, the local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated are merged and spliced ​​to obtain a new point cloud, and a global three-dimensional point cloud map is constructed by removing the dynamic objects.

[0019] Another aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the semantic visual SLAM method based on depth mask segmentation in a dynamic environment as described above are implemented.

[0020] Another aspect of the present invention provides a computer storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the semantic visual SLAM method based on depth mask segmentation in a dynamic environment are implemented as described above.

[0021] The semantic visual SLAM method based on deep mask segmentation in a dynamic environment of the present invention utilizes a semantic segmentation network to detect objects, proposes mask correction based on depth consistency and missed detection compensation based on relative displacement consistency, and realizes accurate segmentation of dynamic targets by semantic threads; classifies detected objects according to dynamic probability, judges the motion state of potential dynamic objects through a deep mask association algorithm, and eliminates dynamic feature points in combination with epipolar geometry constraints, thereby improving positioning accuracy; merges and removes local point clouds of dynamic targets to construct a global dense point cloud map, avoids interference from dynamic objects, presents static scenes, and improves mapping accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. By reading the detailed description of the following implementation, the advantages and benefits of the solutions become clear to those skilled in the art. The drawings are only used to illustrate the preferred implementation and are not considered to be limitations of the present invention. In the drawings:

[0023] Figure 1 The present invention is a flowchart of a semantic visual SLAM method based on depth mask segmentation in a dynamic environment according to an embodiment of the present invention.

[0024] Figure 2 The figure is a schematic diagram of the overall algorithm framework of an embodiment of the present invention.

[0025] Figure 3 The figure is a flowchart of a mask correction algorithm based on depth consistency according to an embodiment of the present invention.

[0026] FIG. 4( a ) is a schematic diagram of the correction effect of over-segmentation elimination during the mask correction process according to an embodiment of the present invention.

[0027] FIG. 4( b ) is a schematic diagram of the under-segmentation compensation correction effect during the mask correction process according to an embodiment of the present invention.

[0028] FIG. 5( a ) is a schematic diagram of the compensation effect of a bottle in the potential dynamic object missed detection compensation based on relative displacement invariance according to an embodiment of the present invention.

[0029] FIG. 5( b ) is a schematic diagram of the compensation effect of a chair in the potential dynamic object missed detection compensation based on relative displacement invariance according to an embodiment of the present invention.

[0030] Figure 6 A schematic diagram of the effect of dynamic feature point recognition based on depth mask association according to an embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of a visual comparison of positioning errors between the method of the present invention and ORB-SLAM2 according to an embodiment of the present invention.

[0032] Figure 8 A schematic diagram comparing the semantic mapping of the method of the present invention and ORB-SLAM2 according to an embodiment of the present invention.

[0033] Fig. 9 It is a schematic diagram of the dynamic feature point elimination results of the method of the present invention and ORB-SLAM2 according to an embodiment of the present invention.

[0034] FIG. 10( a ) is a schematic diagram of a point cloud constructed after removing the person information from the laboratory map according to an embodiment of the present invention.

[0035] FIG10( b ) is a schematic diagram of a point cloud constructed after removing the person information from the elevator map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0037] Figure 1 A flowchart of a semantic visual SLAM method based on depth mask segmentation in a dynamic environment provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, this embodiment mainly includes:

[0038] S101, inputting the RGB image and the depth image into the tracking thread for preprocessing.

[0039] For example, Figure 2 As shown, the present invention is improved on the basis of the ORB-SLAM2 system and consists of four threads: a semantic segmentation thread, a dynamic feature screening thread, a camera tracking thread, and a point cloud mapping thread, so that the system can accurately identify moving objects in the environment.

[0040] S102: Input the preprocessed RGB image into the semantic segmentation thread to perform accurate detection and segmentation of scene targets, obtain the semantic label of each pixel, and generate a segmentation mask, wherein the scene targets can be divided into three categories: static objects, highly dynamic objects, and potentially dynamic objects.

[0041] Exemplarily, the semantic segmentation model of YOLOv8 is used to detect objects in the scene, where the average precision (mAP) of Yolov8s-seg detection is 44.6%, the accuracy of the mask is 36.8%, and the speed when running on the CPU using the ONNX format is 155.7ms. Compared with other YOLO series models, it provides better detection and segmentation effects while maintaining a relatively fast processing speed.

[0042] Preferably, the semantic segmentation thread is implemented based on Pytorch and trained using the coco128-seg.yaml subset in the MS COCO dataset. Objects can be divided into 25 categories, and all types of objects detected by YOLOv8s-seg are divided into three categories according to the potential motion state of the object:

[0043] (1) Static objects, such as walls, televisions, and tables.

[0044] (2) Highly dynamic objects, such as people and animals.

[0045] (3) Potentially dynamic objects, such as mouse, chair, book, keyboard, etc.

[0046] Among them, potentially dynamic objects are those that are usually static but may become dynamic under the influence of other objects (such as people).

[0047] Furthermore, based on these prior semantic information, the feature points are classified so that in the next step, dynamic feature points can be identified and removed by judging the motion state.

[0048] Static ORB feature points are selected and the initial camera pose is restored to improve the accuracy of pose evaluation in the tracking thread; on the other hand, in the map reconstruction thread, pixel-level semantic knowledge is integrated into the corresponding 3D points to reconstruct a semantically dense point cloud.

[0049] In dynamic scenes, the positioning accuracy of the feature point-based visual SLAM system has an important dependence on the accuracy of the semantic segmentation network.

[0050] S103 . Based on the consistency of the depth information of the preprocessed depth image, the mask is corrected and optimized to obtain a mask image.

[0051] For example, it is found in the experiment that the segmentation accuracy of Yolov8s-seg fails to meet the accuracy requirements of the visual SLAM system. Specifically, the mask boundary in the segmentation result is blurred and inaccurate, and the boundary range of the object cannot be accurately captured. In addition, although the YOLO system itself is fast, it is prone to missed detection, which affects the SLAM system's understanding of the scene and positioning accuracy. Therefore, the present invention proposes a mask correction algorithm based on depth consistency and a missed detection compensation algorithm based on relative displacement consistency.

[0052] S104: Input each frame of the corrected mask image into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic object.

[0053] S105, constructing epipolar constraints through matching points in the background area of ​​the RGB image, eliminating dynamic feature points in high dynamic objects and potential dynamic objects in the RGB image, and using static matching points for camera positioning.

[0054] S106. The local point cloud corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated is merged and spliced ​​through the point cloud mapping thread to obtain a new point cloud, and a global three-dimensional point cloud map is constructed without dynamic objects.

[0055] The semantic visual SLAM method based on deep mask segmentation in a dynamic environment of the present invention utilizes a semantic segmentation network to detect objects, proposes mask correction based on depth consistency and missed detection compensation based on relative displacement consistency, and realizes accurate segmentation of dynamic targets by semantic threads; classifies detected objects according to dynamic probability, judges the motion state of potential dynamic objects through a deep mask association algorithm, and eliminates dynamic feature points in combination with epipolar geometry constraints, thereby improving positioning accuracy; merges and removes local point clouds of dynamic targets to construct a global dense point cloud map, avoids interference from dynamic objects, presents static scenes, and improves mapping accuracy.

[0056] In another implementation of the present invention, the RGB image and the depth image are input into the tracking thread for preprocessing, including: performing ORB feature point extraction preprocessing on the RGB image; and performing depth value filtering preprocessing on the depth image.

[0057] For example, Figure 3 As shown, the depth image is read and preprocessed by filtering the depth value. Since the accuracy of the depth camera decreases as the distance between the object and the camera increases, and the unstable depth value will cause stratification in the subsequent point cloud stitching, filtering processing is required in the preprocessing stage.

[0058] In another implementation of the present invention, the result of accurate detection and segmentation is expressed as:

[0059]

[0060] Among them, S k represents the instance segmentation result of the kth frame, They respectively represent the category and mask of the i-th prior object segmented in the current k-th frame of RGB image.

[0061] Exemplarily, YOLOv8s-seg distinguishes different instances in a scene and obtains depth information and position information of each prior object in the environment.

[0062] Let F k represents the RGB image of the current k-th frame, D k represents the depth image of the kth frame, S k Represents the instance segmentation result of the kth frame:

[0063]

[0064] in, Respectively represent F k The category and mask of the i-th prior object segmented.

[0065] Since the segmented object is in motion, the mask range in the current frame k It cannot accurately represent the full range of dynamic objects and may suffer from over-segmentation and under-segmentation.

[0066] In another implementation of the present invention, the mask is corrected and optimized in combination with the depth information consistency of the preprocessed depth image to obtain a mask image, including: in the current frame, for each prior object in the RGB image, its mask is combined with the preprocessed depth image to calculate the average depth in each mask area; traverse each category in the RGB image to obtain the current pixel value and its corresponding depth information, if the difference between the depth of the current pixel and the average depth of the area exceeds a preset threshold, the part is removed from the mask; if there is an inaccuracy in the edge of the depth image mask, a two-pixel dilation operation is performed to generate a processed mask image.

[0067] Exemplarily, using the depth image information, the segmentation range is precisely refined and corrected to achieve accurate segmentation of dynamic objects at the semantic level in the new thread, as follows:

[0068] First, in the current frame k, for F k Each prior object in With the depth image D k Combined, calculate the average depth in each mask area And record it in the history, the formula is expressed as:

[0069]

[0070] Among them, N represents the object i in the mask area The number of all pixels in the j Indicates the depth value corresponding to each pixel in the area.

[0071] Furthermore, each category in the image is traversed to obtain the current pixel value and its corresponding depth information. At the same time, a threshold for detecting depth anomalies is set, which is set to 0.5 here.

[0072] If the difference between the depth of the current pixel and the average depth of the area exceeds the set threshold, this part is removed from the mask and the mask is corrected:

[0073]

[0074] Finally, in case of inaccurate edge of the depth image mask, a two-pixel dilation operation is performed to generate the processed mask image As shown in Figure 4(a) and Figure 4(b), the mask image is a binary image. The pixel value of 255 represents the foreground (white area) and other values ​​represent the background (black area). Figure 4(a) removes the background from the foreground, and Figure 4(b) completes the missed part of the human leg. The mask that integrates the depth information makes the segmentation of dynamic objects more accurate.

[0075] The mask correction algorithm based on depth consistency is based on the assumption that the feature points on a person have little difference in depth value. After the semantic segmentation obtains the image object mask, it matches it with the depth image and calculates the difference between the average depth of the current pixel and the corresponding image. If it exceeds the threshold, it will be removed, and if it is less than the threshold, it will be supplemented. By using the method of depth information, compared with previous technologies, the segmentation accuracy of the semantic thread for dynamic targets is improved, laying a good foundation for the dynamic feature screening thread.

[0076] In another implementation of the present invention, the average depth is expressed as:

[0077]

[0078] Where N represents the number of all pixels of object i in the mask area, p j Indicates the depth value corresponding to each pixel in the area.

[0079] In another implementation of the present invention, it also includes: setting a similarity threshold based on the distance information between the center point of the historical frame and the current frame; performing missed detection judgment based on the similarity threshold to obtain a judgment result; if the judgment result is that a missed detection has occurred, projecting the object of the previous frame into the current frame for compensation processing.

[0080] For example, since the YOLO semantic segmentation algorithm usually faces the problem of missed detection of objects in the environment, in the case of missed detection, the difference between two adjacent images will cause the number of correct matches of feature points to change dramatically, resulting in system instability, making the subsequent mapping based on semantic information ineffective. To solve this problem, a missed detection compensation algorithm based on relative displacement invariance is proposed, which is based on a reasonable assumption: the time difference between two adjacent frames in a video sequence is very short.

[0081] The average processing time for one frame is only 0.04ms, and the change in the displacement of dynamic objects in the room does not change much. This assumption states that the environment in which this system is applicable is mainly indoor environment.

[0082] In visual SLAM, video frames arrive in chronological order, and the detection result of the previous frame can be used to predict the next detection result based on the projection relationship, thereby avoiding missed detection or false detection of the next key frame. Therefore, the displacement invariance of adjacent frames can be used to further improve the segmentation accuracy of Yolov8s-seg.

[0083] Provide the center point coordinates of the detection box And its width w and height h, assuming that the i-th object detection box in the k-th frame The coordinates of the upper left corner (x1, y1), the coordinates of the lower right corner (x2, y2), and the coordinates of the center point for:

[0084]

[0085] Initialize, traverse all the mask center points of the current frame from the second frame, set the similarity threshold, check whether the mask center point in the current frame is similar to a certain center point coordinate in the historical frame, if not similar, determine the specific missed object and mark its corresponding serial number.

[0086] In order to determine the pixel value of the missed detection target, the specific missed detection compensation algorithm is as follows:

[0087] Algorithm 1, missed detection compensation algorithm:

[0088] Input: current frame k, category Mask Including center point coordinates Historical frame k-1 and the coordinates of the center points of its objects

[0089] Output: missed point index, projection compensation.

[0090] Step 1: Traverse all mask center points in the current frame and set the similarity threshold, τ = 20.

[0091] Step 2: That is, to find similar points, the loop is interrupted; otherwise, the center point of the current frame is recorded as a missed point, and the missed point index is output.

[0092] Step 3: if (missed detection occurs), the object in the previous frame is projected into the current frame for compensation processing, that is,

[0093] if If the coordinates extend beyond the boundaries of the current frame k, it is inferred that the object has moved outside the camera coordinates and is not considered lost.

[0094] As shown in Figure 5(a) and Figure 5(b), Figure 5(a) shows the compensation effect of the bottle that was missed in the semantic segmentation, and Figure 5(b) shows the compensation effect of the chair that was missed. Figure 7 This shows that this method makes up for the detection failure of the YOLO algorithm in some frames to a certain extent, and helps the tracking thread to better track dynamic objects.

[0095] The missed detection compensation algorithm based on relative displacement consistency is based on the assumption that the time difference between two adjacent frames in the video sequence is very short, the average processing time for one frame is only 0.04ms, and the displacement change of the object is not large. After segmentation, the center points of each mask are calculated and stored; then the current frame is detected to determine the distance between the center point of each mask in the current frame and the center point of the previous frame. If the distance is less than the threshold, a similar object is found; otherwise, it is not detected, it is judged as missed detection, and the previous storage is projected over. Compared with previous technologies, the segmentation accuracy of the semantic thread for dynamic targets is improved.

[0096] In another implementation of the present invention, each corrected mask image frame is input into a dynamic feature screening thread to determine dynamic feature points in potential dynamic objects, including: inputting each corrected mask image frame into a dynamic feature screening thread; traversing all pixels of the current frame mask image; traversing neighboring pixels around the current pixel and calculating their distances from the current pixel; if the neighboring pixel is an edge pixel of the potential dynamic object mask and does not belong to the same dynamic object as the current pixel, then updating and calculating the minimum value of the edge pixel distances between the two; if the minimum value is less than a preset edge distance threshold, and the depth difference between the two pixels is less than a preset pixel depth threshold, it indicates that interference has occurred, and the feature points in the potential dynamic object are regarded as dynamic feature points.

[0097] For example, since people's interaction with surrounding static objects will change their static state, if the potential dynamic object occupies most of the camera, the feature points will still not be matched correctly, and the moving objects will produce ghosting during the mapping process. A dynamic feature point recognition and elimination algorithm based on mask interaction is proposed to identify the interference of people on the surrounding potential dynamic objects, determine the motion state of the potential dynamic objects, and mark their feature points as a dynamic point set, as shown in the following formula:

[0098]

[0099] Among them, the current frame k, is a predefined high dynamic object mask in the image, is the potential dynamic object mask, C dynamic is a collection of dynamic objects, C static A collection of static objects.

[0100] The interaction judgment algorithm between highly dynamic objects and potentially dynamic objects is shown in Algorithm 2.

[0101] Algorithm 2, depth mask interaction algorithm:

[0102] Input: Corrected depth mask Includes high dynamic object masks Latent dynamic object masks Define the edge distance threshold as 5.0 and the pixel depth threshold as 0.3.

[0103] Output: The potential dynamic object motion state.

[0104] Step1: Traverse all the pixels of the current frame depth mask of.

[0105] Step2: If the current pixel is in the high-dynamic object mask then find the edge pixel of the nearest static object.

[0106] Step3: Traverse the neighboring pixels (including diagonal directions) around the current pixel and calculate their distances from the current pixel.

[0107] Step4: If the neighboring pixel is an edge pixel (pixel value is 122) in the potential dynamic object mask and does not belong to the same dynamic object as the current pixel, then update and calculate the minimum value min_distance of the edge pixel distances between the two.

[0108] Step5: If min_distance < threshold_distance and the depth difference between the two pixels is less than the set threshold, then it is determined that interference has occurred. Treat the potential dynamic object pixel as a dynamic object and mark it as the foreground (white).

[0109] As Figure 6 shown, assume the labels of the chairs are C(1) and C(2), and the label of the person is P. When the person pulls the right chair during movement, P acts on C(2), and the chair C(2) changes from a static background to a dynamic object. Mark the dynamic object in the depth image as the white foreground and map the semantic information of the ordinary chair to the current point cloud. The moving object is effectively removed, and the semantic information of the ordinary chair does not contaminate other environmental points.

[0110] After completing the dynamic detection, two sets of static matching points are obtained. The camera pose between the current frame and the historical frame can be estimated by solving the following least squares problem:

[0111]

[0112] Based on the size of the symmetric transfer polar line distance d and the set threshold τ, the moving points are judged. When the symmetric transfer polar line distance value d corresponding to a certain spatial point is greater than the set threshold τ, τ is taken as 0.5.

[0113]

[0114] Where D represents the set of dynamic matching points, and S represents the set of static matching points.

[0115] Based on the dynamic feature point recognition and elimination algorithm of depth mask interaction, the interaction between people and static objects around them will change their static state. If the difference between the edge distance and the corresponding depth distance is less than the threshold, it is judged that interference has occurred. Compared with previous technologies, considering the motion state of potential dynamic objects, dynamic and static feature points are finely distinguished to improve positioning accuracy.

[0116] In another implementation of the present invention, in order to verify the positioning precision and accuracy of the present invention in dynamic indoor scenes, simulation experiments were carried out using the RGB-D public dataset provided by the Technical University of Munich (TUM), Germany, and real scene test experiments were carried out using a handheld Intel RealSense D435i depth camera to capture laboratory images to simulate conditions in actual environments.

[0117] The experimental platform parameter configuration is as follows: the processor is Intel Core i5-4288U CPU, the graphics card GPU: RTX4090, RAM: 64G, and the environment configuration of the simulation experiment is shown in Table 1.

[0118] Table 1 Simulation experiment environment configuration table

[0119]

[0120] The experiment selected six sets of sequences from the TUM RGB-D Freiburg-3 dataset, including three sets of high-dynamic scene datasets: sitting_halfsphere, sitting_static, walking_halfsphere, and three sets of low-dynamic and high-dynamic scene datasets: walking_rpy, walking_static, and walking_xyz.

[0121] SLAM systems usually use absolute trajectory error (ATE) and relative pose error (RPE) to evaluate performance. The relative pose error is divided into relative translation and rotation drift errors. Root mean square error (RMSE), mean, median and standard deviation (SD) are used as evaluation indicators.

[0122] The present invention is improved on the basis of ORB-SLAM2. The positioning accuracy performance and trajectory estimation diagram of the two are compared. The table gives the specific values ​​of the positioning error of the present invention and ORB-SLAM2, and uses RMSE, Mean, Median, and SD to measure ATE and RPE. For each sequence, 10 experiments were conducted, and the average value of each result was taken to ensure the reliability of the algorithm. The calculation formula of the improvement is shown as follows:

[0123]

[0124] Wherein, τ represents the improvement of the parameter, α is the positioning accuracy value obtained from the present invention, and β is the positioning accuracy value obtained from ORB-SLAM2.

[0125] According to the ATE results in Table 2, in the four high-dynamic sequences, the present invention has a huge improvement in positioning accuracy compared to ORB-SLAM2, with an average improvement of RMSE and SD of 96.19% and 96.11% respectively. Since ORB-SLAM2 itself has very good performance in static environments, the improvement in positioning accuracy in sequences of low-dynamic environments is very limited, with an average improvement of RMSE and SD of 25.40% and 25.25% respectively.

[0126] In general, the present invention performs well in improving the ORB-SLAM2 algorithm. The system improves the accuracy in high-dynamic scenes and also performs well in low-dynamic scenes. These improvements effectively improve the positioning accuracy of the algorithm in dynamic scenes and enhance its robustness.

[0127] Table 2 Comparison of absolute trajectory errors between ORBSLAM2 and the present invention (m)

[0128]

[0129] The present invention provides a visual comparison of the estimated pose and actual pose of each frame between ORB-SLAM2 and the present invention under high dynamic sequences, such as Figure 7 As shown, the red line in each column represents the difference between the estimated trajectory and the ground truth. The longer the red line, the larger the error.

[0130] ORB-SLAM2 only uses geometric methods to process dynamic points. However, in high-dynamic sequences, when the number of dynamic points exceeds the number of static feature points, the loss function of the geometric constraint may not converge correctly, resulting in reduced positioning accuracy, such as Figure 7 As shown in (a)(b)(c)(d), the error with the true value is large.

[0131] The present invention uses the semantic segmentation deep mask algorithm to estimate the pose trajectory of each sequence as follows Figure 7 (e)(f)(g)(h) shows that there is only a slight drift relative to the true value trajectory. The main reason is that the elimination method in this paper deeply corrects the semantic segmentation YOLOv8s-seg to obtain a refined classification of dynamic objects. At the same time, it further distinguishes dynamic and static feature points based on semantic association combined with geometric constraint algorithm, thereby reducing the negative impact of dynamic feature points on geometric constraint algorithm and significantly improving positioning accuracy.

[0132] In order to evaluate the real-time performance of the method, the present invention uses a high dynamic sequence w_xyz, a total of 826 frames of images to complete the time-consuming analysis of the present invention, measures the average calculation time of each module, and compares it with ORB-SLAM2, DS-SLAM, Dyna-SLAM, RDS-SLAM, Blitz-SLAM, and YOLO-SLAM. The time comparison results are shown in Table 3. It can be seen that the running time of the proposed method is only 36.87ms, which is comparable to the running time of ORB-SLAM2, and is much better than other semantic segmentation methods in tracking efficiency, achieving a balance between efficiency and accuracy. The reason is that in this method, semantic segmentation and tracking threads are processed in parallel, and no additional processing time is added. At the same time, the improved method of the present invention requires very few computing resources and a small amount of calculation.

[0133] Table 3 Time comparison

[0134]

[0135]

[0136] In the high dynamic sequence fr3_w_xyz, the global point cloud images obtained by ORB-SLAM2 and the present invention for a certain frame of the scene are compared, as shown in FIG. Figure 8 As shown, most of the information of the two people is retained in the point cloud map constructed by ORB-SLAM2, forming ghosting, which covers the static objects in the point cloud map; the point cloud map established by the present invention completely establishes a three-dimensional map of static objects in the environment and removes the ghosting of people walking dynamically. The reason is that the present invention uses missed detection compensation and mask correction to completely remove dynamic objects.

[0137] In order to demonstrate the effectiveness and robustness of the present invention in the real world, experimental data of real scenes in the laboratory and elevator were collected using Intel Real-sense D435i camera. The size of the captured RGB image and depth image were both 640×480. In the experiment, the camera was moved by hand, and the camera would shake and rotate during the movement. During the shooting process, someone entered the camera's field of view, interfering with the camera's positioning and the construction of the point cloud map. Fig. 9The results of dynamic feature point elimination are shown in Figure 10. The first row is the feature extraction graph obtained by ORB-SLAM2, and the second row is the feature extraction graph obtained by the present invention. The comparison of the two sets of feature extraction graphs shows that the present invention effectively eliminates the negative impact of the moving crowd, and the area is quickly divided by the mask range of two people. Figure 10 (a) and Figure 10 (b) are point cloud images constructed after removing the person information, which completely eliminates the interference of people, static objects are clearly visible, and the geometric structure of the room is well preserved.

[0138] The technical solution of the present invention can retain static feature points as much as possible in two dynamic environments, ensure the accuracy of camera positioning, maintain a high degree of consistency in the estimated trajectory, and construct a clear point cloud image after removing dynamic objects. Data set experiments show that the technical solution of the present invention has good positioning accuracy and robustness in dynamic environments.

[0139] Another aspect of the present invention provides a semantic visual SLAM system based on depth mask segmentation in a dynamic environment, comprising:

[0140] Image preprocessing module: used to input RGB images and depth images into the tracking thread for preprocessing.

[0141] Mask image generation module: used to input the preprocessed RGB image into the semantic segmentation thread for accurate detection and segmentation of scene targets, obtain the semantic label of each pixel, and generate a segmentation mask. Scene targets can be divided into three categories: static objects, highly dynamic objects, and potentially dynamic objects; combined with the depth information consistency of the preprocessed depth image, the mask is corrected and optimized to obtain a mask image.

[0142] Feature point classification module: used to input each frame of the corrected mask image into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic objects; to build epipolar constraints through matching points in the background area of ​​the RGB image to eliminate dynamic feature points in high dynamic objects and potential dynamic objects in the RGB image, and static matching points are used for camera positioning.

[0143] Map construction module: Through the point cloud mapping thread, the local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated are merged and spliced ​​to obtain a new point cloud, and a global three-dimensional point cloud map is constructed without dynamic objects.

[0144] The semantic visual SLAM system based on deep mask segmentation in a dynamic environment of the present invention detects objects using a semantic segmentation network, proposes mask correction based on depth consistency and missed detection compensation based on relative displacement consistency, and realizes accurate segmentation of dynamic targets by semantic threads; classifies detected objects according to dynamic probability, judges the motion state of potential dynamic objects through a deep mask association algorithm, and eliminates dynamic feature points in combination with epipolar geometry constraints, thereby improving positioning accuracy; merges and removes local point clouds of dynamic targets to construct a global dense point cloud map, avoids interference from dynamic objects, presents static scenes, and improves mapping accuracy.

[0145] In another aspect of the present invention, an electronic device includes: a processor, a memory, a communication bus, and a communication interface.

[0146] in:

[0147] The processor, memory and communication interface communicate with each other through a communication bus.

[0148] Communication interface, used to communicate with other electronic devices or servers.

[0149] The processor is used to execute the program, and specifically can execute the steps of the semantic visual SLAM method based on depth mask segmentation in a dynamic environment in any of the above embodiments.

[0150] Specifically, the program may include program codes including computer operation instructions.

[0151] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0152] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0153] The program can be specifically used to enable the processor to execute the steps of implementing any one of the semantic visual SLAM methods based on depth mask segmentation in a dynamic environment described in the embodiment. The specific implementation of each step in the program can refer to the corresponding description in the steps and units performed by the semantic visual SLAM method based on depth mask segmentation in any of the above steps in a dynamic environment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described equipment and modules can refer to the corresponding process description in the aforementioned method embodiment.

[0154] The exemplary embodiments of the present application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the methods of the various embodiments of the present application.

[0155] The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0156] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

[0157] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0158] In the description of the present invention, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0159] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0160] It should be noted that although the specific embodiments of the present invention are described in detail in conjunction with the accompanying drawings, it should not be understood as limiting the scope of protection of the present invention. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still belong to the scope of protection of the present invention.

[0161] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be improper limitations of the embodiments of the present invention.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic visual SLAM method based on deep mask segmentation in a dynamic environment, characterized in that: include: Input the RGB image and depth image into the tracking thread for preprocessing; The preprocessed RGB image is input into the semantic segmentation thread for accurate detection and segmentation of scene objects, the semantic label of each pixel is obtained, and a segmentation mask is generated. The scene objects can be divided into three categories: static objects, highly dynamic objects, and potentially dynamic objects. Combining the depth information consistency of the preprocessed depth image, the mask is corrected and optimized to obtain a mask image; Inputting each frame of the corrected mask image into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic object; Epipolar constraints are constructed by matching points in the background area of ​​the RGB image to eliminate dynamic feature points in high dynamic objects and potential dynamic objects in the RGB image, and static matching points are used for camera positioning; Through the point cloud mapping thread, the local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated are merged and spliced ​​to obtain a new point cloud, and a global three-dimensional point cloud map is constructed without dynamic objects.

2. The method according to claim 1, characterized in that The RGB image and the depth image are input into the tracking thread for preprocessing, including: Performing ORB feature point extraction preprocessing on the RGB image; A depth value filtering preprocessing is performed on the depth image.

3. The method according to claim 1, characterized in that The results of the precise detection and segmentation are expressed as: Among them, S k represents the instance segmentation result of the kth frame, They respectively represent the category and mask of the i-th prior object segmented in the current k-th frame of RGB image.

4. The method according to claim 3, characterized in that The step of combining the depth information consistency of the preprocessed depth image to correct and optimize the mask to obtain the mask image includes: In the current frame, for each prior object in the RGB image, its mask is combined with the preprocessed depth image to calculate the average depth in each mask area; Traversing each category in the RGB image, obtaining the current pixel value and its corresponding depth information, and if the difference between the depth of the current pixel and the average depth of the area exceeds a preset threshold, removing the part from the mask; If there is an inaccuracy in the edge of the depth image mask, a two-pixel dilation operation is performed to generate a processed mask image.

5. The method according to claim 4, characterized in that The average depth is expressed as: Where N represents the number of all pixels of object i in the mask area, p j Indicates the depth value corresponding to each pixel in the area.

6. The method according to claim 3, characterized in that Also includes: Set the similarity threshold based on the distance information between the historical frame and the center point of the current frame; Perform missed detection judgment based on the similarity threshold to obtain a judgment result; If the judgment result is that missed detection has occurred, the object in the previous frame is projected into the current frame for compensation processing.

7. The method according to claim 1, characterized in that The step of inputting each frame of the corrected mask image into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic object includes: Input each frame of the corrected mask image into the dynamic feature screening thread; Traverse all pixels of the mask image of the current frame; Traverse the neighboring pixels around the current pixel and calculate their distance to the current pixel; If the neighboring pixel is an edge pixel of the potential dynamic object mask and does not belong to the same dynamic object as the current pixel, then update and calculate the minimum value of the edge pixel distances between the two; If the minimum value is smaller than a preset edge distance threshold, and the depth difference between two pixels is smaller than a preset pixel depth threshold, it indicates that interference has occurred, and the feature point in the potential dynamic object is regarded as a dynamic feature point.

8. A semantic visual SLAM system based on deep mask segmentation in a dynamic environment, characterized in that: include: Image preprocessing module: used to input RGB images and depth images into the tracking thread for preprocessing; Mask image generation module: used to input the preprocessed RGB image into the semantic segmentation thread to accurately detect and segment the scene objects, obtain the semantic label of each pixel, and generate a segmentation mask, wherein the scene objects can be divided into three categories: static objects, highly dynamic objects, and potentially dynamic objects; combined with the depth information consistency of the preprocessed depth image, the mask is corrected and optimized to obtain a mask image; Feature point classification module: used to input each frame of the corrected mask image into the dynamic feature screening thread to determine the dynamic feature points in the potential dynamic object; construct epipolar constraints through matching points in the background area of ​​the RGB image to eliminate dynamic feature points in high dynamic objects and potential dynamic objects in the RGB image, and static matching points are used for camera positioning; Map construction module: Through the point cloud mapping thread, the local point clouds corresponding to each group of key frames in the RGB image after the dynamic feature points are eliminated are merged and spliced ​​to obtain a new point cloud, and a global three-dimensional point cloud map is constructed without dynamic objects.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the semantic visual SLAM method based on depth mask segmentation in a dynamic environment as claimed in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the semantic visual SLAM method based on depth mask segmentation in a dynamic environment are implemented as claimed in any one of claims 1 to 7.

Images