A visual SLAM method, device, equipment and storage medium

Abstract

The application discloses a visual SLAM method, device and equipment and a storage medium, relates to the technical field of visual SLAM, and comprises the following steps: target detection and feature point extraction are carried out on each frame of RGB image, a plurality of targets and corresponding target detection frames are obtained; the weighted values of the plurality of targets in the next frame of RGB image are compared with a weighted threshold value, the initial dynamic levels of the plurality of targets are updated according to the comparison result, and whether the plurality of targets are non-static targets is determined according to the update result; the chi-square values of feature points contained in the target detection frame corresponding to the non-static target are acquired based on a plurality of depth images, the chi-square value of each feature point is compared with a set threshold value, the feature points smaller than the set threshold value are removed, and static feature points are obtained; and a three-dimensional map is constructed based on the static feature points. The application can adapt to a dynamic environment, and ensures that the constructed map is consistent with the actual dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual SLAM technology, and particularly relates to a visual SLAM method, device, equipment and storage medium. BACKGROUND

[0002] Visual SLAM (Simultaneous Localization and Mapping) technology is a technology combining real-time positioning and map construction, and is usually used in the fields of robots, autonomous driving and augmented reality. Traditional visual SLAM methods are mainly based on geometry and feature matching, including feature point extraction and matching and geometric constraint solving. These methods have achieved certain success on 2D images, but have certain limitations in processing 3D data, especially the relatively weak processing capability for point cloud data. With the rise of deep learning technology, researchers have begun to try to apply deep learning methods to SLAM. Among them, the most notable is to use deep neural networks to learn image features, thereby improving the accuracy and robustness of feature point extraction and matching. In addition, researchers have also tried to use deep learning for map construction and camera positioning tasks.

[0003] Traditional visual SLAM methods take static environment as a precondition, and match the extracted feature points to realize positioning and map construction. In a dynamic environment, the movement of objects can cause the position of feature points to change, making map construction unstable. Especially for long-time moving objects, traditional methods often cannot accurately filter out their feature points, leading to the failure of map updating. In a dynamic environment, the map constructed by the traditional SLAM method may lose consistency due to the movement or addition of objects. SUMMARY

[0004] The present application provides a visual SLAM method, device, equipment and storage medium, which solves the problem that the map constructed by the traditional SLAM system may lose consistency due to the movement or addition of objects.

[0005] In a first aspect, the present application provides a visual SLAM method, comprising the following steps:

[0006] Continuously acquiring multiple frames of RGB images and depth images containing dynamic scenes;

[0007] Performing target detection and feature point extraction on each frame of RGB image to obtain multiple targets and corresponding target detection boxes; wherein each target detection box includes multiple feature points.

[0008] Let the initial frame RGB image be the current frame RGB image. Multiple targets in each frame RGB image have an initial dynamic level. If the position of the same target in the next frame RGB image and the current frame RGB image does not satisfy the epipolar constraint, then obtain a weighted value to characterize its motion state based on the real-time position of the target. Compare the weighted values ​​of multiple targets in the next frame RGB image with a weighted threshold. Update the initial dynamic level of multiple targets according to the comparison result. Determine whether multiple targets are non-static targets based on the update result.

[0009] Based on multi-frame depth images, the chi-square value of the feature points contained in the target detection box corresponding to the non-static target is obtained. The chi-square value of each feature point is compared with a set threshold. Feature points with values ​​less than the set threshold are removed to obtain static feature points.

[0010] The feature points corresponding to the static target in each frame of RGB image are fused with the static feature points to obtain a fused point cloud. The fused point clouds of multiple frames of RGB images are then stitched together to obtain a stitched point cloud. A 3D map is then constructed based on the stitched point cloud.

[0011] Preferably, the weighted values ​​are as follows:

[0012]

[0013] In the formula, g is the weighting value, and θ1 and θ2 represent different weights. Represents vector The cosine of the angle between the target and the polar plane normal vector n, p'2 is the projection point of the target on the plane, C2 is the camera center of the current frame, z(P,P') represents the depth difference of the target observed by the camera in the reference frame and the current frame, P is the position of the target in the reference frame, and P' is the position of the target in the current frame.

[0014] Preferably, the step of comparing the weighted values ​​of multiple targets in the next frame RGB image with a weighted threshold, and updating the initial dynamic levels of the multiple targets based on the comparison results, includes the following steps:

[0015] The initial dynamic level includes static and non-static, and non-static includes low dynamic and high dynamic.

[0016] The weighted threshold includes a first threshold and a second threshold;

[0017] When vector When the absolute value of the cosine of the angle between the target and the polar plane normal vector n is greater than the first threshold, let θ1 = 1 and θ2 = 0, update the target with the initial dynamic level of static to high dynamic, update the target with the initial dynamic level of low dynamic to high dynamic, and do not update the target with the initial dynamic level of high dynamic.

[0018] Otherwise, let θ1=0, θ2=1, obtain the weighted value, compare the weighted value with the second threshold value, if the weighted value is less than the second threshold value, do not update the target with the initial dynamic level as static, update the target with the initial dynamic level as low dynamic to static, update the target with the initial dynamic level as high dynamic to low dynamic; if the weighted value is greater than or equal to the second threshold value, update the target with the initial dynamic level as static to low dynamic, update the target with the initial dynamic level as low dynamic to high dynamic, do not update the target with the initial dynamic level as high dynamic.

[0019] Preferably, the chi-square value of the feature point contained in the target detection box corresponding to the non-static target based on the multi-frame depth image comprises the following steps:

[0020] The depth values of the plurality of feature points based on the depth image are sorted from small to large, and the feature points in the first 75% of the sorting are selected as a plurality of foreground points;

[0021] The mean and standard deviation of the plurality of foreground points are obtained;

[0022] The depth value of each foreground point is subtracted from the mean and divided by the standard deviation to obtain a plurality of foreground points subject to standard normal distribution;

[0023] The plurality of foreground points subject to standard normal distribution are squared to obtain corresponding chi-square values; the chi-square values are specifically as follows:

[0024]

[0025] In the formula, δT is the chi-square value, d is the depth value of the foreground point, m is the number of foreground points, d k and d j are the depth values of the kth foreground point and the jth foreground point.

[0026] Preferably, the comparison of the chi-square value of each feature point with the set threshold value comprises the following steps:

[0027] The size of the plurality of foreground points subject to standard normal distribution and the set threshold value is compared;

[0028] If greater than the set threshold value, it is retained, and if less than the set threshold value, it is deleted.

[0029] Preferably, the target detection and feature point extraction on each frame of RGB image to obtain a plurality of targets and corresponding target detection boxes comprise:

[0030] Feature point extraction on each frame of RGB image through ORB;

[0031] Target detection on each frame of RGB image through the improved YOLO v5 model, comprising:

[0032] The improved YOLO v5 network model comprises an input end, a Backbone main network, a Neck module and a Detect output end; the Backbone main network comprises a Focus structure and a CSP structure;

[0033] An RGB image is received by the input end and preprocessed;

[0034] The preprocessed RGB image is sliced by the Focus structure, the size of the preprocessed RGB image is reduced to one fourth of the original size, and four adjacent pixels are taken together to form a feature map;

[0035] The feature map is divided into two parts by the CSP structure to obtain a first feature map and a second feature map; the first feature map is subjected to convolution processing; the processed first feature map and the second feature map are spliced to obtain a spliced feature map;

[0036] The spliced feature maps of different scales are fused by the Neck module to obtain a convolution feature map;

[0037] The convolution feature map is mapped by the Detect output module to obtain targets and corresponding target detection boxes.

[0038] Preferably, before the three-dimensional map is constructed based on the spliced point cloud, the spliced point cloud needs to be subjected to depth value validity checking, statistical filtering and voxel grid filtering.

[0039] In a second aspect, the present application provides a visual SLAM device, comprising:

[0040] An acquisition module is configured to continuously acquire multiple frames of RGB images and depth images containing dynamic scenes;

[0041] A detection module is configured to perform target detection and feature point extraction on each frame of RGB image to obtain multiple targets and corresponding target detection boxes; each target detection box includes multiple feature points;

[0042] An updating module is configured to set an initial frame of RGB image as a current frame of RGB image, and multiple targets in each frame of RGB image have initial dynamic levels; if the positions of the same target in a next frame of RGB image and the current frame of RGB image do not satisfy the epipolar constraint, a weighting value representing the motion state of the target is obtained based on the real-time position of the target; the weighting values of multiple targets in the next frame of RGB image are compared with a weighting threshold value, and the initial dynamic levels of the multiple targets are updated according to the comparison result; and whether the multiple targets are non-static targets is determined according to the updating result;

[0043] The comparison module is configured to obtain a chi-square value of a feature point included in a target detection frame corresponding to a non-static target based on the multiple frames of depth images, compare the chi-square value of each feature point with a set threshold, remove the feature points smaller than the set threshold, and obtain static feature points.

[0044] The construction module is configured to fuse the feature points corresponding to the static target in each frame of RGB image with the static feature points, obtain fused point clouds, splice the fused point clouds of the multiple frames of RGB images to obtain spliced point clouds, and construct a three-dimensional map based on the spliced point clouds.

[0045] In a third aspect, the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the visual SLAM method when executing the program.

[0046] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the visual SLAM method when executed by a processor.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The present application first performs target detection and feature point extraction on each frame of RGB image to obtain multiple targets and corresponding target detection frames. Then, an initial dynamic level is applied to each target, and the dynamic level of the target in the next frame of image is updated based on the change of the target position of the adjacent two frames of image, and whether the multiple targets are non-static targets is determined according to the update result. Through the above method, the dynamic and static conditions of the target in each frame of image can be determined, and the chi-square test is used to remove the feature points in the target detection frame corresponding to the non-static target to obtain multiple static feature points. The dynamic target region is separated into dynamic and static regions in the present application, and only the static background points are retained. Finally, the static target corresponding feature points in each frame of RGB image are fused with the static feature points to obtain spliced point clouds, and a three-dimensional map is constructed based on the spliced point clouds. The present application can adapt to dynamic environment, accurately screen the feature points of the dynamic target and separate the feature points included in the target detection frame, and ensure that the constructed map is consistent with the actual dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0050] Figure 1 Flow chart of a visual SLAM method of the present application;

[0051] Figure 2 Improved YOLO v5 structure diagram of the present application;

[0052] Figure 3 Dynamic and static constraint principle diagram of the present application;

[0053] Wherein, Figure 3 (a): the target does not move, Figure 3 (b): the target moves and does not satisfy the epipolar constraint, Figure 3 (c): the target moves but satisfies the epipolar constraint;

[0054] Figure 4 Visual odometry structure diagram of the present application;

[0055] Figure 5 Mapping process diagram of the present application;

[0056] Figure 6 Before filtering dynamic feature points in multiple pictures in an embodiment of the present application;

[0057] Wherein, Figure 6 (a): the first picture, Figure 6 (b): the second picture, Figure 6 (c): the third picture;

[0058] Figure 7 After filtering dynamic feature points in multiple pictures in an embodiment of the present application;

[0059] Wherein, Figure 7 (a): the first picture, Figure 7 (b): the second picture, Figure 7 (c): the third picture;

[0060] Figure 8 Before and after dynamic filtering in an embodiment of the present application;

[0061] Wherein, Figure 8 (a): the dense point cloud before filtering dynamic objects, Figure 8 (b): the dense point cloud after filtering dynamic objects, Figure 8 (c): the octree map before filtering dynamic objects; Figure 8 (d): the octree map after filtering dynamic objects. DETAILED DESCRIPTION

[0062] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0063] Under the background of rapid development of deep learning technology, visual SLAM technology has also undergone multiple evolutions and iterations. Some key breakthroughs have promoted the application of deep learning in SLAM: deep learning models the SLAM system as an end-to-end deep learning model, simultaneously processing the input of sensor data and the output of SLAM tasks, thereby achieving more efficient learning and inference; combining the advantages of deep learning in semantic segmentation and object detection, semantic information is integrated into SLAM to improve the semantic understanding ability of the map; using deep learning methods to model spatiotemporal consistency, improving the robustness and accuracy of the SLAM system in complex scenes.

[0064] The improved target detection network YOLO v5 is introduced into the visual odometry (front-end) part of the ORB-SLAM3 system, and is accelerated by TensorRT, and the chi-square test is combined to separate the dynamic target detection area into static and dynamic, remove the dynamic feature points, and only use the static feature points for odometry tracking.

[0065] With reference to Figure 1 The present application provides a visual SLAM method, comprising the following steps:

[0066] Step 1: continuously acquire multiple frames of RGB images and depth images containing dynamic scenes.

[0067] In this embodiment, the RGB image and the depth map (Depth Map) are acquired by a color depth (RGB-D) camera.

[0068] Step 2: target detection and feature point extraction are performed on each frame of RGB image to obtain multiple targets and corresponding target detection boxes.

[0069] In this embodiment, the improved YOLO v5 is used for target detection on the RGB image. Figure 2The improved YOLO v5 structure schematic diagram of the application. Its main structure mainly includes an input end, a Backbone main network, a Neck module and a Detect output end. The function of the input end is to receive an original image, and through a series of preprocessing operations such as normalization and scaling, it meets the subsequent network processing requirements. The Backbone main network is responsible for extracting features from the input image for subsequent target detection tasks, mainly including Focus structure and CSP structure, Focus structure is to slice the image, reduce the size of the input image to one fourth of the original, and take four adjacent pixels together to form a new feature map, thereby effectively reducing the size of the input image; the CSP structure divides the input feature map into two parts, one part is processed through a small convolutional network, and the other part is directly processed to the next layer. Then the two part feature maps are spliced together as the input of the next layer. The Neck module is responsible for further processing of feature maps from different scales, and FPN and PAN build pyramid-shaped feature maps through top-down and bottom-up paths to provide multi-scale context information, so that the target detection obtains more helpful features. The final Detect output module is responsible for mapping the convolutional feature map to the final output of the target detection, mainly using a series of convolution and activation functions to predict the target bounding box position and class probability, and finally through post-processing techniques such as non-maximum suppression, the overlapping bounding boxes are merged and removed to generate the final target detection result.

[0070] The feature point extraction is performed by using ORB (Oriented FAST and Rotated BRIEF) feature extraction, mainly including two main steps of FAST corner extraction and BRIEF.

[0071] Reference Figure 1 In the embodiment, the target detection thread is added between the input of image data and feature matching, and runs in parallel with the feature point extraction thread.

[0072] The front-end module is a visual odometer, which mainly estimates the camera motion through adjacent frame images and restores the local map spatial structure.

[0073] Step 3: The initial frame RGB image is the current frame RGB image, and each target in each frame RGB image has an initial dynamic level. If the positions of the same target in the next frame RGB image and the current frame RGB image do not satisfy the epipolar constraint, the weighted value representing the motion state of the target is obtained based on the real-time position of the target; the weighted values of the multiple targets in the next frame RGB image are compared with the weighted threshold value, and the initial dynamic levels of the multiple targets are updated according to the comparison result, and whether the multiple targets are non-static targets is determined according to the update result.

[0074] In the designed visual odometry, a dynamic target is detected by a YOLO v5 algorithm. However, when processing feature points in a detection box of the dynamic target, directly removing all feature points in the detection box may cause problems in the system. When the detection box of the dynamic target in the key frame is too large, the number of feature points in the current frame may be significantly reduced, which may affect the tracking effect of the visual odometry, and even cause tracking failure. Therefore, the dynamic target feature point elimination operation needs to be optimized to ensure effective tracking of the dynamic target while avoiding tracking instability. To this end, a new dynamic object feature point elimination strategy is adopted.

[0075] In the embodiment, the dynamic level includes static, low dynamic and high dynamic, and the initial dynamic level is artificially determined.

[0076] After training, Yolo v5 can detect and identify objects in the input image, including cars, airplanes, tables, chairs, people, animals, computers and tea cups, etc. In the process of removing dynamic feature points, not only the object needs to be identified, but also it needs to be judged whether the object is a dynamic object. If it is only divided into two categories of dynamic and static, it is too affirmative in the experimental process, such as the tea cup that is moved. Therefore, a dynamic and static weighting constraint is defined to divide the motion level of the object. The constraint rule is as follows:

[0077]

[0078] Figure 3 (a) is that the camera observes the same space point from different angles, C1 and C2 are the camera centers of the reference frame and the current frame respectively, P is a point in space, straight lines l1 and l2 are epipolar lines, p1 and p2 are the projection points of the space point P on the image plane, and if the space point does not move, i.e. in the ideal case, p2 should be located on the epipolar line l2, and the corresponding constraint is:

[0079]

[0080] Where x1 and x2 are the coordinates of the projection points p1 and p2 on the normalized plane, and F is the fundamental matrix.

[0081] The calculation method of the fundamental matrix F is:

[0082] F=K -T t^RK -1 ;

[0083] Where K is the camera intrinsic matrix, R and t represent the rotation vector and translation vector when the camera moves between two adjacent frames.

[0084] If the space point P moves, such as Figure 3The camera moves from the reference frame to the current frame position, and the spatial point P M1 moves to P M2 At this time, the spatial point P M2 moves to the projection position p'2 m2 on the current frame, which will deviate from the corresponding epipolar line l2, resulting in the failure to meet the above constraint. However, when the feature point moves along the optical axis direction of the camera, as shown in (c) of FIG. 1, Figure 3 the spatial point moves from P M2 to P M2 'position. Due to the particularity of the moving direction, the projection point P' M2 'of the moved P m4 'position on the current frame still falls on the epipolar line, satisfying the epipolar constraint. Therefore, in order to accurately judge the real motion state of the feature point, a dynamic-static constraint is defined to better classify its state:

[0085]

[0086] wherein g represents the result of dynamic-static weighting, represents the cosine value of the angle between the vector and the normal vector n of the epipolar plane, the closer the angle is to 90°, the lower the deviation of the projection point p'2 relative to the epipolar line, and the lower the dynamic level of the point, z(P, P') represents the depth difference of the target observed by the camera in the reference frame and the current frame; θ1 and θ2 represent their corresponding weights, when the absolute value of the cosine value is greater than a pre-set threshold , θ1 = 1 and θ2 = 0, i.e. directly classifying it as a high dynamic target, otherwise θ1 = 0 and θ2 = 1, and then calculating the weighted result g, and finally setting a threshold to compare the calculated weighted result g with the threshold to update the level of the target feature point:

[0087]

[0088] After updating, the state of the detected target can be more accurate.

[0089] Through yolov5, the target can be recognized and classified for the first time, and then the dynamic-static weighting constraint is used to complete the second classification of the target feature point. After that, the dynamic target feature points detected are removed.

[0090] Fourth step: based on multiple frame depth images, the chi-square value of the feature points contained in the target detection box corresponding to the non-static target is obtained, the chi-square value of each feature point is compared with the set threshold, and the feature points less than the set threshold are removed to obtain the static feature points.

[0091] After YOLO v5 target detection, there will be many object detection boxes in the input image sequence. Each target box detected by the YOLO v5 algorithm is traversed. The following operations are performed on the feature points in each detection box:

[0092] Step 1. Depth value sorting: Sort these feature points according to their depth values (i.e., the distance from the object to the camera) from small to large. The obtained depth map contains depth information.

[0093] Step 2. Select the top 75% of feature points as foreground points: Assuming that the foreground object has smaller depth values in the image, select the top 75% of n points as foreground points d. These points are more likely to belong to the main object in the detection box.

[0094] Step 3. Calculate the mean and standard deviation: Calculate the mean and standard deviation of the depth values of these foreground points. Here, it is assumed that the depth values of the foreground points follow a Gaussian distribution (normal distribution).

[0095] Step 4. Gaussian distribution to standard normal distribution: Subtract the mean from each foreground point's depth value and divide by the standard deviation. The processed data will follow a standard normal distribution.

[0096] Step 5. Standard normal distribution to chi-square distribution: The square of data following a standard normal distribution will follow a chi-square distribution with 1 degree of freedom.

[0097] Step 6. Chi-square test: Use the chi-square test δT to select the chi-square value corresponding to the 0.9 quantile under 1 degree of freedom as the threshold (detection value). The quantile of the chi-square distribution is a concept in statistics used to determine the critical value of a certain probability under the chi-square distribution. Here, the 0.9 quantile means that 90% of the data is distributed below this value.

[0098] Step 7. Distinguish foreground points and background points: Compare the calculated chi-square value with the threshold. Points that do not conform to this model (i.e., greater than the threshold) are considered background points. The chi-square value is calculated using equation 1. If it is less than the threshold, it is determined to be a foreground object point (dynamic point). If it is greater than the threshold, it is considered a static background point.

[0099] The purpose of the above process is to use depth information and statistical methods to more accurately distinguish foreground objects and backgrounds in the image. The specific equation is shown in the following formula:

[0100]

[0101] In the formula, δT is the chi-square value, d is the depth value of the foreground point, m is the number of foreground points, and d k , d j are the depth values of each point.

[0102] Feature matching: Feature matching is completed by using the fast approximate nearest neighbor algorithm in Opencv to match the ORB features.

[0103] Pose estimation: The input data is an RGB image and a depth image, so the 3D position of the feature points is obtained, and then the camera pose estimation is completed by the method of direct linear transformation.

[0104] Key frame selection: When the number of feature point matches in a certain frame of several consecutive frames is large, or the scene changes significantly, or the camera pose changes greatly, the system will list it as a key frame.

[0105] The back-end module is to optimize the camera pose and local map by BA, the main reason is that the image noise and the uncertainty in the pose calculation affect.

[0106] Insert key frame and update map point: Receive the key frame provided by the front end, then generate new map points, and then optimize the local map. The key frame is a representative frame image in several consecutive frames, when the number of feature point matches in the frame is large, or the scene changes significantly, or the camera pose changes greatly, the system will list it as a key frame.

[0107] Screening key frame: Receive the key frame provided by the front end, according to whether the map points generated by the key frame are redundant, to delete or keep.

[0108] The loop module is to identify the historical scene and compare, and then use the global BA to correct the errors generated by the front end and the back end to update the map. Loop detection and correction is to adjust the built map by identifying historical scenes to eliminate the cumulative error in the mapping process. Map updating and fusion is to fuse the local maps into a whole map according to the overlapping areas of the local maps.

[0109] Reference Figure 4 , the data input of the application is to obtain RGB image and depth image by using RGB-D camera, the feature point extraction thread extracts ORB features from the input image, at the same time, the yolov5 target detection thread determines the category of the identified target according to the defined dynamic and static weighting constraint, then filters out the dynamic feature points, and then performs feature matching on the remaining static feature points, and then uses the static feature points to complete camera pose estimation.

[0110] By adding the yolov5 target detection thread with dynamic and static weighting constraint in the visual odometry, the classification of dynamic and static target feature points can be completed, then the feature points in the dynamic target detection box are filtered out, and then the remaining static feature points are used to complete feature matching and pose estimation, the RGB image data is converted into point cloud, and after filtering and optimization, it is spliced into a map.

[0111] Fifth step: fuse the feature points corresponding to the static target in each frame of RGB image with the static feature points to obtain a fused point cloud, and splice the fused point clouds of multiple frames of RGB images, and construct a three-dimensional map based on the splicing result.

[0112] The application also designs a dense point cloud map construction method and an octree map construction method for dynamic environments. First, for each key frame, static point clouds are generated using color maps and depth maps in combination with target detection technology. Through point cloud splicing and filtering steps, the reconstruction of static dense point clouds is completed. Finally, the octree map construction technology is used to effectively reduce the volume of the map while giving the map more functions.

[0113] Currently, in the visual SLAM system, point cloud maps and octree maps are the two most widely chosen methods in the mapping field. ORB-SLAM3, as a visual SLAM system, uses sparse point clouds for mapping. However, the sparse point cloud mapping method is limited in some application scenarios, limiting its applicability in dynamic environments. Therefore, the ORB-SLAM3 system in dynamic environments is increased with the construction threads of dense reconstruction and octree reconstruction.

[0114] Reference Figure 5 First, based on the information of the key frame, the corresponding depth map and target detection results are combined to achieve the generation of static point clouds by removing dynamic objects. Target detection is completed through the dynamic and static weighted constraints of yolov5, and then the dynamic points are filtered in combination with the depth map information. Subsequently, the generated static point clouds are spliced to fuse the static point clouds obtained from each key frame (determined by the image sequence) into a whole.

[0115] Finally, the merged point cloud is filtered and optimized to improve the accuracy and usability of the map, ensuring that it can better reflect the real environment.

[0116] In the above algorithm, the dynamic objects present in the key frame are processed to ensure that the dense point cloud map spliced from the static background points does not appear ghosting. To enhance the visual effect, the point cloud is filtered, including the following key steps:

[0117] (1) Depth value validity check: When generating point clouds, the system first performs a validity check on the depth values to eliminate points with invalid depth values. These invalid values are usually caused by large errors or direct measurements of zero when the RGB-D camera measures points beyond its effective range.

[0118] (2) Statistical filtering: A statistical filter is applied to the point cloud to remove isolated points while preserving those that are clustered together. This filter looks for the N closest points to each point, calculates the average distance, and removes those points that are far from the average, smoothing the point cloud surface.

[0119] (3) Voxel grid filtering: Finally, a voxel grid filter is applied to the point cloud to downsample it and remove points that are very close to each other in overlapping regions. The voxel filter replaces all points in a voxel with the one closest to the voxel center, ensuring that there is only one point per voxel, thus saving storage space and reducing data redundancy.

[0120] The octree map is obtained by processing the point cloud data using the mapping tool octomap library. The point cloud map is large in size and requires a large storage space. In order to reduce the occupied space, some unnecessary details need to be removed. However, reducing the resolution of the point cloud data will result in a decrease in map quality, so it is converted into an octree map.

[0121] The advantages of the octree map are flexibility, compressibility, and the ability to update at any time, while saving as much space as possible while retaining the functionality of the map.

[0122] The design concept is based on a tree structure, where each node (except for leaf nodes) can be expanded into eight child nodes. The basic principle of the octree map is to divide the three-dimensional space into multiple small cubes. Initially, there is only one cube, and each cube is split into eight equally sized sub-cubes as needed. This splitting process is repeated until the desired modeling accuracy is achieved.

[0123] Embodiment

[0124] The TUM dataset used in this embodiment is a commonly used dataset in the field of computer vision and robotics, provided by the Technical University of Munich (TUM) in Germany. This dataset contains sensor data from various different scenarios and is mainly used for evaluation and research in tasks such as SLAM and visual positioning. Since there are dynamic objects in the assumed environment of this experiment, this embodiment selects seven datasets from the Dynamic Objects category in the TUM dataset for experimentation, namely the static environment dataset fr1_desk, the low dynamic scenario datasets fr1_room and fr2-desk-with-person, and the four high dynamic scenario datasets: fr3_walking_xyz, fr3_walking_rpy, fr3_walking_static, and fr3_walking_halfsphere.

[0125] The experimental results are compared as followsFigure 6 and Figure 7 As shown in FIG. 13, it can be visually seen that, after the processing of the application, the dynamic feature points located on the human body have been effectively filtered out, and the background feature points in the detection frame of the human have been effectively retained, proving that the feature point filtering strategy proposed in the application is effective.

[0126] The verification experiment selects the absolute trajectory error (ATE) to evaluate the performance of the SLAM system in positioning and trajectory tracking. The root mean square error (RMSE) is used to quantify the difference between the estimated pose and the ground truth pose. The standard deviation (SD) is used to enhance the comprehensiveness of the evaluation process.

[0127] Table 1 Comparison of absolute trajectory error

[0128]

[0129] As can be seen from the results of the seven data sets in Table 1 in the two systems, in the same static environment, although the system of the application performs slightly worse than ORB-SLAM3 on the fr1_desk data set, both systems can exhibit good accuracy. In a low dynamic environment, the improvement of the system of the application is limited, mainly because ORB-SLAM3 itself can use the RANSAC algorithm to reduce the interference of low dynamic objects to a certain extent. In a high dynamic environment, the results generated by the system of the application have been greatly improved compared to ORB-SLAM3. In the fr3_walking_rpy, fr3_walking_static, fr3_walking_xyz and fr3_w_halfsphere four data sets, the system accuracy can be improved by 96.61%, the lowest can be improved by 88.83%, and the average can be improved by 93.44%. This intuitively reflects that the visual odometry and dynamic feature point elimination strategy designed in the application is effective, and the adaptability of ORB-SLAM3 in dynamic scenes is significantly improved.

[0130] Referring to Figure 8 In the mapping experiment, the static environment fr1_desk data set and the high dynamic environment fr3_walking_xyz data set before and after filtering dynamic targets were selected for mapping comparison test to determine the adaptability of the system of the application in different environments. The experimental results show that the dynamic scene data set only has clear static objects in the reconstruction results after filtering by the system of the application, and the human point cloud has been completely filtered out. It can be seen that using the map construction design proposed in the application, the SLAM system can perform static environment reconstruction in a dynamic environment, and the effect is good.

[0131] Based on the same concept, the application further provides a visual SLAM device, comprising an acquisition module, a detection module, an updating module, a comparison module and a construction module.

[0132] The acquisition module is configured to continuously acquire a plurality of frames of RGB images and depth images containing dynamic scenes.

[0133] The detection module is configured to perform target detection and feature point extraction on each frame of RGB image to obtain a plurality of targets and corresponding target detection boxes; each target detection box includes a plurality of feature points.

[0134] The updating module is configured to set an initial frame of RGB image as a current frame of RGB image, and each target in each frame of RGB image has an initial dynamic level; if the positions of the same target in a next frame of RGB image and the current frame of RGB image do not satisfy the epipolar constraint, a weighting value representing the motion state of the target is acquired based on the real-time position of the target; the weighting values of the plurality of targets in the next frame of RGB image are compared with a weighting threshold value, and the initial dynamic levels of the plurality of targets are updated according to the comparison result, and whether the plurality of targets are non-static targets is determined according to the updating result.

[0135] The comparison module is configured to acquire a chi-square value of a feature point included in a target detection box corresponding to a non-static target based on a plurality of frames of depth images, compare the chi-square value of each feature point with a set threshold value, remove the feature points smaller than the set threshold value, and obtain static feature points.

[0136] The construction module is configured to fuse the feature points corresponding to static targets in each frame of RGB image with the static feature points to obtain fused point clouds, splice the fused point clouds of the plurality of frames of RGB images to obtain spliced point clouds, and construct a three-dimensional map based on the spliced point clouds.

[0137] The application further provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the visual SLAM method when executing the program.

[0138] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the visual SLAM method.

[0139] Although the preferred embodiments of the application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the application.

[0140] Obviously, a person skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A visual SLAM method, characterized by, The method comprises the following steps: obtaining a plurality of frames of RGB images and depth images containing dynamic scenes in succession; detecting targets and extracting feature points from each frame of RGB image to obtain a plurality of targets and corresponding target detection boxes; each target detection box includes a plurality of feature points; taking an initial frame of RGB image as a current frame of RGB image, and each target in each frame of RGB image has an initial dynamic level; if the positions of the same target in a next frame of RGB image and the current frame of RGB image do not satisfy the epipolar constraint, a weighting value representing the motion state of the target is obtained based on the real-time position of the target; the weighting values of the plurality of targets in the next frame of RGB image are compared with a weighting threshold value, and the initial dynamic levels of the plurality of targets are updated according to the comparison result, and whether the plurality of targets are non-static targets is determined according to the updating result; obtaining chi-square values of feature points included in the target detection box corresponding to the non-static target based on the plurality of frames of depth images, comparing the chi-square value of each feature point with a set threshold value, and removing the feature points smaller than the set threshold value to obtain static feature points; fusing the feature points corresponding to the static target in each frame of RGB image with the static feature points to obtain fused point clouds, splicing the fused point clouds of the plurality of frames of RGB images to obtain spliced point clouds, and constructing a three-dimensional map based on the spliced point clouds; the weighting value is specifically as follows: ； wherein, g is a weighted value, and denote different weights, denotes a vector and a cosine value of an angle between the polar plane normal vector and the polar plane normal vector is a projection point of the target on the plane, C 2 is a camera center of the current frame, denotes a depth difference value of the target observed by the camera in the reference frame and the current frame, P is a position of the target under the reference frame, is a position of the target under the current frame; the chi-square values of the feature points included in the target detection box corresponding to the non-static target based on the plurality of frames of depth images comprise the following steps: sorting the depth values of the plurality of feature points from small to large based on the depth image, and selecting the top 75% of the feature points as a plurality of foreground points; obtaining the mean and standard deviation of the plurality of foreground points; subtracting the mean from the depth value of each foreground point and dividing by the standard deviation to obtain a plurality of foreground points subject to standard normal distribution; squaring the plurality of foreground points subject to standard normal distribution to obtain corresponding chi-square values; the chi-square value is specifically as follows: ； wherein, is a chi-square value, d is a foreground point depth value, m is a foreground point number, and is a depth value of the k th foreground point and the j th foreground point.

2. A visual SLAM method as claimed in claim 1, characterized in that, the comparison of the weighting values of the plurality of targets in the next frame of RGB image with the weighting threshold value and the updating of the initial dynamic levels of the plurality of targets according to the comparison result comprise the following steps: the initial dynamic level includes static and non-static, and the non-static includes low dynamic and high dynamic; the weighting threshold value includes a first threshold value and a second threshold value; When the vector With the polar plane normal vector The absolute value of the cosine value of the included angle is greater than the first threshold value, and the initial dynamic level is static , The target with the initial dynamic level of static is updated to high dynamic, the target with the initial dynamic level of low dynamic is updated to high dynamic, and the target with the initial dynamic level of high dynamic is not updated Otherwise let , , the weighted value is obtained, the weighted value is compared with the second threshold value, if the weighted value is less than the second threshold value, the target with the initial dynamic level of static is not updated, the target with the initial dynamic level of low dynamic is updated to static, and the target with the initial dynamic level of high dynamic is updated to low dynamic; if the weighted value is greater than or equal to the second threshold value, the target with the initial dynamic level of static is updated to low dynamic, the target with the initial dynamic level of low dynamic is updated to high dynamic, and the target with the initial dynamic level of high dynamic is not updated.

3. The visual SLAM method of claim 1, wherein, the comparison of the chi-square value of each feature point with the set threshold value comprises the following steps: comparing the size of the plurality of foreground points subject to standard normal distribution with the set threshold value; if greater than the set threshold value, the feature point is retained, and if less than the set threshold value, the feature point is deleted.

4. The visual SLAM method of claim 1, wherein, the target detection and feature point extraction from each frame of RGB image to obtain a plurality of targets and corresponding target detection boxes comprise: extracting feature points from each frame of RGB image through ORB; detecting targets from each frame of RGB image through an improved YOLO v5 model, comprising: the improved YOLO v5 network model comprises an input end, a Backbone main network, a Neck module and a Detect output end; the Backbone main network comprises a Focus structure and a CSP structure; Receiving an RGB image through an input end and pre-processing the RGB image; Slicing the pre-processed RGB image through a Focus structure, reducing the size of the pre-processed RGB image to one fourth of the original size, and taking four adjacent pixels together to form a feature map; Dividing the feature map into two parts through a CSP structure to obtain a first feature map and a second feature map, performing convolution processing on the first feature map, and splicing the processed first feature map and the second feature map to obtain a spliced feature map; Fusing the spliced feature maps of different scales through a Neck module to obtain a convolution feature map; Mapping the convolution feature map through a Detect output module to obtain a target and a corresponding target detection frame.

5. The visual SLAM method of claim 1, wherein, Before constructing a three-dimensional map based on the spliced point cloud, the spliced point cloud needs to be subjected to a depth value validity check, statistical filtering and voxel grid filtering. 6.A visual SLAM device of the visual SLAM method according to claim 1, characterized in that, Comprise: An acquisition module configured to continuously acquire multiple frames of RGB images and depth images containing dynamic scenes; A detection module configured to perform target detection and feature point extraction on each frame of RGB image to obtain multiple targets and corresponding target detection frames; wherein each target detection frame includes multiple feature points; An update module configured to set an initial frame of RGB image as a current frame of RGB image, and set the multiple targets in each frame of RGB image as initial dynamic levels; if the positions of the same target in a next frame of RGB image and the current frame of RGB image do not satisfy the epipolar constraint, obtain a weighting value representing the motion state of the target based on the real-time position of the target; compare the weighting values of the multiple targets in the next frame of RGB image with a weighting threshold value, update the initial dynamic levels of the multiple targets according to the comparison result, and determine whether the multiple targets are non-static targets according to the update result; A comparison module configured to obtain the chi-square values of the feature points included in the target detection frame corresponding to the non-static target based on the multiple frames of depth images, compare the chi-square value of each feature point with a set threshold value, remove the feature points smaller than the set threshold value, and obtain static feature points; A construction module configured to fuse the feature points corresponding to the static targets in each frame of RGB image with the static feature points, obtain fused point clouds, splice the fused point clouds of the multiple frames of RGB image to obtain spliced point clouds, and construct a three-dimensional map based on the spliced point clouds.

7. A computer device, comprising: The storage medium stores a computer program, and the computer program is executed by the processor to implement the visual SLAM method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the visual SLAM method of any one of claims 1-5.

Images