A Dynamic Target Removal LiDAR SLAM Processing Method and System Based on Projection and Deep Learning

Abstract

This invention relates to the field of real-time localization and mapping (RTL) technology, and particularly to a dynamic target removal LiDAR SLAM processing method and system based on projection and deep learning. For point cloud data acquired in real-time by LiDAR, a depth projection map of the point cloud data is obtained based on multi-view projection from a multi-directional virtual camera. The coordinate projection correspondence between each point in the point cloud data and the pixel in the depth projection map is recorded. The depth projection map is input into a YOLOv8 target detection model, which performs target detection and segmentation and records key information of the detected targets. Based on the target detection and segmentation results and previous dynamic target detection results, the pose transformation information between point cloud frames is obtained, and the overlap degree of identical targets in the point cloud data is calculated. Dynamic target point identification is performed based on the target overlap degree. A global point cloud map is generated based on the pose transformation information and the point cloud frames from which dynamic targets are removed. This invention can identify and remove moving objects in real time, improving the accuracy of pose estimation in moving environments.

Description

Technical Field

[0001] This invention relates to the field of real-time positioning and mapping technology, and in particular to a dynamic target culling lidar SLAM processing method and system based on projection and deep learning. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) can be described as a robot moving from an unknown location in an unknown environment, performing self-localization based on its position and a map during movement, and simultaneously building an incremental map based on its self-localization. It can be applied to localization and navigation in fields such as intelligent robots and autonomous driving. Currently, mature 3D LiDAR SLAM algorithms such as LOAM, LIO-SAM, and FAST-LIO typically assume a static environment. When moving objects are present, the system's position estimation accuracy decreases, loop closure detection robustness deteriorates, and the created environmental map contains ghosting issues. Mainstream dynamic point removal algorithms like Removert typically use the depth difference between the constructed point cloud map and the original LiDAR data to determine whether a point is dynamic. These algorithms suffer from problems such as inability to detect dynamic points in real time and high computational cost. Therefore, how to identify and remove dynamic points in LiDAR point cloud data in real time to achieve more accurate pose estimation and map construction has become an urgent problem to be solved in localization and navigation applications. Summary of the Invention

[0003] This invention provides a dynamic target culling lidar SLAM processing method and system based on projection and deep learning, which can solve the impact of moving objects on pose estimation and map building during the SLAM process, so as to achieve more accurate pose estimation and map building.

[0004] According to the design scheme provided by the present invention, on the one hand, a dynamic target culling LiDAR SLAM processing method based on projection and deep learning is provided, comprising:

[0005] For point cloud data collected in real time by lidar, a depth projection map of the point cloud data is obtained based on multi-view projection of a multi-directional virtual camera, and the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map is recorded.

[0006] The depth projection map is input into the pre-trained YOLOv8 object detection model. The pre-trained YOLOv8 object detection model is used to perform object detection and segmentation and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type and the object tracking ID.

[0007] Based on the target detection and segmentation results and the previous dynamic target detection results, the pose transformation information between point cloud frames is obtained, and the overlap degree of the same target in the point cloud data is calculated. Dynamic target points are identified based on the target overlap degree.

[0008] A global point cloud map is generated based on pose transformation information and point cloud frames that have been used to remove dynamic target point clouds.

[0009] As a dynamic target culling lidar SLAM processing method based on projection and deep learning in this invention, the YOLOv8 target detection model further adopts CSPDarknet53 as the backbone network, and the target loss function is composed of standard binary cross-entropy loss, DFL loss and CIoU loss weighted together to train the model, so as to train the YOLOv8 target detection model based on the target loss function and using sample data.

[0010] As part of the dynamic target culling LiDAR SLAM processing method based on projection and deep learning of this invention, the method further includes acquiring a depth projection map of point cloud data based on multi-view projection from a multi-directional virtual camera, comprising:

[0011] Based on the intrinsic parameter matrix of the virtual camera, the coordinate position of the point cloud in the lidar coordinate system, and the rotation matrix from the virtual camera coordinate system to the lidar coordinate system, a projection expression is constructed to project the point cloud in the lidar coordinate system onto the pixel coordinate position in the virtual camera pixel coordinate system. Based on this projection expression, a depth projection map of the point cloud data in a predetermined direction is obtained. The virtual cameras are placed around the lidar sensor, and there is a viewpoint overlap between each virtual camera.

[0012] As part of the dynamic target culling lidar SLAM processing method based on projection and deep learning of this invention, the method further obtains point cloud inter-frame pose transformation information based on the target detection and segmentation results and the coordinate projection correspondence, including:

[0013] First, target detection attributes are labeled for each 3D point cloud in the point cloud data based on the target detection segmentation results and coordinate projection correspondence. The attribute labeling includes point type ID, point tracking ID and virtual camera ID.

[0014] Then, for the original point cloud containing target detection attributes, it is arranged according to the LiDAR scan line number, and corner point cloud and surface point cloud are obtained through corner point and surface point detection, so as to obtain the inter-frame pose transformation information of the point cloud based on the corner point cloud and surface point cloud. If there is a point in the feature point cloud with the same attributes as the previously detected dynamic target point, the point is removed from the feature point cloud and does not participate in the inter-frame pose transformation estimation of the point cloud.

[0015] As part of the dynamic target culling LiDAR SLAM processing method based on projection and deep learning of this invention, further, identifying dynamic target point clouds in point cloud data based on target overlap includes:

[0016] First, the overlap, depth difference, and center point difference of the point clouds in adjacent point cloud data frames are calculated based on the number of points of the target object on adjacent point cloud data frames.

[0017] Then, the overall probability of the target object's motion is obtained by weighted summation of the overlap, depth difference, and center point difference. Combined with a preset threshold, it is determined whether the target object is a moving object, and all point clouds of the moving object are used as dynamic target point clouds.

[0018] As part of the dynamic target culling lidar SLAM processing method based on projection and deep learning of this invention, a global point cloud map is further generated based on pose transformation information and point cloud frames from dynamically culled point clouds, including:

[0019] First, the pose estimated by the odometry of the current frame is used as the origin. Keyframes with a preset radius are found and a local map is built based on the keyframes.

[0020] Next, the pose estimated by inter-frame odometry is used as the initial value. The current frame is matched with the local map to remove dynamic target point clouds and determine the pose estimate of the current frame.

[0021] Then, based on the preset pose angle difference between the current frame and the previous frame of the point cloud data and the straight-line distance between the two frames, a list of key frames of the point cloud data is constructed, and the pose information of the key frames in the key frame list is stored in the key frame pose list.

[0022] Finally, dynamic target point clouds are removed from the keyframe list of point cloud data. Based on the keyframe list and keyframe pose list after removing dynamic target point clouds, point cloud stitching is performed to output a global point cloud map.

[0023] As part of the dynamic target culling lidar SLAM processing method based on projection and deep learning of this invention, a key frame list of point cloud data is further constructed based on the preset attitude angle difference between the current frame and the previous frame of point cloud data and the straight-line distance between the two frames, including:

[0024] If the difference between the current frame and the previous frame in one of the three angles—pitch, roll, and yaw—is greater than a preset radian, or if the straight-line distance between the current frame and the previous frame is greater than a preset distance, then the point cloud data of the current frame will be added to the keyframe point cloud list.

[0025] Furthermore, the present invention also provides a dynamic target culling LiDAR SLAM processing system based on projection and deep learning, comprising: a projection mapping module, a detection and segmentation module, a pose acquisition module, and a map generation module, wherein,

[0026] The projection mapping module is used to obtain a depth projection map of the point cloud data based on the multi-view projection of the multi-angle virtual camera for the point cloud data collected in real time by the LiDAR, and to record the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map.

[0027] The detection and segmentation module is used to input the depth projection map into the pre-trained YOLOv8 object detection model, use the pre-trained YOLOv8 object detection model to perform object detection and segmentation and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type and the object tracking ID.

[0028] The pose acquisition module is used to obtain the pose transformation information between point cloud frames based on the target detection and segmentation results and the coordinate projection correspondence, and to determine the target overlap in the point cloud data so as to identify the dynamic target point cloud in the point cloud data based on the target overlap.

[0029] The map generation module is used to generate a global point cloud map based on pose transformation information and point cloud frames that have removed dynamic target point clouds.

[0030] The beneficial effects of this invention are:

[0031] This invention starts with point cloud multi-view projection and 2D image target detection, and implements a dynamic target removal LiDAR SLAM algorithm based on projection and deep learning. It can not only quickly and accurately detect and track objects in LiDAR point cloud frames, but also judge and remove dynamic target points in real time based on the point cloud target detection and tracking results. In the presence of moving objects in the surrounding environment, it can identify and remove moving objects in real time, improve the pose estimation accuracy in motion environments, and solve problems such as ghosting in the created environment map. It is convenient for application deployment in the fields of robotics, autonomous driving and other positioning and navigation fields, and has good application prospects. Attached image description:

[0032] Figure 1 This is a schematic diagram of the dynamic target culling LiDAR SLAM processing flow based on projection and deep learning in the embodiment.

[0033] Figure 2 This is a schematic diagram of the principle framework of the dynamic target elimination lidar SLAM algorithm in the embodiment;

[0034] Figure 3 This is a schematic diagram illustrating the placement of the virtual camera in the embodiment;

[0035] Figure 4 This is a schematic diagram of the superposition of stationary and moving objects in the point cloud in the embodiment;

[0036] Figure 5 This is a schematic diagram of the multi-view point cloud projection results in the embodiment;

[0037] Figure 6 This is a schematic diagram of the object segmentation and tracking results from the YOLO v8 projection map in the embodiment.

[0038] Figure 7 This is a schematic diagram of point cloud target detection results in the embodiment;

[0039] Figure 8 This is a schematic diagram of the dynamic target point removal results in the point cloud example.

[0040] Figure 9 This is a schematic diagram of the pose estimation and map construction results in the example. Detailed implementation method:

[0041] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0042] To address the impact of moving objects on pose estimation and map construction during SLAM, this invention provides a dynamic target culling LiDAR SLAM processing method based on projection and deep learning, starting from multi-view projection of point clouds and target detection in 2D images. (See [link to relevant documentation]). Figure 1 As shown, it contains the following:

[0043] S101. For the point cloud data collected in real time by the lidar, a depth projection map of the point cloud data is obtained based on the multi-view projection of the multi-directional virtual camera, and the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map is recorded.

[0044] Specifically, for the point cloud data collected in real time by lidar, multi-view projection can be performed using a pre-set multi-directional virtual camera to obtain depth projection maps in the X, Y, Z or XYZ directions.

[0045] The depth projection map of point cloud data obtained by multi-view projection based on multi-directional virtual cameras can be designed to include the following:

[0046] Based on the intrinsic parameter matrix of the virtual camera, the coordinate position of the point cloud in the lidar coordinate system, and the rotation matrix from the virtual camera coordinate system to the lidar coordinate system, a projection expression is constructed to project the point cloud in the lidar coordinate system onto the pixel coordinate position in the virtual camera pixel coordinate system. Based on this projection expression, a depth projection map of the point cloud data in a predetermined direction is obtained. The virtual cameras are placed around the lidar sensor, and there is a viewpoint overlap between each virtual camera.

[0047] like Figure 3 As shown, four virtual cameras can be placed around the lidar sensor, with each virtual camera having a certain degree of overlap in their viewpoints. The multiple overlapping virtual cameras serve two purposes: first, to ensure the integrity of the target in the point cloud within the projection image; and second, to allow for multi-angle observation of the target, increasing the probability of it being correctly identified in the two-dimensional projection image.

[0048] The projection process of a point cloud can be represented by formula (2), where P uv R represents the pixel coordinates of a point projected from the LiDAR coordinate system onto the pixel coordinate system of the image, where K represents the intrinsic parameter matrix of the virtual camera, and R represents the pixel coordinates of the image. cl t cl With T cl Let P represent the rotation matrix, translation matrix, and transformation matrix from the Camera coordinate system to the Lidar coordinate system, respectively. lidar The coordinates of the point are in the Lidar coordinate system.

[0049]

[0050] According to formula (1), the pixels in the projection image correspond one-to-one with the three-dimensional points in the Lidar coordinate system. Therefore, attributes such as type ID, virtual camera ID, and tracking ID can be added to the three-dimensional points based on the image detection results.

[0051] S102. Input the depth projection map into the pre-trained YOLOv8 object detection model, use the pre-trained YOLOv8 object detection model to perform object detection and segmentation, and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type, and the object tracking ID.

[0052] In the YOLOv8 object detection model, CSPDarknet53 can be used as the backbone network, and the target loss function for model training can be composed of standard binary cross-entropy loss, DFL loss and CIoU loss weighted together. The YOLOv8 object detection model can then be trained using sample data based on this target loss function.

[0053] The YOLO series of algorithms transforms object detection into a regression problem. It uses convolutional layers to predict the bounding boxes and class probabilities of all objects in an image, without needing to pre-search for regions where objects might exist. Therefore, it has a significant speed advantage over other sliding window-based algorithms. The YOLO series uses the product of the probability of an object being contained in the grid and the intersection-over-union (IoU) ratio of the ground truth edit box and the predicted bounding box to represent the confidence score of each bounding box. Then, non-maximum suppression (NMS) is used to select the bounding box with the highest confidence score as the final detection result.

[0054] YOLOv8 is the latest open-source deep learning-based image object detection algorithm, demonstrating extremely high detection accuracy and real-time performance. In addition to object detection, YOLO v8 also provides image classification, object segmentation, and object tracking capabilities. Compared to previous versions, YOLO v8 features the following improvements:

[0055] (a) The anchor-free mechanism allows for the direct prediction of the target object's position and size without pre-defining anchor points. This anchor-free model further reduces the number of predicted detection boxes, lowers the computational complexity of the algorithm, and improves the real-time performance of the target detection task.

[0056] (b) An optimized CSPDarknet53 was used as the backbone network. This backbone network maintained the use of the Cross Stage Partial (CSP) module while replacing the C3 module in YOLO v5 with the C2f module to achieve lightweight feature extraction. C2f was optimized using the DarknetBottleNeck structure to reduce the number of model parameters while retaining useful feature information. Each convolutional module (ConvModule) consists of a convolutional layer, a batch normalization layer, and a SiLU activation function. The parameter n is used for depth control and varies in models of different scales. YOLOv8n (nano), YOLOv8s (small), YOLOv8m (medium), YOLOv8l (large), and YOLOv8x (extra large) are deep neural network models of different scales. The nano model is the smallest, with slightly lower detection accuracy but the fastest detection speed; the extra large model is the largest, with the highest detection accuracy but the worst real-time performance.

[0057] (c) Different loss functions: The loss function of the YOLO v8 algorithm consists of a weighted sum of classification loss and location-based regression loss. As shown in Equation (2), the classification loss is calculated using the standard binary cross-entropy (BCE).

[0058] BCE loss =ln(x,y)=y n log(σ(x n )+(1-y n ))log(1-σ(x n )) (Formula 2)

[0059] The location-based regression loss consists of two parts: Distribution Focal Loss (DFL) and Complete Intersection over Union (CIoU), which are expressed by Equations (3) and (4), respectively.

[0060] DFL(S i S i+1 )=-((y i+1 -y)logS i +(yy i )logS i+1 ) (Formula 3)

[0061]

[0062]

[0063] As shown in Equation (5), the final loss function is a weighted sum of BCE loss, DFL and CIoU loss.

[0064] f loss =λ1BCE loss +λ2DFL+λ3CIoU loss (Formula 5)

[0065] Because SLAM algorithms have high real-time requirements, in this embodiment, the YOLOv8n (nano) model can be used to train object recognition and segmentation on multi-view projection images, classifying objects in the projection images into categories such as vehicles, pedestrians, and trees. The training dataset consists of 1272 projection images from different scenes randomly selected from the KITTI dataset 00-09 sequence.

[0066] S103. Based on the target detection and segmentation results and the previous dynamic target detection results, obtain the pose transformation information between point cloud frames and calculate the overlap degree of the same target in the point cloud data. Identify dynamic target points based on the target overlap degree.

[0067] Specifically, the point cloud frame pose transformation information obtained based on the target detection and segmentation results and the coordinate projection correspondence can be designed to include the following:

[0068] First, target detection attributes are labeled for each 3D point cloud in the point cloud data based on the target detection segmentation results and coordinate projection correspondence. The attribute labeling includes point type ID, point tracking ID and virtual camera ID.

[0069] Then, for the original point cloud containing target detection attributes, it is arranged according to the LiDAR scan line number, and corner point cloud and surface point cloud are obtained through corner point and surface point detection, so as to obtain the inter-frame pose transformation information of the point cloud based on the corner point cloud and surface point cloud. If there is a point in the feature point cloud with the same attributes as the previously detected dynamic target point, the point is removed from the feature point cloud and does not participate in the inter-frame pose transformation estimation of the point cloud.

[0070] The YOLO v8 algorithm's object tracking function can uniquely identify the same object in different images. By using this capability, two consecutive LiDAR point cloud frames can be transformed to the same coordinate system, and the degree of overlap between the two objects can be determined to identify whether the object has moved. Figure 4 As shown, the red and white dots represent points on adjacent LiDAR point cloud frames, and stationary objects should maintain a high degree of overlap.

[0071] However, considering the different LiDAR scanning angles, the shapes of objects appearing in the point cloud vary slightly. Therefore, in addition to considering the overlap of objects on two LiDAR point cloud frames, it is also necessary to use information such as the average depth difference and the center point distance difference to comprehensively determine whether the objects contained in the region are in motion.

[0072] Specifically, identifying dynamic target point clouds in point cloud data based on target overlap can include:

[0073] First, the overlap, depth difference, and center point difference of the point clouds in adjacent point cloud data frames are calculated based on the number of points of the target object on adjacent point cloud data frames.

[0074] Then, the overall probability of the target object's motion is obtained by weighted summation of the overlap, depth difference, and center point difference. Combined with a preset threshold, it is determined whether the target object is a moving object, and all point clouds of the moving object are used as dynamic target point clouds.

[0075] Let the overlap of the object on two Lidar point cloud frames be r. overlap The calculation method is shown in formula (6). Where N1 represents the number of points of the object on the first point cloud frame, N1 findThis represents the number of points found in the second point cloud frame within a radius of 0.05m, with the points on the first point cloud frame as the origin. N2 represents the number of points containing the object in the second point cloud frame. find Using the points on the second point cloud frame as the origin, find the number of corresponding points in the first point cloud frame within a radius of 0.05m.

[0076]

[0077] Let r be the depth difference of the object in two LiDAR point cloud frames. range The calculation method is shown in formula (7). Where N1 and N2 have the same meaning as above, p1 and p2 represent the points on the first and second Lidar point cloud frames respectively, dis represents the distance from the point to the origin of the coordinate system, and t... range This is the depth difference threshold, which is set to 0.1 here.

[0078]

[0079] Let r be the difference in the center points of the object on two LiDAR point cloud frames. center Here, the center point is defined as the coordinates of the center position of the object's smallest bounding rectangle, and its calculation method is shown in formula (8). Where p1 center p2 center These represent the coordinates of the center point of the object on the first and second LiDAR point cloud frames, respectively; dis represents the straight-line distance between the two points; t center The threshold for the center point difference is set to 0.1 here.

[0080]

[0081] Let the overall probability of whether an object moves be r. move The calculation method is shown in formula (9). Where λ1, λ2, and λ3 are the weight values ​​for overlap, depth difference, and center point difference, respectively, and can be taken as 0.7, 0.15, and 0.15. move When the value is less than 0.9, the object is considered to be in motion, and the point to which it belongs is a dynamic target point.

[0082] r move =λ1r overlap +λ2r range +λ3r center (Formula 9)

[0083] S104. Generate a global point cloud map based on the pose transformation information and the point cloud frames from which dynamic target point clouds are removed.

[0084] Specifically, the generation of a global point cloud map based on pose transformation information and point cloud frames that have had dynamic target point clouds removed can be designed to include the following:

[0085] First, the pose estimated by the odometry of the current frame is used as the origin. Keyframes with a preset radius are found and a local map is built based on the keyframes.

[0086] Next, the pose estimated by inter-frame odometry is used as the initial value. The current frame is matched with the local map to remove dynamic target point clouds and determine the pose estimate of the current frame.

[0087] Then, based on the preset pose angle difference between the current frame and the previous frame of the point cloud data and the straight-line distance between the two frames, a list of key frames of the point cloud data is constructed, and the pose information of the key frames in the key frame list is stored in the key frame pose list.

[0088] Finally, dynamic target point clouds are removed from the keyframe list of point cloud data. Based on the keyframe list and keyframe pose list after removing dynamic target point clouds, point cloud stitching is performed to output a global point cloud map.

[0089] Specifically, a list of keyframes for the point cloud data is constructed based on the preset pose angle difference between the current frame and the previous frame, and the straight-line distance between the two frames. This list may include:

[0090] If the difference between the current frame and the previous frame in one of the three angles—pitch, roll, and yaw—is greater than a preset radian, or if the straight-line distance between the current frame and the previous frame is greater than a preset distance, then the point cloud data of the current frame will be added to the keyframe point cloud list.

[0091] Based on the above embodiments, the specific implementation algorithm may include: First, projecting multi-view data from the point cloud acquired in real-time by the LiDAR using a pre-defined multi-directional virtual camera to obtain depth projection maps in the x, Y, Z, or XYZ directions; second, using the YOLO v8 algorithm to perform target detection on the depth projection maps, obtaining information such as the bounding rectangle, segmentation mask, type, and target tracking ID of the detected target. Then, based on the target detection results, adding attribute information such as type ID, virtual camera ID, and target tracking ID to all 3D points in the point cloud. Finally, based on the coordinate transformation relationship between point cloud frames obtained from the Scan to Scan function in LiDAR SLAM, transforming the point cloud data of the current frame and the previous frame to a unified coordinate system, and comparing the point cloud overlap, average depth difference, and center point distance difference of regions with the same virtual camera ID and target tracking ID in the two point cloud frames. Regions with low point cloud overlap, large average depth difference, and far center point distance are marked as dynamic target points. Marked dynamic target points are not included in the subsequent Scan to Map pose optimization environment map construction. Points with the same virtual camera ID and target tracking ID are not included in the Scan to Scan pose estimation of the next frame. Compared with traditional LiDAR SLAM algorithms, this solution adds point cloud projection and target detection, and dynamic target removal to the existing point cloud preprocessing, inter-frame odometry, backend optimization and mapping algorithms.

[0092] In point cloud projection and target detection, the system receives point cloud data acquired in real time by a LiDAR scanner and simultaneously outputs raw point cloud data containing target detection information such as category IDs to the inter-frame odometry module. To achieve real-time point cloud target detection, this module consists of multiple processing nodes. Each node processes only the projection and target detection of one virtual camera, and the data flows to the next node after completion, thus completing the entire module's task in a workflow manner.

[0093] In dynamic target removal, the system receives pose transformation information and corresponding point cloud information from the inter-frame odometry output. After identifying dynamic targets, the system transmits the point cloud information after dynamic target removal to the backend optimization and mapping, and simultaneously returns the tracking ID of the identified dynamic target to the inter-frame odometry. After the inter-frame odometry receives the message, points marked with the same tracking ID will not be used in the inter-frame pose transformation estimation.

[0094] Furthermore, based on the above method, this embodiment of the invention also provides a dynamic target culling LiDAR SLAM processing system based on projection and deep learning, comprising: a projection mapping module, a detection and segmentation module, a pose acquisition module, and a map generation module, wherein,

[0095] The projection mapping module is used to obtain a depth projection map of the point cloud data based on the multi-view projection of the multi-angle virtual camera for the point cloud data collected in real time by the LiDAR, and to record the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map.

[0096] The detection and segmentation module is used to input the depth projection map into the pre-trained YOLOv8 object detection model, use the pre-trained YOLOv8 object detection model to perform object detection and segmentation and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type and the object tracking ID.

[0097] The pose acquisition module is used to obtain the pose transformation information between point cloud frames based on the target detection and segmentation results and the coordinate projection correspondence, and to determine the target overlap in the point cloud data so as to identify the dynamic target point cloud in the point cloud data based on the target overlap.

[0098] The map generation module is used to generate a global point cloud map based on pose transformation information and point cloud frames that have removed dynamic target point clouds.

[0099] To verify the effectiveness of this solution, the following explanation is based on experimental data:

[0100] based on Figure 2 The algorithm steps shown are validated on a moving vehicle. The specific steps for implementing the algorithm can be described as follows:

[0101] (1) The original point cloud acquired by Lidar is projected from multiple perspectives using a multi-angle virtual camera to obtain depth projection maps in the X, Y, Z or XYZ directions, and the correspondence between each point in the point cloud and its pixel coordinates in the projection map is recorded.

[0102] (2) Use the YOLO v8 algorithm to perform target detection and segmentation on the point cloud projection map, and record key information such as the bounding rectangle of the detected target, the segmentation mask, the category ID, and the tracking ID.

[0103] (3) Based on the target detection and segmentation results and the correspondence between point cloud projection, assign information such as category ID, tracking ID and virtual camera ID to the point cloud to achieve target detection of the point cloud.

[0104] (4) Arrange the original point cloud containing target detection information according to the line number during Lidar scanning, and perform corner point and surface point detection, while outputting corner point cloud and surface point cloud.

[0105] (5) Receive feature point information such as corner points and face points, and calculate the current frame P. c Up to the previous frame P l Pose transformation parameters T lcInformation is then published. Simultaneously, the dynamic target point information obtained in step 6 is received; points with the same virtual camera ID and target tracking ID as the dynamic target points are not included in the pose estimation of the next frame.

[0106] (6) Using the inter-frame transform parameters T obtained in step 5 lc , will the current frame P c Up to the previous frame P l Switch to the same coordinate system. Compare the point cloud overlap, average depth difference, and center point distance difference of regions with the same virtual camera ID and target tracking ID in the two point cloud frames. Regions with low point cloud overlap, large average depth difference, and far center point distance are marked as dynamic target points.

[0107] (7) Use the keyframe pose list pose The Kd-Tree is constructed using the input, and the pose estimated by the inter-frame odometry of the current frame is used as the origin to find the radius r. search Keyframes within 50m constitute a local map P. local .

[0108] (8) Using the pose estimated by inter-frame odometry in step 5 as the initial value, the pose of the current frame P is determined by Scan to Map. c To local map P local Matching is performed to further refine the pose estimation value of the current frame. At the same time, the dynamic target point information obtained in step 6 is received. Points with the same virtual camera ID and target tracking ID as the dynamic target points are not included in the pose optimization estimation.

[0109] (9) Calculate the current frame P c Compared to the previous frame P l The keyframe is determined by the differences in pitch, roll, and yaw angles, as well as the straight-line distance between the two points. If the difference in any of the three angles (pitch, roll, yaw) is greater than 0.2 radians, or the distance is greater than 1 meter, the current frame is considered a keyframe, and the point cloud data of the current frame is added to the keyframe point cloud list. pointcloud In the middle, the pose information is put into the keyframe pose list. pose middle.

[0110] (10) List pointcloud After retrieving all data and deleting dynamic target points, use a list. pose The pose information stored in the database is used to stitch the point cloud into a global point cloud map and output it.

[0111] The experimental data are as follows: Figure 5As shown in the figure, numbers 1-4 represent the projection results of four cars in four virtual cameras. It can be seen from the projection images that the structural characteristics of the same object appear different when viewed from different angles. The actual effect of using the YOLO v8 algorithm for object detection in projection images is shown below. Figure 6 As shown in the figure, id represents the tracking ID of the object. The same object has the same tracking ID in different images. Figure 7 The results show point cloud object detection; different colors indicate objects detected and tracked by different virtual cameras. Figure 8 The result of removing dynamic target points in the point cloud is shown in the figure. The points marked in red are the dynamic target points. Figure 9 The results of pose estimation and map construction are shown in the figure. The yellow curve represents the true pose value, and the green curve represents the estimated pose value.

[0112] The above data further demonstrates that the proposed solution can identify and remove moving objects in real time when such objects are present in the surrounding environment, thereby improving the accuracy of pose estimation in motion environments, solving problems such as ghosting in the created environmental maps, and enabling its application in the fields of intelligent robots and autonomous driving positioning and navigation.

[0113] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0115] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0116] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0117] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic target culling LiDAR SLAM processing method based on projection and deep learning, characterized in that, Include: For point cloud data collected in real time by lidar, a depth projection map of the point cloud data is obtained based on multi-view projection of a multi-directional virtual camera, and the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map is recorded. The depth projection map is input into the pre-trained YOLOv8 object detection model. The pre-trained YOLOv8 object detection model is used to perform object detection and segmentation and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type and the object tracking ID. Based on the target detection and segmentation results and previous dynamic target detection results, inter-frame pose transformation information of the point cloud is obtained, and the overlap degree of the same target in the point cloud data is calculated. Dynamic target point recognition is then performed based on the target overlap degree. The process of obtaining inter-frame pose transformation information of the point cloud based on the target detection and segmentation results and previous dynamic target detection results includes: First, target detection attribute annotation is performed on each 3D point cloud in the point cloud data according to the target detection and segmentation results and the coordinate projection correspondence. This attribute annotation includes point type ID, point tracking ID, and virtual camera ID. Then, for the original point cloud containing target detection attributes, it is arranged according to the LiDAR scan line number, and corner point cloud and face point cloud are obtained through corner point and face point detection. The system obtains inter-frame pose transformation information of point cloud based on corner point clouds and face point clouds. If there are points with the same attributes as previously detected dynamic target points in the corner point cloud and face point cloud, the points are removed from the point cloud. Dynamic target point clouds in point cloud data are identified based on target overlap, including: first, calculating the overlap, depth difference, and center point difference of point clouds in adjacent point cloud data frames based on the number of points of the target object in adjacent point cloud data frames; then, obtaining the comprehensive probability of target object motion by weighted summation of overlap, depth difference, and center point difference, and determining whether the target object is a moving object based on a preset threshold, and taking all point clouds of moving objects as dynamic target point clouds. A global point cloud map is generated based on pose transformation information and point cloud frames that have been used to remove dynamic target point clouds.

2. The dynamic target culling lidar SLAM processing method based on projection and deep learning according to claim 1, characterized in that, A depth projection map of point cloud data is obtained based on multi-view projection using a multi-angle virtual camera, including: Based on the intrinsic parameter matrix of the virtual camera, the coordinate position of the point cloud in the lidar coordinate system, and the rotation matrix from the virtual camera coordinate system to the lidar coordinate system, a projection expression is constructed to project the point cloud in the lidar coordinate system onto the pixel coordinate position in the virtual camera pixel coordinate system. Based on this projection expression, a depth projection map of the point cloud data in a predetermined direction is obtained. The virtual cameras are placed around the lidar sensor, and there is a viewpoint overlap between each virtual camera.

3. The dynamic target culling lidar SLAM processing method based on projection and deep learning according to claim 1, characterized in that, In the YOLOv8 object detection model, CSPDarknet53 is used as the backbone network, and the target loss function for model training is composed of standard binary cross-entropy loss, DFL loss and CIoU loss weighted together. The YOLOv8 object detection model is trained based on this target loss function and sample data.

4. The dynamic target culling lidar SLAM processing method based on projection and deep learning according to claim 1, characterized in that, A global point cloud map is generated based on pose transformation information and point cloud frames from which dynamic target point clouds are removed, including: First, the pose estimated by the odometry of the current frame is used as the origin. Keyframes with a preset radius are found and a local map is built based on the keyframes. Next, the pose estimated by inter-frame odometry is used as the initial value. The current frame is matched with the local map to remove dynamic target point clouds and determine the pose estimate of the current frame. Then, based on the preset pose angle difference between the current frame and the previous frame of the point cloud data and the straight-line distance between the two frames, a list of key frames of the point cloud data is constructed, and the pose information of the key frames in the key frame list is stored in the key frame pose list. Finally, dynamic target point clouds are removed from the keyframe list of point cloud data. Based on the keyframe list and keyframe pose list after removing dynamic target point clouds, point cloud stitching is performed to output a global point cloud map.

5. The dynamic target culling lidar SLAM processing method based on projection and deep learning according to claim 4, characterized in that, A list of keyframes for the point cloud data is constructed based on the preset pose angle difference between the current frame and the previous frame, and the straight-line distance between the two frames. This list includes: If the difference between the current frame and the previous frame in one of the three angles—pitch, roll, and yaw—is greater than a preset radian, or if the straight-line distance between the current frame and the previous frame is greater than a preset distance, then the point cloud data of the current frame will be added to the keyframe point cloud list.

6. A dynamic target culling lidar SLAM processing system based on projection and deep learning, characterized in that, It includes: a projection mapping module, a detection and segmentation module, a pose acquisition module, and a map generation module. The projection mapping module is used to obtain a depth projection map of the point cloud data based on the multi-view projection of the multi-angle virtual camera for the point cloud data collected in real time by the LiDAR, and to record the coordinate projection correspondence between each point cloud in the point cloud data and the pixel point in the depth projection map. The detection and segmentation module is used to input the depth projection map into the pre-trained YOLOv8 object detection model, use the pre-trained YOLOv8 object detection model to perform object detection and segmentation and record the key information of the detected object. The key information includes the bounding rectangle of the corresponding object, the segmentation mask, the object type and the object tracking ID. The pose acquisition module is used to obtain inter-frame pose transformation information of point clouds based on target detection and segmentation results and coordinate projection correspondence, and to determine the target overlap in the point cloud data. This overlap is used to identify dynamic target point clouds in the point cloud data. The process of obtaining inter-frame pose transformation information based on target detection and segmentation results and previous dynamic target detection results includes: first, labeling each 3D point cloud in the point cloud data with target detection attributes based on the target detection and segmentation results and coordinate projection correspondence. These attributes include point type ID, point tracking ID, and virtual camera ID. Then, for the original point cloud containing target detection attributes, the points are arranged according to the LiDAR scan line number, and corner points are obtained through corner and face point detection. Point clouds and face point clouds are used to obtain inter-frame pose transformation information of point clouds based on corner point clouds and face point clouds. If there are points with the same attributes as previously detected dynamic target points in the corner point clouds and face point clouds, the points are removed from the point clouds. Dynamic target point clouds in point cloud data are identified based on the target overlap degree, including: First, the overlap degree, depth difference degree, and center point difference degree of the point clouds in adjacent point cloud data frames are calculated based on the number of points of the target object in adjacent point cloud data frames; then, the overall motion probability of the target object is obtained by weighted summation of the overlap degree, depth difference degree, and center point difference degree, and a preset threshold is used to determine whether the target object is a moving object, and all point clouds of moving objects are used as dynamic target point clouds. The map generation module is used to generate a global point cloud map based on pose transformation information and point cloud frames that have removed dynamic target point clouds.

7. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 5.

Images