Dynamic object detection and filtering method during positioning

Abstract

The invention discloses a method for detecting and filtering a dynamic object during positioning, which comprises the following steps of: (1) projecting a laser point cloud to a camera pixel plane through external parameters of a camera and a laser radar; (2) detecting a potential dynamic object in the camera image through a target detection algorithm, and marking a point cloud in a detection frame corresponding to the potential dynamic object as a potential dynamic point; calculating the proportion of potential dynamic points in the whole point cloud, and adding the rest points into the static point cloud; (3) setting a dynamic point cloud threshold, and dynamically setting a centroid Euclidean distance threshold based on the object category of the target detection frame; (4) according to the proportion of the potential dynamic point clouds in the overall point clouds in the step (2), adaptively filtering the potential dynamic point clouds of different strategies to obtain static point clouds; and (5) positioning by using the static point cloud obtained by filtering. By utilizing the method, the positioning and mapping effects can be improved, and the method has the advantages of high robustness, high detection precision, good real-time performance and the like.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method for detecting and filtering dynamic objects during positioning. Background Art

[0002] In the fields of autonomous driving, robot navigation, etc., the fusion technology of LiDAR and camera plays an important role in environmental perception. LiDAR can provide high-precision three-dimensional point cloud data, while cameras can capture rich visual information. However, dynamic objects (such as pedestrians, vehicles, etc.) will interfere with the point cloud data of LiDAR, affecting the accuracy of positioning and mapping. Therefore, how to effectively identify and filter out these dynamic point clouds has become an important research direction.

[0003] In existing technologies, independent LiDAR or camera are usually used for dynamic object detection, but the detection accuracy and robustness of a single sensor are limited. By fusing the data of LiDAR and camera, dynamic objects can be identified more accurately, thereby improving the quality of point cloud data and improving the effects of positioning and mapping.

[0004] The invention patent with application number 202310868495.9 discloses a method, device, electronic device and storage medium for detecting feature points of dynamic objects. The method first obtains the current frame image and the previous frame image, extracts feature points from the current frame image, and generates image feature points. Then, object detection is performed on these feature points to determine the detection frame of the object, and the first background feature point set and the first object feature point set corresponding to each detection frame are determined according to the detection frame. If the total area of ​​the detection frame meets the preset conditions, the first background feature point set and the previous frame image are transmitted to determine the first displacement vector set corresponding to the background feature point. After that, for each detection frame, the second displacement vector set corresponding to the object is determined according to its corresponding first object feature point set and the previous frame image. Finally, based on the first displacement vector set and the second displacement vector set, it is determined whether the object in the detection frame is a dynamic object feature point. The main problem of this method is that it relies on a single image sensor, has low detection accuracy in complex environments such as light changes and rainy days, poor ability to cope with environmental changes, and weak robustness. In addition, the camera mainly provides two-dimensional image information and cannot obtain more comprehensive spatial information of dynamic points.

[0005] The invention patent with application number 202410501205.1 discloses a multi-sensor fusion dynamic object tracking method based on YOLO. The method first detects and matches the key feature points in the image, and then projects the laser point cloud onto the image. Then the YOLO algorithm is used to recognize the image target, and according to the matching relationship between the key feature points in the two adjacent frames of the image, the dynamic object is found in the adjacent image frames to achieve the tracking of the dynamic object. The problem with this method is that it only realizes the tracking of dynamic objects, but does not realize the filtering of dynamic objects, and cannot improve the accuracy of positioning in mapping. In addition, the method is designed to extract image feature points, generate descriptors and match processes, resulting in the algorithm's real-time performance is not strong enough. In addition, the algorithm directly regards the object obtained by target detection as a dynamic object, which in fact will lead to the situation where a stationary vehicle is mistakenly detected as a dynamic object, and the algorithm accuracy is not strong. Summary of the invention

[0006] In view of the problems of weak robustness, insufficient real-time performance, and low detection accuracy in the prior art, the present invention provides a method for detecting and filtering dynamic objects during positioning. It uses visual target detection combined with lidar data to detect and filter dynamic objects to improve positioning and mapping effects. It has the advantages of strong robustness, high detection accuracy, and good real-time performance.

[0007] A method for detecting and filtering dynamic objects during positioning, comprising the following steps:

[0008] (1) Project the laser point cloud onto the camera pixel plane using the external parameters of the camera and lidar;

[0009] (2) Detect potential dynamic objects in the camera image through the target detection algorithm, and mark the point cloud in the detection box corresponding to the potential dynamic object as a potential dynamic point; calculate the proportion of the potential dynamic point in the overall point cloud, and add the remaining points to the static point cloud;

[0010] (3) Setting a dynamic point cloud threshold: Based on the object category of the target detection box, dynamically set the threshold of the centroid Euclidean distance;

[0011] (4) According to the proportion of the potential dynamic point cloud in the overall point cloud in step (2), adaptively filter the potential dynamic point cloud using different strategies to obtain a static point cloud;

[0012] (5) Use the filtered static point cloud for positioning.

[0013] The specific process of step (1) is as follows:

[0014] (1-1) Calibrate the camera sensor internal parameters to obtain the camera internal parameter matrix K; calibrate the external parameters of the camera and lidar, including the rotation matrix from the laser sensor to the camera sensor and translation vectors Synchronize camera and lidar timestamps through hardware;

[0015] (1-2) Based on the rotation matrix obtained in step (1-1) and translation vectors Map the laser point cloud to the camera pixel plane.

[0016] The specific process of step (2) is as follows:

[0017] (2-1) Identify potential dynamic objects in the image through target detection algorithms;

[0018] (2-2) Based on the detection frame obtained in step (2-1), the points in the point cloud obtained in step (1) that are located in the potential dynamic object detection frame are set as potential dynamic points, and the proportion α of the potential dynamic points in the entire point cloud is calculated;

[0019] (2-3) Based on the potential dynamic points obtained in step (2-2), the remaining points in the original point cloud except the potential dynamic points are added to the static point cloud.

[0020] The specific process of step (3) is as follows:

[0021] (3-1) Setting the dynamic point cloud threshold β;

[0022] (3-2) Setting different centroid Euclidean distance thresholds γ i , where the subscript i corresponds to different preset objects, and the centroid Euclidean distance threshold of pedestrians is smaller than that of vehicles.

[0023] The specific process of step (4) is as follows:

[0024] (4-1) If the ratio α in step (2-2) is less than the threshold β set in step (3-1), all potential dynamic points are filtered as dynamic points, and the remaining points are added to the static point cloud;

[0025] (4-2) If the ratio α in step (2-2) is greater than the threshold β set in step (3-1), the static point cloud in step (2-3) is used for initial positioning, and the distance between the potential dynamic point in step (2-2) and its corresponding point in the past several frames is calculated. If the distance is greater than the γ in step (3-2), i , then the points in the potential dynamic point cluster are regarded as dynamic points, otherwise they are added to the static point cloud.

[0026] Step (5) also includes: saving the original point cloud for downstream path planning and obstacle avoidance.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention dynamically adjusts the filtering parameters according to the proportion of dynamic points and the categories of different objects to ensure that static points are not mistakenly deleted. Especially in complex scenes with many dynamic objects such as pedestrians and vehicles, the present invention can adaptively adjust the processing strategy to ensure positioning accuracy while retaining more static points, and further improve the stability of the static point cloud through the fusion of the centroid motion analysis method and multi-frame point cloud, effectively improving the robustness and practicality of the system. In addition, the dynamic point cloud filtering and static point cloud of the present invention are used for positioning and mapping, while the original point cloud is used for obstacle avoidance and path planning. It is multifunctional and can adapt to complex practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a flowchart of a method for detecting and filtering dynamic objects during positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0031] The core technology of this invention is to accurately distinguish static and dynamic objects through the fusion processing of vision and laser point cloud, combined with target detection and multi-scale Euclidean distance judgment, thereby improving the accuracy of point cloud positioning and mapping in complex dynamic environments.

[0032] like Figure 1 As shown, a method for detecting and filtering dynamic objects during positioning includes the following steps:

[0033] S01, project the laser point cloud onto the camera pixel plane through external parameters.

[0034] S02, detect potential dynamic objects through the target detection algorithm, mark the corresponding points as potential dynamic points, calculate the proportion of potential dynamic points in the overall point cloud, and add the remaining points to the static point cloud.

[0035] S03, multi-scale threshold setting: set the dynamic point cloud threshold, dynamically set the threshold of the centroid Euclidean distance based on the object category of the target detection box.

[0036] S04, adaptive dynamic point cloud filtering: According to the proportion of potential dynamic point cloud in the overall point cloud, adaptively perform potential dynamic point cloud filtering with different strategies to obtain static point cloud.

[0037] S05, using the static point cloud and the original point cloud for different processing flows.

[0038] The present invention first calibrates the camera sensor internal parameters to obtain the camera internal parameter matrix K; calibrates the external parameters of the camera and the laser radar, including the rotation matrix and translation vectors They represent the rotation matrix and translation vector from the laser sensor to the camera sensor respectively; the timestamps of the camera and lidar are synchronized through hardware.

[0039] Based on the obtained external parameter rotation matrix and translation vectors Map the laser point cloud to the camera pixel plane using the following formula:

[0040]

[0041] in, is the coordinate of the nth point of the mth frame point cloud in the laser radar coordinate system, s m,n is the depth of the point after being projected into the camera coordinate system, p m,n Indicates the pixel coordinates of the point. In this way, the pixel coordinates of the laser point cloud in the two-dimensional image are obtained.

[0042] Through the target detection algorithm, potential preset dynamic objects in the image, such as people and vehicles, are identified. According to the results of target detection, the point cloud located in the potential dynamic object detection box is set as potential dynamic points, and the proportion α of potential dynamic points in the entire point cloud is calculated.

[0043] Set the dynamic point cloud ratio threshold β and set different centroid Euclidean distance thresholds γ i , where the subscript i corresponds to different preset objects, and the centroid Euclidean distance threshold of pedestrians is stricter than that of vehicles.

[0044] If the potential dynamic point ratio α is less than the preset threshold β, all potential dynamic points are filtered as dynamic points, and the remaining points are added to the static point cloud;

[0045] Otherwise, if the potential dynamic point ratio α is less than the preset threshold β, the static point cloud is initially positioned to obtain the initial estimated pose of the current frame. and They represent the pose of the current k-th frame in the world coordinate system w. Project the corresponding point cloud of the past n frames to the current k-th frame coordinate system according to the pose of the corresponding frame to obtain the corresponding coordinates, which are shown as follows:

[0046]

[0047] in, and Respectively represent the rotation matrix and translation vector from the i-th frame to the j-th frame, where p i represents the coordinates of the point in the i-th frame, p k represents the coordinates of the point in the kth frame. To simplify the superscripts and subscripts, the specific index of the point in a certain frame is omitted in the formula. Therefore, this formula can be expressed as the coordinate transformation of any point in the i-th frame to the k-th frame.

[0048]

[0049] Calculate the average centroid of any detection box in the current frame and n frames before the current frame:

[0050]

[0051] Where N is the number of points in the corresponding detection frame. l represents the number of frames, l = {kn,…,k-2,k-1}

[0052] Calculate the distance between the potential dynamic point and its corresponding point in the past n frames:

[0053]

[0054] If d is greater than the preset centroid Euclidean distance threshold γ i , then the points in the potential dynamic point cluster are regarded as dynamic points, otherwise they are added to the static point cloud.

[0055] At this point, the dynamic point cloud and the final static point cloud are obtained.

[0056] Based on the obtained static point cloud, re-positioning is performed to obtain the final pose of the current frame. The original point cloud is saved for downstream path planning and obstacle avoidance.

[0057] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting and filtering dynamic objects during positioning, characterized in that: The following steps are involved: (1) Project the laser point cloud onto the camera pixel plane using the external parameters of the camera and lidar; (2) Detect potential dynamic objects in the camera image through the object detection algorithm, and mark the point cloud in the detection box corresponding to the potential dynamic object as a potential dynamic point; Calculate the proportion of potential dynamic points in the overall point cloud and add the remaining points to the static point cloud; (3) Setting a dynamic point cloud threshold: Based on the object category of the target detection box, dynamically set the threshold of the centroid Euclidean distance; (4) According to the proportion of the potential dynamic point cloud in the overall point cloud in step (2), adaptively filter the potential dynamic point cloud using different strategies to obtain a static point cloud; (5) Use the filtered static point cloud for positioning.

2. The method for detecting and filtering dynamic objects during positioning according to claim 1, characterized in that: The specific process of step (1) is as follows: (1-1) Calibrate the camera sensor internal parameters to obtain the camera internal parameter matrix K; calibrate the external parameters of the camera and lidar, including the rotation matrix from the laser sensor to the camera sensor and translation vectors Synchronize camera and lidar timestamps through hardware; (1-2) Based on the rotation matrix obtained in step (1-1) and translation vectors Map the laser point cloud to the camera pixel plane.

3. The method for detecting and filtering dynamic objects during positioning according to claim 2, characterized in that: The specific process of step (2) is as follows: (2-1) Identify potential dynamic objects in the image through target detection algorithms; (2-2) Based on the detection frame obtained in step (2-1), the points in the point cloud obtained in step (1) that are located in the potential dynamic object detection frame are set as potential dynamic points, and the proportion α of the potential dynamic points in the entire point cloud is calculated; (2-3) Based on the potential dynamic points obtained in step (2-2), the remaining points in the original point cloud except the potential dynamic points are added to the static point cloud.

4. The method for detecting and filtering dynamic objects during positioning according to claim 3, characterized in that: The specific process of step (3) is as follows: (3-1) Setting the dynamic point cloud threshold β; (3-2) Setting different centroid Euclidean distance thresholds γ i , where the subscript i corresponds to different preset objects, and the centroid Euclidean distance threshold of pedestrians is smaller than that of vehicles.

5. The method for detecting and filtering dynamic objects during positioning according to claim 4, characterized in that: The specific process of step (4) is as follows: (4-1) If the ratio α in step (2-2) is less than the threshold β set in step (3-1), all potential dynamic points are filtered as dynamic points, and the remaining points are added to the static point cloud; (4-2) If the ratio α in step (2-2) is greater than the threshold β set in step (3-1), the static point cloud in step (2-3) is used for initial positioning, and the distance between the potential dynamic point in step (2-2) and its corresponding point in the past several frames is calculated. If the distance is greater than the γ in step (3-2), i , then the points in the potential dynamic point cluster are regarded as dynamic points, otherwise they are added to the static point cloud.

6. The method for detecting and filtering dynamic objects during positioning according to claim 1, characterized in that: Step (5) also includes: saving the original point cloud for downstream path planning and obstacle avoidance.

Images