Lidar and camera based long range obstacle detection method and system

By performing ground segmentation and road boundary extraction on the LiDAR point cloud, combining it with the DBSCAN algorithm for two-stage clustering, and fitting depth values ​​using camera image features, the problem of low accuracy in long-distance obstacle detection by LiDAR and cameras is solved, achieving more efficient obstacle detection.

CN118038411BActive Publication Date: 2026-11-10ARMY MILITARY TRANSPORTATION UNIV OF PLA ZHENJIANG
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410221375.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-28
Publication Date
2026-11-10
Estimated Expiration
2044-02-28

AI Technical Summary

Technical Problem

In existing technologies, LiDAR and cameras suffer from low accuracy in detecting obstacles at long distances and coarse fusion granularity, which affects the perception range and safety of intelligent vehicles.

Method used

By performing ground segmentation and road boundary extraction on the LiDAR point cloud, and combining the DBSCAN algorithm for two-stage clustering, depth values ​​are fitted using camera image features to achieve accurate obstacle detection.

Benefits of technology

It improves the accuracy and granularity of long-distance obstacle detection, enhances the environmental perception capabilities of intelligent vehicles, increases the detection distance by 11.75m, and has high operating efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118038411B_ABST
    Figure CN118038411B_ABST
Patent Text Reader

Abstract

The application discloses a kind of long-range obstacle detection method and system based on laser radar and camera, it is related to obstacle detection technical field, including: obtaining original point cloud and original image;Original point cloud is carried out ground segmentation and road boundary extraction processing, and the point cloud after processing is obtained;Based on the DBSCAN algorithm under the first parameter, the point cloud after processing is carried out one-stage clustering, and the initial point cloud cluster is obtained;Based on calibration information, the initial point cloud cluster is projected into the target region in the pixel coordinate system of original image;Based on straight line and plane fitting, the target region is estimated in depth, and difference image is obtained;Difference image is projected to laser radar coordinate system, and estimation point cloud is obtained;Based on the DBSCAN algorithm under the second parameter, the estimation point cloud is carried out two-stage clustering, and the detection point cloud cluster of target obstacle is obtained.The application alleviates the technical problems of low detection accuracy and coarse fusion granularity for long-range obstacles in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of obstacle detection technology, specifically to a long-range obstacle detection method and system based on lidar and camera. Background Technology

[0002] Intelligent vehicle environmental perception involves using sensors to acquire information about its surrounding environment, providing data on drivable space for decision-making and planning, and ensuring that intelligent vehicles drive safely and smoothly.

[0003] Currently, LiDAR and cameras are the two most commonly used sensing sensors. LiDAR generates point cloud data by actively emitting and receiving laser light, obtaining precise three-dimensional distances and reflectivity of reflection points. Thanks to its accurate ranging capabilities, it is widely used in obstacle detection, localization, and mapping. Cameras acquire high-density, rich-textured environmental information, more closely resembling the eye's perception of the environment, and are also frequently used for obstacle detection and tracking, localization, and mapping.

[0004] Both lidar and cameras have their limitations. Compared to the overall environment, lidar provides limited data, exhibiting a distribution pattern of denser data at near distances and sparser data at farther distances. This can lead to missed detection of distant obstacles, reducing the perception range and consequently affecting driving safety and smoothness. Besides being susceptible to background interference, lighting changes, and occlusion, cameras currently cannot provide precise depth information; even depth cameras can only provide depth information over a small area. To fully utilize the advantages of both sensors, existing technologies fuse lidar and camera information for obstacle detection and tracking, but this generally remains at the level of obstacle type identification and narrowing down the processing area, with relatively coarse granularity in the fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for long-range obstacle detection based on lidar and camera in order to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a method for long-range obstacle detection based on lidar and a camera, comprising: acquiring an original point cloud containing a distant ground and road based on lidar, and acquiring an original image containing a distant ground and road based on a camera; performing ground segmentation and road boundary extraction processing on the original point cloud to obtain a processed point cloud; performing one-stage clustering on the processed point cloud based on the DBSCAN algorithm under a first parameter to obtain an initial point cloud cluster; the first parameter includes a first neighborhood radius and a first neighborhood point cloud threshold; and based on calibration information, dividing the initial point cloud cluster... The target region is projected into the pixel coordinate system of the original image; the depth of the target region is estimated based on line and plane fitting to obtain a difference image; the difference image is back-projected into the lidar coordinate system based on the calibration information to obtain an estimated point cloud; the estimated point cloud is clustered in two stages based on the DBSCAN algorithm under the second parameter to obtain a detection point cloud cluster of the target obstacle; the second parameter includes a second neighborhood radius and a second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold.

[0007] Further, the original point cloud is subjected to ground segmentation and road boundary extraction processing, including: filtering out point clouds in the original point cloud whose height is lower than a first preset threshold; selecting a preset number of seed points in the filtered point cloud in ascending order of height value, and calculating the average height of the preset number of seed points; selecting ground candidate points based on the average height and a second preset threshold; performing iterative plane fitting on the ground candidate points to obtain the optimal ground fitting plane parameters; filtering out ground point clouds based on the optimal ground fitting plane parameters to perform ground segmentation on the original point cloud; and filtering out point clouds outside the road boundaries by obtaining the road boundaries in the original point cloud.

[0008] Further, the calibration information includes a camera intrinsic parameter matrix; based on the calibration information, projecting the initial point cloud cluster onto a target region within the pixel coordinate system of the original image includes: projecting the initial point cloud cluster onto a target region within the pixel coordinate system of the original image using the following coordinate transformation formula: Where (x,y,z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and the 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix, (u,v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system.

[0009] Further, depth estimation of the target region based on line and plane fitting includes: performing horizontal feature value line fitting on the pixels in the target region to obtain multiple fitted lines; performing vertical feature difference plane fitting on the multiple fitted lines to obtain a fitting plane of the target region; and performing interpolation estimation of pixel depth values ​​based on the fitting plane.

[0010] Secondly, embodiments of the present invention also provide a long-range obstacle detection system based on lidar and a camera, comprising: an acquisition module, a preprocessing module, a first clustering module, a projection module, an estimation module, a back-projection module, and a second clustering module; wherein, the acquisition module is used to acquire an original point cloud containing a distant ground and road based on lidar, and to acquire an original image containing a distant ground and road based on a camera; the preprocessing module is used to perform ground segmentation and road boundary extraction processing on the original point cloud to obtain a processed point cloud; the first clustering module is used to perform a one-stage clustering of the processed point cloud based on the DBSCAN algorithm under a first parameter to obtain an initial point cloud cluster; the first parameter includes a first neighborhood radius and a first neighborhood point cloud. The threshold; the projection module is used to project the initial point cloud cluster onto the target region in the pixel coordinate system of the original image based on calibration information; the estimation module is used to perform depth estimation of the target region based on line and plane fitting to obtain a difference image; the back projection module is used to back project the difference image onto the lidar coordinate system based on the calibration information to obtain an estimated point cloud; the second clustering module is used to perform two-stage clustering of the estimated point cloud based on the DBSCAN algorithm under the second parameter to obtain a detection point cloud cluster of the target obstacle; the second parameter includes a second neighborhood radius and a second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius, and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold.

[0011] Furthermore, the preprocessing module is also configured to: filter out point clouds in the original point cloud whose height is lower than a first preset threshold; select a preset number of seed points in the filtered point cloud in ascending order of height value, and calculate the average height of the preset number of seed points; select ground candidate points based on the average height and a second preset threshold; perform iterative plane fitting on the ground candidate points to obtain the optimal ground fitting plane parameters; filter out ground point clouds based on the optimal ground fitting plane parameters to perform ground segmentation on the original point cloud; and filter out point clouds outside the road boundaries by obtaining the road boundaries in the original point cloud.

[0012] Furthermore, the calibration information includes a camera intrinsic parameter matrix; the projection module is also used to: project the initial point cloud cluster onto a target region within the pixel coordinate system of the original image using the following coordinate transformation formula: Where (x,y,z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and the 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix, (u,v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system.

[0013] Furthermore, the estimation module is also used to: perform horizontal feature value line fitting on the pixels in the target region to obtain multiple fitted lines; perform vertical feature difference plane fitting on the multiple fitted lines to obtain a fitting plane of the target region; and perform interpolation estimation of pixel depth values ​​based on the fitting plane.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in the first aspect.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0016] This invention provides a method and system for long-range obstacle detection based on LiDAR and a camera. First, the original point cloud is segmented into ground and road boundaries are extracted. Then, the long-range point cloud on the road is clustered in one stage using a relaxed-parameter DBSCAN algorithm. The clustered point cloud clusters are projected onto the image pixel coordinate system using calibration information. Next, focusing on pixel value and depth features, the depth values ​​of some pixels without depth values ​​are estimated by fitting feature value lines laterally and feature difference planes vertically. After successful estimation, the pixel coordinates are back-projected onto the LiDAR coordinate system, and a second stage of clustering is performed using a strict-parameter DBSCAN algorithm to achieve accurate detection of long-range obstacles. This invention alleviates the technical problems of low accuracy and coarse granularity in the detection of long-range obstacles in existing technologies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a long-range obstacle detection method based on lidar and camera, provided for an embodiment of the present invention;

[0019] Figure 2 A point cloud map after ground segmentation and road boundary extraction is provided in an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a LiDAR road boundary area scanning provided in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the DBSCAN algorithm provided in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a cluster of points obtained by clustering and a projected image provided in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of a long-range obstacle detection system based on lidar and camera, provided as an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1

[0026] Figure 1 This is a flowchart of a long-range obstacle detection method based on lidar and camera according to an embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps:

[0027] Step S102: Acquire the original point cloud containing distant ground and roads based on lidar, and acquire the original image containing distant ground and roads based on camera.

[0028] Step S104: Perform ground segmentation and road boundary extraction on the original point cloud to obtain the processed point cloud.

[0029] Step S106: Based on the DBSCAN algorithm under the first parameter, perform one-stage clustering on the processed point cloud to obtain an initial point cloud cluster; the first parameter includes the first neighborhood radius and the first neighborhood point cloud threshold.

[0030] Step S108: Based on the calibration information, project the initial point cloud clusters onto the target region within the pixel coordinate system of the original image.

[0031] Step S110: Depth estimation of the target region is performed based on line and plane fitting to obtain a difference image.

[0032] Step S112: Based on the calibration information, the difference image is back-projected into the lidar coordinate system to obtain the estimated point cloud.

[0033] Step S114: Based on the DBSCAN algorithm under the second parameter, perform two-stage clustering on the estimated point cloud to obtain the detection point cloud cluster of the target obstacle; the second parameter includes the second neighborhood radius and the second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold.

[0034] Specifically, step S104 also includes the following steps:

[0035] Step S1041: Filter out point clouds in the original point cloud whose height is lower than the first preset threshold.

[0036] Step S1042: Select a preset number of seed points from the filtered point cloud in order of height value from low to high, and calculate the average height of the preset number of seed points.

[0037] Step S1043: Select ground candidate points based on average height and a second preset threshold;

[0038] Step S1044: Perform iterative plane fitting on the ground candidate points to obtain the optimal ground fitting plane parameters;

[0039] Step S1045: Based on the best ground fitting plane parameters, filter out the ground point cloud to perform ground segmentation on the original point cloud;

[0040] Step S1046: By obtaining the road boundary in the original point cloud, the point cloud outside the road boundary is filtered out.

[0041] Specifically, the raw point cloud of lidar contains a large number of point clouds outside the ground and road boundaries, which need to be filtered out in advance to improve the accuracy of point cloud clustering within the road and the efficiency of the method. This includes two steps: ground segmentation and road boundary extraction.

[0042] The core idea of ​​ground segmentation of the original point cloud is as follows: after filtering out point clouds with excessively low heights, a certain number of seed points are selected in ascending order of height values ​​to calculate the average height. Then, a threshold is applied to this average height to select candidate ground points. Iterative plane fitting is performed on all candidate ground points to obtain the optimal ground fitting plane parameters, thereby filtering out ground point clouds. Figure 2This is a point cloud map provided by an embodiment of the present invention, after ground segmentation and road boundary extraction.

[0043] Point clouds outside the road boundary can be filtered out by extracting road boundaries. Figure 3 This is a schematic diagram of a LiDAR road boundary area scanning method according to an embodiment of the present invention. Figure 3 As shown by the dashed lines, each scan line of the lidar rotates one revolution, and the resulting point cloud after motion compensation forms an approximate "ring". Compared to the road point cloud A, the roadside center point cloud C, and the sidewalk point cloud D, the road edge point cloud B exhibits abrupt changes in curvature, height, and other information. To eliminate noise interference, a sliding window method can be used to acquire a certain number of point clouds to the left and right of B. The mean and variance are then used to verify and extract the road edge. The extraction results are shown below. Figure 2 As shown.

[0044] Specifically, the point cloud reflected by an obstacle exhibits a strong correlation in terms of distance and density characteristics, and this correlation is the core principle for achieving cluster detection. The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is a typical spatial clustering algorithm based on distance and density information. Figure 4 This is a schematic diagram of a DBSCAN algorithm provided according to an embodiment of the present invention, such as... Figure 4 As shown, the core idea of ​​this algorithm is to start from an initial seed point, obtain the point cloud within the neighborhood radius, and iteratively expand the search by traversing the point cloud within the neighborhood, continuously expanding the clustered point cloud clusters until the number of point clouds in the iterative neighborhood is lower than the neighborhood point cloud threshold, thereby quickly and correctly clustering point cloud clusters of arbitrary shapes, and having good adaptability to noisy point clouds.

[0045] As shown above, the neighborhood radius r and the neighborhood point cloud threshold n min These are the core parameters of the DBSCAN algorithm. In this embodiment of the invention, when r is a small number and n... min Large values ​​indicate strict parameters, while smaller values ​​indicate loose parameters. According to the algorithm principle, overly strict parameters may lead to over-segmentation of the target, while overly loose parameters may lead to under-segmentation.

[0046] The hardware structure of LiDAR results in point clouds exhibiting a distribution pattern of denser points at near distances and sparser points at farther distances. It is easy to see that if fixed parameters are used, the clustering effect of the DBSCAN algorithm will change with distance. Generally, to meet the requirement of accurate clustering of nearby obstacles, the parameters are set to be relatively strict, which will lead to missed detection of distant obstacles and affect the environmental perception range. To achieve accurate detection of distant obstacles, this embodiment of the invention will perform two-stage clustering based on the DBSCAN algorithm: the first stage clustering uses relaxed parameters. Based on the first stage clustering, the second stage fuses the existing point clouds and those obtained through depth estimation, using strict parameters to complete the clustering.

[0047] Specifically, the calibration information includes the camera intrinsic parameter matrix; step S108 also includes the following steps:

[0048] The initial point cloud cluster is projected onto the target region within the pixel coordinate system of the original image using the following coordinate transformation formula:

[0049]

[0050] Where (x, y, z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix of size 3×3, (u, v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system. The clustered point cloud clusters and the projected image are shown below. Figure 5 As shown.

[0051] Compared to individual pixels, point cloud clusters projected onto an image have a sparser arrangement. However, within the local area of ​​the projection region, if pixel values ​​change smoothly, depth values ​​also change smoothly. Based on this, for pixels with existing depth values, two features are considered: horizontally, a linear feature value is fitted; vertically, although the same feature value differs across rows, the feature difference changes smoothly, so a difference plane is fitted. If the fitting is successful, pixels to be estimated are sampled according to the median principle. The estimated depth is obtained for these pixels by first verifying pixel values ​​and then estimating depth values. Finally, the pixel coordinates and depth values ​​are fused and back-projected into the LiDAR coordinate system using a coordinate transformation formula to increase the density of the clustered point cloud.

[0052] Specifically, step S110 also includes the following steps:

[0053] Step S1101: Perform horizontal feature value line fitting on the pixels in the target area to obtain multiple fitted lines;

[0054] Step S1102: Perform longitudinal feature difference plane fitting on multiple fitted lines to obtain the fitting plane of the target region;

[0055] Step S1103: Based on the fitted plane, perform interpolation estimation of pixel depth values.

[0056] Specifically, the lateral eigenvalue linear fitting includes the following steps:

[0057] Horizontally, the point cloud is projected onto the same row of the pixel coordinate system, generating m pixels p1, p1...p... m Its composition is as shown in equation (1), the average value of all points is as shown in equation (2), and the covariance matrices for grayscale pixels, depth and horizontal coordinates are constructed as shown in equations (3) and (4):

[0058] p i =(u,σ,d),i∈(0,i) (1)

[0059]

[0060]

[0061]

[0062] In equation (1): u, σ, and d are the x-coordinate, pixel value, and depth value of the pixel, respectively. For M σ M d The singular value decomposition (SVD) method is used to obtain the eigenvalues ​​λ1 and λ2 and the eigenvectors δ1 and δ2. When the eigenvalues ​​meet the linearity requirement (λ1>3λ2), the direction vector of the fitted line is δ1.

[0063] Specifically, the longitudinal feature difference plane fitting includes the following steps:

[0064] Based on the linear fitting, a feature difference plane is fitted. It is assumed that two straight lines for fitting the features of different rows of pixels are l1(q1, q2...q...). m ),l2(q'1,q'2...q' n ), generate an mn×3 dimensional difference matrix, as shown in equation (5):

[0065]

[0066] In the formula, q and q' are (u,v,σ), for matrix M mn Perform singular value decomposition using ×3 to obtain the normal vector of the fitted plane.

[0067] For depth features, the difference matrix is ​​constructed and the normal vector of the fitting plane is obtained in the same way as for pixel features.

[0068] As described above, the embodiments of the present invention provide a long-range obstacle detection method based on lidar and camera. Compared with the traditional DBSCAN algorithm and the adaptive threshold DBSCAN algorithm, the present invention can improve the stable obstacle detection distance by 11.75m and 3.74m respectively, and can make up for the weakness of the adaptive threshold DBSCAN algorithm in anti-interference ability when clustering long-range obstacles. The average time to detect all obstacles in front of the intelligent vehicle based on the method provided by the embodiments of the present invention is 61.5ms, and the overall operating efficiency is high.

[0069] Example 2

[0070] Figure 6 This is a schematic diagram of a long-range obstacle detection system based on lidar and camera according to an embodiment of the present invention. Figure 6 As shown, the system includes: an acquisition module 10, a preprocessing module 20, a first clustering module 30, a projection module 40, an estimation module 50, a back projection module 60, and a second clustering module 70.

[0071] Specifically, the acquisition module 10 is used to acquire raw point clouds containing distant ground and roads based on lidar, and to acquire raw images containing distant ground and roads based on a camera.

[0072] The preprocessing module 20 is used to perform ground segmentation and road boundary extraction on the original point cloud to obtain the processed point cloud.

[0073] The first clustering module 30 is used to perform one-stage clustering on the processed point cloud based on the DBSCAN algorithm under the first parameter to obtain an initial point cloud cluster; the first parameter includes the first neighborhood radius and the first neighborhood point cloud threshold.

[0074] Projection module 40 is used to project the initial point cloud cluster onto the target area within the pixel coordinate system of the original image based on calibration information.

[0075] The estimation module 50 is used to estimate the depth of the target region based on line and plane fitting to obtain a difference image.

[0076] The back projection module 60 is used to back project the difference image onto the lidar coordinate system based on the calibration information to obtain the estimated point cloud.

[0077] The second clustering module 70 is used to perform two-stage clustering on the estimated point cloud based on the DBSCAN algorithm under the second parameter to obtain the detection point cloud cluster of the target obstacle; the second parameter includes the second neighborhood radius and the second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold.

[0078] Specifically, preprocessing module 20 is also used for:

[0079] Points with heights below a first preset threshold in the original point cloud are filtered out.

[0080] After filtering, select a preset number of seed points in the point cloud according to the height value from low to high, and calculate the average height of the preset number of seed points.

[0081] Ground candidate points are selected based on average height and a second preset threshold;

[0082] Iterative plane fitting is performed on the ground candidate points to obtain the optimal ground fitting plane parameters;

[0083] Based on the optimal ground-fitting plane parameters, ground point clouds are filtered out to perform ground segmentation on the original point cloud;

[0084] By obtaining the road boundaries from the original point cloud, the point cloud outside the road boundaries is filtered out.

[0085] Specifically, the calibration information includes the camera intrinsic parameter matrix; the projection module 40 is also used for:

[0086] The initial point cloud cluster is projected onto the target region within the pixel coordinate system of the original image using the following coordinate transformation formula:

[0087]

[0088] Where (x,y,z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and the 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix, (u,v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system.

[0089] Specifically, module 50 is estimated to also be used for:

[0090] By performing lateral feature value line fitting on the pixels within the target area, multiple fitted lines are obtained;

[0091] By performing longitudinal feature difference plane fitting on multiple fitted lines, a fitting plane for the target region is obtained;

[0092] Based on the fitted plane, interpolation estimation of pixel depth values ​​is performed.

[0093] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in Embodiment 1.

[0094] This invention also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described in Embodiment 1.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0096] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for long-range obstacle detection based on lidar and camera, characterized in that, include: The method involves acquiring raw point clouds containing distant ground and roads using lidar and acquiring raw images containing distant ground and roads using a camera. The original point cloud is subjected to ground segmentation and road boundary extraction to obtain the processed point cloud; Based on the DBSCAN algorithm with the first parameter, a one-stage clustering is performed on the processed point cloud to obtain an initial point cloud cluster; the first parameter includes the first neighborhood radius and the first neighborhood point cloud threshold. Based on the calibration information, the initial point cloud cluster is projected onto the target region within the pixel coordinate system of the original image; Depth estimation of the target region is performed based on line and plane fitting to obtain a difference image; Based on the calibration information, the difference image is back-projected onto the lidar coordinate system to obtain the estimated point cloud; Based on the DBSCAN algorithm under the second parameter, the estimated point cloud is clustered in two stages to obtain the detection point cloud cluster of the target obstacle; the second parameter includes the second neighborhood radius and the second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold. Depth estimation of the target region based on line and plane fitting includes: By performing lateral feature value line fitting on the pixels within the target region, multiple fitted lines are obtained; By performing longitudinal feature difference plane fitting on the plurality of fitted lines, a fitting plane for the target region is obtained; Based on the fitted plane, interpolation estimation of pixel depth values ​​is performed; The longitudinal feature difference plane fitting includes: Assume that the two lines fitted to the pixel value features of different rows are l1(q1,q2...q...). m ),l2(q'1,q'2...q' n Generate an mn×3 dimensional difference matrix, as shown in the following equation: In the formula, q and q' represent (u,v,σ), where (u,v) are the coordinates of the point cloud in the pixel coordinate system, and σ is the pixel value. For matrix M... mn Perform singular value decomposition using ×3 to obtain the normal vector of the fitted plane.

2. The method according to claim 1, characterized in that: The original point cloud is subjected to ground segmentation and road boundary extraction processing, including: Points with heights lower than a first preset threshold in the original point cloud are filtered out. After filtering, select a preset number of seed points from the point cloud in order of height value from low to high, and calculate the average height of the preset number of seed points. Based on the average height and the second preset threshold, candidate ground points are selected; Iterative plane fitting is performed on the candidate ground points to obtain the optimal ground fitting plane parameters; Based on the optimal ground fitting plane parameters, the ground point cloud is filtered out to perform ground segmentation on the original point cloud; By obtaining the road boundaries in the original point cloud, the point cloud outside the road boundaries is filtered out.

3. The method according to claim 1, characterized in that: The calibration information includes the camera intrinsic parameter matrix; Based on the calibration information, the initial point cloud cluster is projected onto a target region within the pixel coordinate system of the original image, including: The initial point cloud cluster is projected onto the target region within the pixel coordinate system of the original image using the following coordinate transformation formula: Where (x,y,z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and the 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix, (u,v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system.

4. A long-range obstacle detection system based on lidar and camera, characterized in that, include: The module comprises an acquisition module, a preprocessing module, a first clustering module, a projection module, an estimation module, a backprojection module, and a second clustering module; among which, The acquisition module is used to acquire raw point clouds containing distant ground and roads based on lidar, and to acquire raw images containing distant ground and roads based on a camera; The preprocessing module is used to perform ground segmentation and road boundary extraction on the original point cloud to obtain the processed point cloud. The first clustering module is used to perform one-stage clustering on the processed point cloud based on the DBSCAN algorithm under the first parameter to obtain an initial point cloud cluster; the first parameter includes a first neighborhood radius and a first neighborhood point cloud threshold. The projection module is used to project the initial point cloud cluster onto a target area within the pixel coordinate system of the original image based on calibration information. The estimation module is used to perform depth estimation of the target region based on line and plane fitting to obtain a difference image; The back projection module is used to back project the difference image onto the lidar coordinate system based on the calibration information to obtain the estimated point cloud; The second clustering module is used to perform two-stage clustering on the estimated point cloud based on the DBSCAN algorithm under the second parameters to obtain the detection point cloud cluster of the target obstacle; the second parameters include the second neighborhood radius and the second neighborhood point cloud threshold, wherein the second neighborhood radius is smaller than the first neighborhood radius and the second neighborhood point cloud threshold is greater than the first neighborhood point cloud threshold. The estimation module is further used for: By performing lateral feature value line fitting on the pixels within the target region, multiple fitted lines are obtained; By performing longitudinal feature difference plane fitting on the plurality of fitted lines, a fitting plane for the target region is obtained; Based on the fitted plane, interpolation estimation of pixel depth values ​​is performed; The longitudinal feature difference plane fitting includes: Assume that the two straight lines fitted to the pixel value features of different rows are l1(q1, q2 ...q m ),l2(q'1,q'2...q' n Generate an mn×3 dimensional difference matrix, as shown in the following equation: In the formula, q and q' represent (u,v,σ), where (u,v) are the coordinates of the point cloud in the pixel coordinate system, and σ is the pixel value. For matrix M... mn Perform singular value decomposition using ×3 to obtain the normal vector of the fitted plane.

5. The system according to claim 4, characterized in that: The preprocessing module is also used for: Points with heights lower than a first preset threshold in the original point cloud are filtered out. After filtering, select a preset number of seed points from the point cloud in order of height value from low to high, and calculate the average height of the preset number of seed points. Based on the average height and the second preset threshold, candidate ground points are selected; Iterative plane fitting is performed on the candidate ground points to obtain the optimal ground fitting plane parameters; Based on the optimal ground fitting plane parameters, the ground point cloud is filtered out to perform ground segmentation on the original point cloud; By obtaining the road boundaries in the original point cloud, the point cloud outside the road boundaries is filtered out.

6. The system according to claim 4, characterized in that: The calibration information includes the camera intrinsic parameter matrix; the projection module is also used for: The initial point cloud cluster is projected onto the target region within the pixel coordinate system of the original image using the following coordinate transformation formula: Where (x,y,z) are the coordinates of the point cloud in the radar coordinate system, R and t are the 3×3 rotation matrix and the 3×1 translation vector, respectively, K is the camera intrinsic parameter matrix, (u,v) are the coordinates of the point cloud in the pixel coordinate system, and d is the depth value of the point cloud in the pixel coordinate system.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-3.

Citation Information

Patent Citations

  • Long-distance target sensing method based on laser radar and camera fusion

    CN113192091A

  • Target detection method based on fusion of vision, lidar, and millimeter wave radar

    US20220277557A1