Multi-modal fusion road edge detection system and method

Abstract

The invention discloses a multi-modal fusion road edge detection system and method, and the system comprises a data collection module, a joint calibration module, a ground point cloud segmentation module, a feature point extraction module, a feature point classification module, a feature point filtering module, a Bayesian ridge regression module, and an iterative algorithm module. Segmenting ground point cloud by using random sampling consistency, extracting feature points based on laser radar scanning characteristics and an isosceles triangle method, identifying and classifying the feature points by means of image lane lines, and filtering out abnormal values by using layered filtering, a random sampling consistency algorithm and iterative Bayesian ridge regression; according to the method, the regression method under the Bayesian framework is adopted, the road boundary fitting robustness can be remarkably improved, and particularly in scenes with strong noise interference and dense abnormal value distribution, the real road edge can be accurately recognized through data uncertainty modeling, and a high-reliability road structure sensing result is provided for an automatic driving vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly to a multi-modal fusion road edge detection system and method. Background Art

[0002] In the development process of autonomous driving vehicles, road edge detection is a key link in environmental perception, and its accuracy and stability play a decisive role in the safe driving and intelligent decision-making of vehicles. Currently, road edge recognition algorithms are mainly divided into two categories: those based on visual images and those based on radar point clouds. For methods based on visual images, such as using deep learning networks to process high-definition images or multi-modal networks (RGB-depth images, RGB-disparity images, etc.), although road edge information can be obtained, they highly depend on information such as image texture and color, and are limited in application in environments with insufficient light (night, forest, etc.). Among radar detection methods, millimeter-wave radar has strong robustness and stability and can penetrate bad weather, but its spatial resolution and measurement range are limited; although lidar has the advantages of high precision, long-distance detection, and 360° scanning characteristics, it cannot identify color information and the point cloud is sparse at long distances, and is vulnerable to noise interference in complex scenes, affecting the detection accuracy and resulting in false detections or missed detections. Therefore, the road edge detection method using a single sensor is difficult to meet the high-precision requirements of autonomous driving, and integrating the advantages of radar and camera has become a key direction for improving detection performance. Summary of the Invention

[0003] To address the above problems, the present invention provides a multi-modal fusion road edge detection system and method, which combines image methods to detect road edges in the lidar coordinate system, effectively solving the problems of low detection accuracy, poor stability, and difficulty in ensuring real-time performance, and significantly improving the environmental perception ability of autonomous driving vehicles.

[0004] To solve the above technical problems, a technical solution adopted by the present invention is:

[0005] A multi-modal fusion road edge detection system, comprising:

[0006] A data acquisition module, configured to obtain the point cloud data of the lidar and the image data of the camera;

[0007] A joint calibration module, configured to perform joint calibration on the camera and the lidar to obtain the conversion relationship between the three-dimensional coordinates of the point cloud and the two-dimensional coordinates of the image;

[0008] A ground point cloud segmentation module, configured to segment the ground point cloud and the non-ground point cloud to reduce interference information; for the lidar point cloud in front of the vehicle, the random sample consensus method is used, and based on the characteristics of the linear change and small undulation of the road surface, large obstacles such as trees, benches, pedestrians, and vehicles are filtered to obtain the ground point cloud and the non-ground point cloud;

[0009] The feature point extraction module, based on the LiDAR scanning characteristics, first uses weighted height and smoothness features to screen candidate points, then uses the isosceles triangle method for secondary screening. It uses the connection pattern between the point cloud and neighboring points on flat roads to accurately extract road edge feature points.

[0010] The feature point classification module uses Ultra-Fast-Lane-Detection-v2 to detect lane lines in images. It converts lane lines from image coordinates to radar coordinates using a coordinate conversion formula. It then uses a random sampling consensus method to remove interference and determine the road segmentation direction, thereby accurately dividing feature points into left-side and right-side road feature points.

[0011] The feature point filtering module first uses layered filtering and a random sampling consensus algorithm for preliminary processing. Layered filtering selects the closest left and right points along the road dividing line. The random sampling consensus algorithm establishes a quadratic polynomial model to remove erroneous points. Based on this, an iterative Bayesian ridge regression process is used, combining the Bayesian theorem and ridge regression ideas, to obtain reliable road boundary points.

[0012] The Bayesian Ridge Regression module is used to single-shot screen candidate points on the left and right boundaries. By introducing the ridge regression concept and parameter regularization terms within the framework of the Bayesian theorem, it improves the accuracy of single-shot screening and avoids overfitting problems caused by excessive model complexity.

[0013] Iterative algorithm module, the iterative algorithm screens the left and right boundary candidate points multiple times in an iterative manner to obtain the true road boundary, thereby achieving the purpose of removing outliers.

[0014] Furthermore, the joint calibration module specifically includes:

[0015] A camera calibration unit, configured to calibrate the intrinsic and extrinsic parameters of the camera and obtain a conversion relationship between a world coordinate system and a pixel coordinate system;

[0016] A laser radar calibration unit, used to perform external parameter calibration on the laser radar and obtain a conversion relationship between the laser radar and the world coordinate system;

[0017] The joint calibration unit is used to jointly establish the conversion relationship between the lidar and camera and the world coordinate system to obtain the direct conversion relationship between point cloud data and image data.

[0018] Furthermore, the ground point cloud segmentation module specifically includes:

[0019] The detection area division unit is used to pre-divide the acquired point cloud into detection areas to reduce the number of point clouds;

[0020] The ground segmentation unit is used to segment the ground in the divided detection area. By using the random sample consensus method, the ground point cloud and the non-ground point cloud are segmented to reduce interference information and improve the subsequent processing efficiency.

[0021] Furthermore, the feature point extraction module specifically includes:

[0022] The height feature extraction module based on weight: Let represent the i-th point cloud in the N-th circle. In the neighborhood of , the height difference between the points between the two side points and is expressed as:

[0023]

[0024] where and are the heights of and respectively, and are the height within the neighborhood of and the average height respectively, Hightand Hight min are the upper and lower thresholds of the height difference respectively, Hight m is the variance threshold of the height;

[0025] The smoothness feature extraction module: When the neighborhood range n of is relatively small, the connection angles between and the left and right side points are approximately 180°. The smoothness feature is expressed as:

[0026]

[0027] where v l and v r are the direction vectors of with the leftmost point and the rightmost point in the XY axis within the neighborhood respectively, is the angle threshold parameter;

[0028] The isosceles triangle feature extraction module: It is expressed as follows:

[0029] d l -d r ≤D m

[0030] where d l and d r are the connection distances between and respectively, D m ​is the distance threshold parameter.

[0031] A multi-modal fusion road edge detection method, using a multi-modal fusion road edge detection system as described above, includes the following steps:

[0032] Step S1: The lidar collects the point cloud data of the vehicle's surrounding environment, and the camera collects the image data of the vehicle's surrounding. The two modalities of data collected are fused, and the image pixel data is converted to the lidar point cloud three-dimensional coordinate system through joint calibration to obtain the point cloud projection in the lidar coordinate system.

[0033] Step S2: Assume the road surface as a plane model, divide the road detection area in advance, and use the RANSAC method for iterative operation to separate the ground and non-ground point clouds.

[0034] Step S3: Extract feature points from the segmented ground point cloud. First, screen candidate points based on specific features, and then use the isosceles triangle method for secondary screening.

[0035] Step S4: Utilize the result of image lane line detection, convert the lane segmentation line to point cloud through coordinate transformation, and classify the feature points as left and right road edge feature points through the transformed point cloud.

[0036] Step S5: Use the methods of hierarchical filtering, random sample consensus algorithm, and iterative Bayesian ridge regression to work together to remove outliers in the left and right road boundary feature points and complete the extraction of road boundary point cloud.

[0037] Further, the step S1 is specifically as follows:

[0038] Step S11: Obtain the timestamp data of the lidar and the camera, match the timestamps of the lidar and the camera through the time axis, find their relative positions on the time axis, and complete time synchronization through the truncation method.

[0039] Step S12: Select a calibration board with appropriate size and known parameters, place it in the overlapping field of view of the camera and the lidar, and extract the feature points of the point cloud and the image.

[0040] Step S13: Obtain the conversion equation between the image two-dimensional coordinate system and the world coordinate system through the feature point coordinates extracted from the image and the known calibration board parameters, as follows:

[0041]

[0042] Among them, (u, v) are pixel coordinates, and (X, Y, Z) are world coordinates. Since the world coordinate system is established on the calibration board and the Z-axis is perpendicular to the calibration board, so Z = 0, A is the camera internal parameter matrix, [R t] is the camera external parameter matrix, specifically, R is a 3×3 rotation matrix, (r1, r2, r3) are three rotation vectors in the rotation matrix, t is a 3×1 translation vector, and s is a scale factor;

[0043] Step S14: Obtain the conversion equation between the point cloud three-dimensional coordinate system and the world coordinate system from the feature point coordinates extracted from the point cloud and the known calibration board parameters, as follows:

[0044]

[0045] Among them, (X r , Y r , Z r ) are the point cloud coordinates in the point cloud three-dimensional coordinate system, (X, Y, Z) are the world coordinates, R LC is a 3×3 rotation matrix, t LC is a 3×1 translation vector; obtain the objective function:

[0046]

[0047] Step S15: According to the conversion relationship between the camera and the world coordinate system and the conversion relationship between the lidar and the world coordinate system, eliminate the world coordinate system to complete the data fusion between the lidar point cloud and the image.

[0048] Furthermore, the specific content of step S3 is as follows:

[0049] Step S31: For the point cloud data obtained by the lidar, traverse according to the scan line information; for each scan line, calculate the distance between adjacent points;

[0050] Step S32: For each pair of adjacent points, calculate their height difference; if the height difference exceeds the set threshold, mark it as an abnormal point, and for adjacent points within the threshold range, calculate the height eigenvalue based on weight according to the weight corresponding to the scan line where they are located;

[0051] Step S33: For each point cloud point, determine a neighborhood range; the neighborhood range is set according to the accuracy of the lidar and the complexity of the actual road scene, calculate the connection vector between this point and each point within the neighborhood; through vector operations, calculate the angle between adjacent connection vectors;

[0052] Step S34: If the change in the angle is small, it indicates that the point cloud around this point is relatively smooth, conforming to the characteristics of the road edge point cloud, and this point can be regarded as a point that meets the smoothness characteristics; conversely, if the angle change is large, this point is in an irregular area and is considered not to meet the smoothness characteristics, and it can be excluded from subsequent considerations;

[0053] Step S35: For the selected candidate points, the isosceles triangle method is used for further precise screening; for each candidate point, a certain number of its neighborhood points are selected to construct an isosceles triangle, and the distances and angles between the candidate point and different neighborhood points are calculated;

[0054] Step S36: Set an error range; calculate the length difference between the two sides of the constructed triangle. If the length difference is within the error range and the interior angle of the triangle is close to a specific angle, it is considered that the triangle meets the isosceles triangle characteristics; if not, the candidate point corresponding to the triangle can be regarded as noise or other interference points.

[0055] Further, the specific steps of Step S5 are as follows:

[0056] Step S51: The Bayesian regression model uses the Gaussian distribution as the initial distribution, and takes the road boundary point cloud dataset P temp ={(x i ,y i ), i = 1, 2, …, n} as the input quantity, where P temp ={(x i ,y i ), i = 1, 2, …, n} are the coordinates of the point cloud; assuming that the coordinates of the road boundary are predicted through the regression model, then there is a corresponding relationship:

[0057]

[0058] where ε i is Gaussian noise with zero mean and variance σ 2 , and ε i ~(0, σ 2 );

[0059] Step S52: In Bayesian ridge regression, the likelihood function represents the probability of observing the data y given the regression coefficient β, which is expressed as:

[0060]

[0061] where n is the number of point clouds. Combining the prior distribution and the likelihood function, Bayes' theorem is used to infer the posterior distribution of the regression coefficient;

[0062] Step S53. The posterior distribution \(p(\beta|y, X)\propto p(y|X, \beta)\cdot p(\beta)\). The posterior mean \(\mu\) and the posterior covariance matrix \(\Gamma\) are as follows:

[0063] \(\mu=(X T X + \lambda I) -1 X T y

[0064] \(\Gamma=\sigma 2 (X T X + \lambda I) -1

[0065] where \(\lambda\) is the regularization parameter. By solving, the optimal regression coefficient \(\beta\) is obtained, that is, the optimal fitting line of the road boundary.

[0066] Step S54. The iterative algorithm screens the left and right boundary candidate points \(P train,l and \(P train,r \) multiple times through an iterative method to obtain the true road boundaries \(P boundary,l and \(P boundary,r \), so as to achieve the purpose of removing outliers.

[0067] Furthermore, in Step S54, the iterative algorithm is specifically as follows:

[0068] Step1. First, use the left and right boundary point cloud data \(T = \{P train,l , P train,r \}\) as the training set and add it to the Bayesian ridge regression model for training. Calculate parameters such as the posterior mean \(\mu\) and the covariance matrix \(\Gamma\) to obtain the estimate of the model parameter \(\beta\).

[0069] Step2. According to the corresponding \(\{x train,l , x train,r \}\) in the training set \(T\) and the parameters of the Bayesian ridge regression model, traverse each point in order and calculate the predicted value Combined with the actual value \(y i The error RMSE can be obtained:

[0070]

[0071] Step3. Compare the error RMSE with a preset threshold; if RMSE is less than the threshold, then this point is considered a correct road boundary point, and continue to traverse to other points; if the RMSE of a certain point is greater than the threshold, then this point is considered an incorrect point, stop traversing, remove this point from the training set to obtain a new training set \(T'\), and repeat Step1, Step2, Step3;

[0072] Step4. When the training set after the \(n\)th iteration of the training set \(T nIf all the data in [[]] is traversed and there is no point cloud considered as an error point, then the training set T n is used as the correct road boundary points {P boundary,l , P boundary,r} and output.

[0073] The beneficial effects of the present invention are as follows:

[0074] The present invention adopts a regression method under the Bayesian framework, which can significantly improve the robustness of road boundary fitting. Especially in scenarios with strong noise interference and dense outliers, it can accurately identify the real road edge by modeling the data uncertainty, providing a highly reliable road structure perception result for autonomous driving vehicles. Description of the Drawings

[0075] Figure 1 is the flowchart of the multi-modal fusion road edge detection system of the present invention;

[0076] Figure 2 is the schematic diagram of lidar detection for multi-modal fusion road edge detection of the present invention. Detailed Embodiments

[0077] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0078] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

[0079] The present invention discloses a multi-modal fusion road edge detection system, including:

[0080] A data acquisition module for acquiring the point cloud data of the lidar and the image data of the camera;

[0081] A joint calibration module for jointly calibrating the camera and the lidar to obtain the conversion relationship between the three-dimensional coordinates of the point cloud and the two-dimensional coordinates of the image;

[0082] A ground point cloud segmentation module for segmenting the ground point cloud and the non-ground point cloud to reduce interference information. For the lidar point cloud in front of the vehicle, the random sample consensus method is used. Based on the characteristics of the linear change and small undulation of the road surface, large obstacles such as trees, benches, pedestrians, and vehicles are filtered to obtain the ground point cloud and the non-ground point cloud;

[0083] The feature point extraction module first uses weighted height and smoothness features to screen candidate points based on the LiDAR scanning characteristics. The weighted height feature considers the difference in the distance between adjacent points caused by changes in the scan line length and the change in the road height difference threshold. The smoothness feature is determined by the angle of the line connecting the neighboring points. The isosceles triangle method is then used for secondary screening. The point cloud on a flat road is connected to the pattern of the lines connecting the neighboring points to accurately extract the road edge feature points, reducing the amount of calculation and meeting the real-time requirements.

[0084] The feature point classification module uses Ultra-Fast-Lane-Detection-v2 to detect lane lines in images. It converts lane lines from image coordinates to radar coordinates using a coordinate conversion formula. It then uses a random sampling consensus method to remove interference and determine the road segmentation direction. This accurately divides feature points into left-side and right-side road feature points, effectively addressing the challenge of road boundary classification in complex vehicle conditions.

[0085] The feature point filtering module first uses layered filtering and a random sampling consensus algorithm for preliminary processing. Layered filtering selects the closest left and right points along the road dividing line. The random sampling consensus algorithm establishes a quadratic polynomial model to remove erroneous points. Based on this, an iterative Bayesian ridge regression process, combining Bayesian theorem and ridge regression ideas, adaptively adjusts parameters based on the complex shape of road boundaries, avoiding overfitting, accurately removing outliers, and obtaining reliable road boundary points.

[0086] The Bayesian Ridge Regression module is used to single-shot screen candidate points on the left and right boundaries. By introducing the ridge regression concept and parameter regularization terms within the framework of the Bayesian theorem, it improves the accuracy of single-shot screening and avoids overfitting problems caused by excessive model complexity.

[0087] Iterative algorithm module, the iterative algorithm screens the left and right boundary candidate points multiple times in an iterative manner to obtain the true road boundary, thereby achieving the purpose of removing outliers.

[0088] As an embodiment, the joint calibration module of the present invention specifically includes:

[0089] The camera calibration unit is used to calibrate the internal and external parameters of the camera and obtain the conversion relationship between the world coordinate system and the pixel coordinate system;

[0090] The laser radar calibration unit is used to calibrate the external parameters of the laser radar and obtain the conversion relationship between the laser radar and the world coordinate system;

[0091] The joint calibration unit is used to jointly establish the conversion relationship between the lidar and camera and the world coordinate system to obtain the direct conversion relationship between point cloud data and image data.

[0092] As an implementation method, the ground point cloud segmentation module specifically includes:

[0093] A detection area division unit, which is used to perform preliminary detection area division on the acquired point cloud to reduce the number of point clouds;

[0094] A ground segmentation unit, which is used to perform ground segmentation on the divided detection area. By using the random sample consensus method, the ground point cloud and the non-ground point cloud are segmented to reduce interference information and improve the subsequent processing efficiency.

[0095] As an implementation manner, the feature point extraction module specifically includes:

[0096] A height feature extraction module based on weight: Based on the height feature with weight, it is assumed that the height difference on the road surface should be less than a certain threshold, otherwise it will be recognized as a road edge point; at the same time, due to the different distances from the lidar, the height threshold will also change; use to represent the i-th point cloud in the N-th circle, which means that in the neighborhood of , the height difference between the two side points and can be expressed as:

[0097]

[0098] In the above formula and are respectively and 's height, and are respectively the height and the average height within the neighborhood of , Hight max and Hight min are respectively the upper and lower threshold values of the height difference, Hight m is the height variance threshold;

[0099] Smoothness feature extraction module: When the neighborhood range n of is relatively small, the connection angles between and the points on the left and right sides should be approximately 180°, as shown in the (N + 1)-th circle scan line in Figure 2 . Therefore, the smoothness feature can be expressed as:

[0100]

[0101] where v l and v r are 's direction vectors on the XY axis with the leftmost point and the rightmost point in the neighborhood respectively, is the angle threshold parameter;

[0102] Isosceles triangle feature extraction module: On a flat road, the lines connecting a certain point cloud on the same scan line to the two point clouds on the far left and far right within the neighborhood should form an isosceles triangle. However, when encountering a convex boundary, the lengths of the two sides of the triangle will be different, as shown in Figure 2 shown below. Therefore, it can be expressed as follows:

[0103] d l -d r ≤D m

[0104] where d l and d r are respectively and the connecting line distances between two points, and D m is the distance threshold parameter.

[0105] The present invention discloses a multi-modal fusion road edge detection method, which uses the above-mentioned multi-modal fusion road edge detection system, as shown in Figure 1 shown below, and includes:

[0106] Step S1: The lidar collects the point cloud data of the vehicle surrounding environment, and the camera collects the image data of the vehicle surrounding. The two types of modal data collected are fused, and through joint calibration, the image pixel data is converted into the lidar point cloud three-dimensional coordinate system to obtain the point cloud projection in the lidar coordinate system;

[0107] Step S2: Assume the road surface as a plane model, divide the road detection area in advance, and use the RANSAC method for iterative operation to separate the ground and non-ground point clouds;

[0108] Step S3: Extract feature points on the segmented ground point cloud. First, screen candidate points based on specific features, and then use the isosceles triangle method for secondary screening;

[0109] Step S4: Utilize the result of image lane line detection, convert the lane dividing line into point cloud through coordinate transformation, and classify the feature points into left and right road edge feature points through the converted point cloud;

[0110] Step S5: Use the methods of hierarchical filtering, random sample consensus algorithm, and iterative Bayesian ridge regression to work together to remove the outliers in the left and right road boundary feature points and complete the extraction of road boundary point cloud.

[0111] The following elaborates on the steps in detail.

[0112] Step S1: The lidar collects the point cloud data of the vehicle's surrounding environment, and the camera collects the image data of the vehicle's surrounding. The two types of modality data collected are fused, and through joint calibration, the image pixel data is converted into the lidar point cloud three-dimensional coordinate system to obtain the point cloud projection in the lidar coordinate system.

[0113] Step S11: Obtain the timestamp data of the lidar and the camera, match the timestamps of the lidar and the camera through the time axis, find their relative positions on the time axis, and complete time synchronization through the truncation method.

[0114] Step S12: Select a calibration board with appropriate size and known parameters, place it in the overlapping field of view of the camera and the lidar, and extract the feature points of the point cloud and the image.

[0115] Step S13: From the feature point coordinates extracted from the image and the known calibration board parameters, the conversion equation between the image two-dimensional coordinate system and the world coordinate system can be obtained. The equation is as follows:

[0116]

[0117] Where, (u, v) are pixel coordinates, (X, Y, Z) are world coordinates. Since the world coordinate system is established on the calibration board and the Z-axis is perpendicular to the calibration board, so Z = 0, A is the camera internal parameter matrix, [R t] is the camera external parameter matrix, specifically R is a 3×3 rotation matrix, (r1, r2, r3) are the three rotation vectors in the rotation matrix, t is a 3×1 translation vector, and s is the scale factor.

[0118] Step S14: From the feature point coordinates extracted from the point cloud and the known calibration board parameters, the conversion equation between the point cloud three-dimensional coordinate system and the world coordinate system can be obtained. The equation is as follows:

[0119]

[0120] Where (X r , Y r , Z r ) are the point cloud coordinates in the point cloud three-dimensional coordinate system, (X, Y, Z) are world coordinates, R LC is a 3×3 rotation matrix, t LC is a 3×1 translation vector; the objective function is obtained:

[0121]

[0122] Step S15: According to the conversion relationship between the camera and the world coordinate system and the conversion relationship between the lidar and the world coordinate system, eliminate the intermediate variable, that is, the world coordinate system, and the data fusion between the lidar point cloud and the image can be completed.

[0123] Step S3: Extract feature points from the segmented ground point cloud. First, screen candidate points based on specific features, and then use the isosceles triangle method for secondary screening;

[0124] Step S31: For the point cloud data obtained by the lidar, traverse according to the scan line information; for each scan line, calculate the distance between adjacent points;

[0125] Step S32: For each pair of adjacent points, calculate their height difference; if the height difference exceeds the set threshold, it will be marked as an abnormal point, while for adjacent points within the threshold range, calculate the height feature value based on the weight corresponding to the scan line where they are located;

[0126] Step S33: For each point cloud point, determine a neighborhood range; the neighborhood range is set according to the accuracy of the lidar and the complexity of the actual road scene, calculate the connection vectors between this point and each point within the neighborhood; through vector operations, calculate the angle between adjacent connection vectors;

[0127] Step S34: If the change in the angle is small, it means that the point cloud around this point is relatively smooth and conforms to the characteristics of the road edge point cloud, and this point can be used as a point that meets the smoothness feature; on the contrary, if the change in the angle is large, it means that this point is in an irregular area and is considered not to meet the smoothness feature, and it can be excluded from subsequent considerations;

[0128] Step S35: For the selected candidate points, use the isosceles triangle method for further precise screening; for each candidate point, select a certain number of its neighborhood points to construct an isosceles triangle; specifically, calculate the distances and angles between the candidate point and different neighborhood points;

[0129] Step S36: Check whether the formed triangle meets the approximate conditions of an isosceles triangle; set a reasonable error range. For example, for the difference in the lengths of the two sides of the triangle, if the difference is within the set error range and the interior angle of the triangle is close to a specific angle (such as between 60° and 120°, adjusted according to the actual situation of the road), then it is considered that this candidate point meets the isosceles triangle feature, indicating that this is a road edge point; if not, this candidate point can be regarded as noise or other interference points.

[0130] Step S5: Use the methods of hierarchical filtering, random sample consensus algorithm, and iterative Bayesian ridge regression to work together to remove outliers in the left and right road boundary feature points and complete the extraction of road boundary point cloud;

[0131] Step S51: The Bayesian regression model uses the Gaussian distribution as the initial distribution, and the road boundary point cloud data set P temp ={(x i ,y i), i = 1, 2, ..., n} as the input quantity, P temp = {(x i , y i ), i = 1, 2, …, n} are the coordinates of the point cloud; assuming that the coordinates of the road boundary are predicted through a regression model, there is a corresponding relationship:

[0132]

[0133] where ε i is Gaussian noise with zero mean and variance σ 2 , ε i ~ (0, σ 2 );

[0134] Step S52: In Bayesian ridge regression, the likelihood function represents the probability of observing the data y given the regression coefficient β; for a linear regression model, the likelihood function can be expressed as:

[0135]

[0136] where n is the number of point clouds; combining the prior distribution and the likelihood function, Bayes' theorem can be used to infer the posterior distribution of the regression coefficient;

[0137] Step S53: The posterior distribution p(β|y, X) ∝ p(y|X, β)·p(β), and the posterior mean μ and posterior covariance matrix Γ are as follows respectively:

[0138] μ = (X T X + λI) -1 X T y

[0139] Γ = σ 2 (X T X + λI) -1

[0140] where λ is the regularization parameter, which controls the complexity of the model; by solving these parameters, the optimal regression coefficient β, that is, the optimal fitting line of the road boundary, can be obtained;

[0141] Step S54: The iterative algorithm screens the left and right boundary candidate points P train,l and P train,r multiple times in an iterative manner to obtain the true road boundaries P boundary,l and P boundary,r , so as to achieve the purpose of removing outliers;

[0142] The running process of the iterative algorithm mainly includes three parts: training, prediction, and testing, which are specifically as follows:

[0143] Step1. First, use the left and right boundary point cloud data T = {P train,l , P train,r} as the training set and add it to the Bayesian ridge regression model for training. Calculate parameters such as the posterior mean μ and covariance matrix Γ to obtain an estimate of the model parameter β;

[0144] Step2. According to the corresponding {x train,l , x train,r} in the training set T and the parameters of the Bayesian ridge regression model, traverse each point in order and calculate the predicted value Combined with the actual value y i The root mean square error RMSE can be obtained:

[0145]

[0146] Step3. Compare the error RMSE with a preset threshold; if the RMSE of a certain point is less than the threshold, then consider this point as a correct road boundary point and continue to traverse to other points; if the RMSE of a certain point is greater than the threshold, then consider this point as an incorrect point, stop traversing, remove this point from the training set to obtain a new training set T', and repeat Step1, Step2, and Step3;

[0147] Step4. When all the data in the training set T n after the nth iteration have been traversed and there are no point clouds considered as incorrect points, then use the training set T n as the correct road boundary point {P boundary,l , P boundary,r} and output it.

[0148] According to the multi-modal fusion road edge detection system and method of this embodiment, the recognition of real road edges in scenarios with strong noise interference and dense outlier distributions can be achieved.

Claims

1. A multi-modal fusion road edge detection system, characterized in that, include: Data acquisition module, used to obtain point cloud data from the lidar and image data from the camera; The joint calibration module is used to jointly calibrate the camera and lidar to obtain the conversion relationship between the 3D coordinates of the point cloud and the 2D coordinates of the image; The ground point cloud segmentation module is used to separate the ground point cloud from the non-ground point cloud to reduce interference information. For the LiDAR point cloud in front of the vehicle, the module uses a random sampling consistency method to filter out large obstacles such as trees, benches, pedestrians, and vehicles based on the linear changes and small undulations of the road surface to obtain the ground point cloud and non-ground point cloud. The feature point extraction module, based on the LiDAR scanning characteristics, first uses weighted height and smoothness features to screen candidate points, then uses the isosceles triangle method for secondary screening. It uses the connection pattern between the point cloud and neighboring points on flat roads to accurately extract road edge feature points. The feature point classification module uses Ultra-Fast-Lane-Detection-v2 to detect lane lines in images. It converts lane lines from image coordinates to radar coordinates using a coordinate conversion formula. It then uses a random sampling consensus method to remove interference and determine the road segmentation direction, thereby accurately dividing feature points into left-side and right-side road feature points. The feature point filtering module first uses layered filtering and a random sampling consensus algorithm for preliminary processing. Layered filtering selects the closest left and right points along the road dividing line. The random sampling consensus algorithm establishes a quadratic polynomial model to remove erroneous points. Based on this, an iterative Bayesian ridge regression process is used, combining the Bayesian theorem and ridge regression ideas, to obtain reliable road boundary points. The Bayesian Ridge Regression module is used to single-shot screen candidate points on the left and right boundaries. By introducing the ridge regression concept and parameter regularization terms within the framework of the Bayesian theorem, it improves the accuracy of single-shot screening and avoids overfitting problems caused by excessive model complexity. Iterative algorithm module, the iterative algorithm screens the left and right boundary candidate points multiple times in an iterative manner to obtain the true road boundary, thereby achieving the purpose of removing outliers.

2. The multimodal fusion road edge detection system according to claim 1, characterized in that: The joint calibration module specifically includes: A camera calibration unit, configured to calibrate the intrinsic and extrinsic parameters of the camera and obtain a conversion relationship between a world coordinate system and a pixel coordinate system; A laser radar calibration unit, used to perform external parameter calibration on the laser radar and obtain a conversion relationship between the laser radar and the world coordinate system; The joint calibration unit is used to jointly establish the conversion relationship between the lidar and camera and the world coordinate system to obtain the direct conversion relationship between point cloud data and image data.

3. The multimodal fusion road edge detection system according to claim 1, wherein: The ground point cloud segmentation module specifically includes: The detection area division unit is used to pre-divide the acquired point cloud into detection areas to reduce the number of point clouds; The ground segmentation unit is used to perform ground segmentation on the divided detection area. It uses the random sampling consistency method to segment the ground point cloud and the non-ground point cloud, reduce interference information, and improve subsequent processing efficiency.

4. A multimodal fusion road edge detection system according to claim 1, characterized in that: The feature point extraction module specifically includes: Weight-based height feature extraction module: Use to represent the i-th point cloud in the N-th circle. In the neighborhood of , the height difference between the points on both sides of the point and is expressed as: It is expressed as: Among them, and are respectively and the heights of, and are respectively the height within the neighborhood of and the average height of, Hight max and Hight min are respectively the upper and lower thresholds of the height difference, Hight m is the threshold of the height variance; Smoothness feature extraction module: When the neighborhood range n of is relatively small, the connection angles with the points on the left and right sides are approximately 180°, and the smoothness feature is expressed as: where v l and v r are the direction vectors on the XY axis of the leftmost point and the rightmost point in the domain respectively, is the angle threshold parameter; Isosceles triangle feature extraction module: expressed as follows: d l -d r ≤D m where d l and d r are respectively the connecting line distances between two points, D m is the distance threshold parameter.

5. A multi-modal fusion road edge detection method, which uses a multi-modal fusion road edge detection system according to any one of claims 1 to 4, characterized in that, The steps include: Step S1: The lidar collects the point cloud data of the vehicle's surrounding environment, and the camera collects the image data of the vehicle's surrounding. The two types of modality data collected are fused, and through joint calibration, the image pixel data is converted into the lidar point cloud three-dimensional coordinate system to obtain the point cloud projection in the lidar coordinate system. Step S2: Assume the road surface as a plane model, divide the road detection area in advance, and use the RANSAC method for iterative operation to separate the ground and non-ground point clouds. Step S3: Extract feature points from the segmented ground point cloud. First, select candidate points based on specific features, and then use the isosceles triangle method for secondary screening. Step S4: Utilize the result of image lane line detection, convert the lane segmentation line into point cloud through coordinate transformation, and classify the feature points as left and right road edge feature points through the transformed point cloud. Step S5: Use the methods of hierarchical filtering, random sample consensus algorithm, and iterative Bayesian ridge regression to work together to remove the outliers in the left and right road boundary feature points and complete the extraction of road boundary point cloud.

6. A multimodal fusion road edge detection method according to claim 5, characterized in that: The specific content of step S1 is as follows: Step S11: Obtain the timestamp data of the lidar and the camera, match the timestamps of the lidar and the camera through the time axis, find their relative positions on the time axis, and complete time synchronization through the truncation method. Step S12: Select a calibration board with appropriate size and known parameters, place it in the overlapping field of view of the camera and the lidar, and extract the feature points of the point cloud and the image. Step S13: Obtain the conversion equation between the image two-dimensional coordinate system and the world coordinate system through the feature point coordinates extracted from the image and the known calibration board parameters, as follows: where (u, v) are pixel coordinates, (X, Y, Z) are world coordinates. Since the world coordinate system is established on the calibration board and the Z-axis is perpendicular to the calibration board, so Z = 0, A is the camera internal parameter matrix, [R t] is the camera external parameter matrix, specifically R is a 3×3 rotation matrix, (r1, r2, r3) are the three rotation vectors in the rotation matrix, t is a 3×1 translation vector, and s is the scale factor. Step S14: Obtain the conversion equation between the point cloud three-dimensional coordinate system and the world coordinate system through the feature point coordinates extracted from the point cloud and the known calibration board parameters, as follows: where (X r , Y r , Z r ) are the point cloud coordinates in the three-dimensional coordinate system of the point cloud, (X, Y, Z) are the world coordinates, R LC is a 3×3 rotation matrix, and t LC is a 3×1 translation vector; the objective function is obtained as follows: Step S15: According to the conversion relationship between the camera and the world coordinate system and the conversion relationship between the lidar and the world coordinate system, eliminate the world coordinate system to complete the data fusion between the lidar point cloud and the image.

7. A multimodal fusion road edge detection method according to claim 5, characterized in that: The specific content of step S3 is as follows: Step S31: For the point cloud data obtained by the lidar, traverse according to the scan line information; for each scan line, calculate the distance between adjacent points. Step S32: For each pair of adjacent points, calculate their height difference; if the height difference exceeds the set threshold, mark it as an outlier, and for adjacent points within the threshold range, calculate the weight-based height eigenvalue according to the weight corresponding to the scan line where they are located. Step S33: For each point cloud point, determine a neighborhood range. The neighborhood range is set according to the accuracy of the lidar and the complexity of the actual road scene. Calculate the connection vectors between this point and each point within the neighborhood. Through vector operations, find the angle between adjacent connection vectors. Step S34: If the change in the angle is small, it indicates that the point cloud around this point is relatively smooth and conforms to the characteristics of the road edge point cloud. This point can be regarded as a point that meets the smoothness feature. Conversely, if the change in the angle is large, this point is in an irregular area and is considered not to meet the smoothness feature, and it can be excluded from subsequent considerations. Step S35: For the selected candidate points, use the isosceles triangle method for further precise screening. For each candidate point, select a certain number of its neighborhood points to construct an isosceles triangle, and calculate the distances and angles between the candidate point and different neighborhood points. Step S36: Set an error range. Calculate the length difference between the two sides of the constructed triangle. If the length difference is within the error range and the interior angle of the triangle is close to a specific angle, it is considered that the triangle meets the isosceles triangle feature. If not, the candidate point corresponding to the triangle can be regarded as noise or other interference points.

8. A multimodal fusion road edge detection method according to claim 5, characterized in that: The specific content of step S5 is as follows: Step S51: The Bayesian regression model uses the Gaussian distribution as the initial distribution, and takes the road boundary point cloud data set P temp ={(x i ,y i ), i = 1, 2, …, n} as the input quantity, where P temp ={(x i ,y i ), i = 1, 2, …, n} are the coordinates of the point cloud; assuming that the coordinates of the road boundary are predicted through the regression model, there is a corresponding relationship: where, ε i is Gaussian noise with zero mean and variance σ 2 , and ε i ~(0, σ 2 ); Step S52: In Bayesian ridge regression, the likelihood function represents the probability of observing data y given the regression coefficient β, which is expressed as: where n is the number of point clouds. Combining the prior distribution and the likelihood function, Bayes' theorem is used to infer the posterior distribution of the regression coefficient. Step S53: The posterior distribution p(β|y,X) ∝ p(y|X,β)·p(β), and the posterior mean μ and the posterior covariance matrix Γ are as follows: μ=(X T X + λI) -1 X T y Γ = σ 2 (X T X + λI) -1 where λ is the regularization parameter. By solving, the optimal regression coefficient β, that is, the optimal fitting line of the road boundary, is obtained. Step S54: The iterative algorithm repeatedly screens the left and right boundary candidate points P train,l and P train,r to obtain the true road boundaries P boundary,l and P boundary,r , thereby achieving the purpose of removing outliers.

9. A multi-modal fusion road edge detection method according to claim 8, characterized in that: In step S54, the specific iterative algorithm is as follows: Step1. First, use the left and right boundary point cloud data T = {P train,l , P train,r} as the training set and add it to the Bayesian ridge regression model for training. Calculate parameters such as the posterior mean μ and covariance matrix Γ to obtain an estimate of the model parameter β; Step2. According to the corresponding {x train,l , x train,r} in T of the training set and the parameters of the Bayesian ridge regression model, traverse each point in order and calculate the predicted value Combined with the actual value y i The error RMSE can be obtained: Step3. Compare the error RMSE with a preset threshold. If RMSE is less than the threshold, it is considered that this point is a correct road boundary point, and continue to traverse to other points. If the RMSE of a certain point is greater than the threshold, it is considered that this point is an incorrect point, stop traversing, remove this point from the training set to obtain a new training set T', and repeat Step1, Step2, Step3. Step4. When all the data in the training set T after the nth iteration of the training set have been traversed and there is no point cloud considered as an error point, then the training set T n is used as the correct road boundary points {P n , P boundary,l , boundary,r} and output.

Images