Lane line extraction-based laser radar visual combined calibration method

Abstract

The invention discloses a laser radar visual joint calibration method based on lane line extraction, and the method comprises the steps: S1, obtaining environment point cloud data from a laser radar sensor, carrying out the plane fitting through an RANSAC algorithm, carrying out the lane line point cloud extraction, and carrying out the clustering of the extracted lane line point cloud, and forming a line feature point cloud; s2, acquiring environment image data from a camera, performing image semantic segmentation on the environment image data, extracting a pixel region of a lane line in the environment image data by using a DeepLab-V3 network, and then performing feature mask binarization processing; s3, finding out line features in the environment point cloud data and the environment image data, then carrying out consistency matching, solving a three-line perspective problem, and obtaining preliminary external parameter estimation by establishing a corresponding relation of three line features; and S4, performing refined calibration on the preliminary matching result by using an optimization algorithm, and outputting external parameter calibration results of the laser radar and the camera through multiple measurements and data fusion, including a rotation matrix and a translation vector.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sensors, and in particular to a laser radar vision joint calibration method based on lane line extraction. Background Art

[0002] In recent years, intelligence has become a development trend in various industries, and the data collected by a single sensor can no longer accurately describe environmental data information. The development of multimodal sensor fusion technology makes up for the shortcomings of a single sensor. It can achieve data complementarity between different sensors and is widely used in remote sensing, robotics, autonomous driving, smart cities, three-dimensional space information reconstruction and other fields. For the above fields, accurate calibration of lidar and camera is a prerequisite for building a multi-sensor perception system.

[0003] Autonomous vehicles and intelligent transportation systems require multi-sensor perception systems for environmental perception, decision-making, and control. LiDAR and cameras are the most commonly used sensors in autonomous driving systems, and the distance and visual information they provide are crucial for environmental modeling. However, existing sensor calibration methods often require the use of complex calibration plates or initial calibration in a controlled environment, which increases the complexity and cost of calibration and limits the practical application of large-scale autonomous vehicles and scenarios that are difficult to calibrate again, such as smart cities and smart intersections. Therefore, a simple, efficient, and accurate calibration method is urgently needed.

[0004] The above calibration is mainly used for radar camera external parameter calibration, which mainly describes the posture relationship between two sensors. In the field of sensor external parameter calibration, there are three key technical points: feature extraction, coordinate transformation and optimization solution, among which:

[0005] Feature extraction: Extract feature points on the calibration plate from the camera image and lidar point cloud and fit the two-dimensional plane. For the checkerboard calibration plate, the findChessboardCorners function in OpenCV can generally be used to extract image corners, and PCL can be used to fit the three-dimensional plane in the point cloud. For non-chessboard calibration plates, morphological processing and Hough transform methods can be used to extract feature points.

[0006] Coordinate transformation: Establish the transformation relationship between the camera coordinate system, the lidar coordinate system, and the world coordinate system. For a simple example, assume that the transformation matrix from the camera coordinate system to the world coordinate system is T iw , the transformation matrix T from the lidar coordinate system to the world coordinate system pw , then the transformation matrix from camera to lidar is

[0007] Optimization solution: Use the feature points with the same name on the calibration plate to establish an error function, and solve the external parameters by minimizing the error function. The error function can be expressed as the reprojection error of the feature points in the two coordinate systems. For a simple example, for a pair of 2D-3D points (p x , P x ), the error function can be expressed as e x =||p x -pro j(T pi P x )||where pro j(·) represents the projection of a 3D point onto a 2D image plane by minimizing the reprojection error Σ of all feature points x e x , the transformation matrix T can be optimized and solved pi .

[0008] At present, the more mature laser radar and camera calibration methods include calibration based on calibration plate and motion based calibration. Among them:

[0009] Calibration based on calibration plate means manually arranging calibration plate in calibration scene, extracting calibration plate features and building models to complete external parameter calibration of LiDAR and camera. Commonly used two-dimensional calibration plates include checkerboard calibration plate and non-checkerboard calibration plate, and their principles and technical solutions are similar. This calibration method combines multiple features such as points, lines, and surfaces, and comprehensively utilizes two-dimensional and three-dimensional information for external parameter calibration. This method can not only use feature points in two-dimensional images, but also combine plane features in three-dimensional point clouds. The checkerboard calibration plate used in this method is a rectangular plate printed with black and white checkerboards. The camera extracts the corner points in the checkerboard plane and matches them with the three-dimensional plane fitted by the four outer edges of the calibration plate extracted from the LiDAR point cloud, so as to obtain high-precision 2D-3D point feature pairs. The advantage of this method is that its calculation method is mature, the feature point extraction accuracy is high, and the camera intrinsic parameters and the external parameters of the camera and LiDAR can be calibrated at the same time. For example, Zhang Q L. completed the extrinsic calibration of a camera and laser range finder (improves camera calibration) based on a checkerboard and using point-on-surface geometric constraints in his paper (from the International Conference on Intelligent Robots and Systems (IROS)). The calibration method based on the calibration plate has the advantages of high calibration result accuracy, mature and stable calculation method, and the ability to easily obtain the same-name features of the laser radar and camera. Since the placement of the calibration plate in the natural environment is usually manually controlled, the high reliability of the calibration process can be ensured. However, it also has the following shortcomings:

[0010] 1) Strong dependence on the environment: The calibration plate-based method needs to be performed in a static environment, which requires the calibration plate to be stably placed in the scene. If the environmental conditions are unstable or there are large dynamic changes, such as when the position of the sensors changes due to external factors such as jitter, recalibration is required, which increases the complexity of the calibration work. This also means that in some dynamic scenes or when the sensor moves frequently, the applicability of the calibration plate calibration method is limited.

[0011] 2) Manual intervention is required: During the calibration process, it is usually necessary to manually place the calibration plate, crop the feature point cloud, mark the features with the same name, etc., which increases the complexity and instability of the calibration work. Manual operation is not only time-consuming and labor-intensive, but may also introduce human errors, affecting the accuracy of the calibration results.

[0012] 3) Difficulty in repeated calibration: In practical applications, the use environment and installation method of sensors may change, so frequent recalibration is required. For example, in an autonomous vehicle, a slight change in the sensor installation position will affect the calibration result, which requires recalibration after each sensor installation or adjustment. This repeated calibration requirement not only increases the workload, but may also affect the real-time performance and efficiency of the system.

[0013] 4) Strong dependence on calibration plate: The calibration plate-based method relies heavily on the accuracy of the calibration plate and the quality of feature point extraction. If there are manufacturing errors in the calibration plate itself or the feature point extraction during the calibration process is inaccurate, the final calibration result will be affected. In addition, the size and type of the calibration plate also limit its application scope. For example, a large-sized calibration plate is difficult to use in a small space, while a small-sized calibration plate does not work well in a large-scale calibration.

[0014] Motion-based calibration regards the calibration of camera and lidar extrinsic parameters as a hand-eye calibration problem, and does not require the camera and lidar to have overlapping fields of view. The basic principle of this method is to inversely solve the calibration parameters through a series of sensor movement sequences, and model it as a problem of solving homogeneous transformation equations. This calibration method is mostly used in the field of industrial robots, and has less restrictions on the field of view of cameras and radars, so it has unique advantages in certain specific application scenarios. The motion-based calibration method first needs to collect a series of motion data of the sensor at different positions and postures. First, easy-to-identify feature points are arranged in the environment, and observation data containing position and posture information of the camera and lidar at different positions are obtained through multiple movements in a known environment. After obtaining the motion sequence, the extrinsic calibration problem of the camera and lidar is modeled as a classic hand-eye calibration problem. The core of the hand-eye calibration problem is to solve the relative transformation relationship between the two sensors, which is usually expressed as a homogeneous transformation matrix. In this process, edge detection and other methods are usually used to extract features in the environment, and the influence of sensor noise is reduced through data filtering and optimization algorithms. For a simple example, assume that the motion sequences at different positions are A and B respectively. i and B i , where A i is the motion of the laser radar, B i is the movement of the camera, then its transformation relationship can be expressed as: A i X=XB i Where X is the transformation matrix between the camera and the lidar. By solving this equation, the extrinsic calibration parameters can be obtained. This calibration method is often used in industrial robots and automation systems, especially in scenarios where sensors need to be moved frequently or installed on robotic arms. In large scene environments, the motion calibration method can quickly estimate the initial extrinsic parameter values ​​of the sensor, and make fine adjustments based on this to improve the calibration efficiency. However, the motion calibration method has the following shortcomings:

[0015] 1) Harsh calibration conditions: The accuracy of motion-based calibration depends on artificially arranged environmental feature information, and sufficient feature points must exist in the environment to ensure the accuracy of the calibration results. If the identifiable environmental features are sparse or unevenly distributed, the calibration results may be inaccurate or even impossible to complete.

[0016] 2) The measured value is susceptible to interference: In this method, the noise of sensors such as IMU has a great impact on the calibration results. In practical applications, sensor data often contains noise and errors. If they are not processed, the calibration accuracy will be significantly reduced. Although the influence of noise can be reduced by edge detection and data filtering, this increases the computational complexity and has limited effect.

[0017] 3) Strong dependence on initial values: Motion calibration methods are mostly used to estimate initial values ​​or adjust sensor external parameters in large scene environments, and the calibration accuracy is not high. If the initial value estimation is inaccurate, it will directly affect the accuracy and robustness of the calibration results. This dependence on initial values ​​makes it difficult to obtain satisfactory calibration results in complex environments or applications with high precision requirements. Summary of the invention

[0018] Although the calibration method based on the calibration plate has high accuracy and stability in a static environment, it is highly dependent on the environment and manual operations, and is difficult to adapt to dynamic and complex application scenarios. In addition, the need for repeated calibration and the dependence on the calibration plate also increase the complexity and instability of the calibration work. Although the method based on motion calibration has solved the need for calibration without a calibration plate to a certain extent, it has the disadvantages of harsh hardware requirements, susceptibility to interference of measured values, and strong dependence on initial values, which limits its wide application. Therefore, in order to solve the above problems, the present invention provides a laser radar vision joint calibration method based on lane line extraction, which is more automated and robust than the calibration method based on the calibration plate, and the present invention does not rely on additional IMU and other attitude sensors, does not require artificial arrangement of environmental detection points, and provides a more efficient and reliable solution than the method based on motion calibration, and has obvious technical advantages and application prospects.

[0019] To achieve the above object, the present invention provides a laser radar vision joint calibration method based on lane line extraction, which specifically includes:

[0020] Step S1: LiDAR point cloud extraction and processing: firstly, environmental point cloud data is obtained from the LiDAR sensor, the obtained data is subjected to plane fitting by the RANSAC algorithm, then lane line point cloud is extracted, and finally the extracted lane line point cloud is clustered to form a line feature point cloud;

[0021] Step S2: Image processing: First, obtain the environment image data from the camera, perform image semantic segmentation on the environment image data, use the DeepLab-V3 network to extract the pixel area of ​​the lane line in the environment image data, and finally perform feature mask binarization processing;

[0022] Step S3: Find line features in the environmental point cloud data and the environmental image data respectively, use the line features in the environmental image data and the line features in the environmental point cloud data for consistency matching, solve the three-line perspective problem, and obtain a preliminary external parameter estimation by establishing a corresponding relationship between three of the line features;

[0023] Step S4: Use the optimization algorithm to refine and calibrate the preliminary matching results, improve the calibration accuracy through multiple measurements and data fusion, and output the external parameter calibration results of the lidar and camera, including the rotation matrix and translation vector.

[0024] In one embodiment of the present invention, step S1 is specifically as follows:

[0025] Step S101: Use the RANSAC algorithm to perform plane fitting on the acquired environmental point cloud data to separate the ground plane point cloud P ground and the non-ground plane point cloud P object , and set the thickness of the ground plane;

[0026] Step S102: Based on the intensity value of the point cloud fed back by the laser radar, ground Extract lane line point cloud P lane , and extract the lane line part by setting the intensity threshold and retaining the points whose intensity value is greater than the intensity threshold;

[0027] Step S103: Use the RANSAC line fitting algorithm to fit a straight line in the remaining points, remove outliers, and merge the dashed line segments into a continuous line segment;

[0028] Step S104: extract the lane line point cloud P lane Clustering is performed to form a line feature point cloud, and then abnormal points are removed again.

[0029] In one embodiment of the present invention, the thickness of the ground plane in step S101 is set to 0.2 m.

[0030] In one embodiment of the present invention, the binarization process in step S2 is specifically as follows:

[0031] Generate a binary mask M based on the semantic segmentation result line , where the lane line part is set to 1 and the other parts are set to 0, then the part with a value of 1 in the binary mask represents the pixel area of ​​the lane line.

[0032] In one embodiment of the present invention, step S3 specifically includes:

[0033] Step S301: In the environment image data, the line features of the lane lines are extracted by using Canny edge detection and Hough transform It is a binary mask M generated by the semantic segmentation result line , obtained by performing Hough transform on the detected pixel-level edges after Canny edge detection;

[0034] In the environmental point cloud data, the RANSAC algorithm is used to perform straight line fitting to extract line features

[0035] Step S302: extract line features from the environmental point cloud data Transform to world coordinate system W:

[0036]

[0037] Among them, I represents the image coordinate system in the environmental image data, P represents the point cloud coordinate system in the environmental point cloud data, and p I is the point cloud in the image coordinate system, p P is the point cloud in the point cloud coordinate system, is the transformation matrix from the world coordinate system to the image coordinate system, is the transformation matrix from the point cloud coordinate system to the world coordinate system, is the translation vector from the world coordinate system to the image coordinate system;

[0038] Step S303: Obtain the parameters of preliminary extrinsic parameter estimation:

[0039]

[0040] Among them, R is the rotation matrix of the external parameter, and t is the translation vector of the external parameter.

[0041] In one embodiment of the present invention, step S4 is specifically as follows:

[0042] Step S401: Project the lane line features in the environmental point cloud data onto the image plane using the following formula:

[0043]

[0044] Among them, K is the camera intrinsic parameter matrix, is the i-th lane feature point in the environmental point cloud data, is the i-th lane feature point projected onto the image plane;

[0045] Step S402: Calculate the projection of each lane line vector in the environment point cloud data according to each lane line feature point projected onto the image plane, and simultaneously obtain each lane line vector in the environment image data, and calculate the projection of each lane line vector in the environment point cloud data and the cosine cosine (θ i ):

[0046]

[0047] in, is the i-th lane line vector in the environment image data, is the projection of the lane line vector in the corresponding environment point cloud data on the image plane;

[0048] Step S403: Calculate the cost function J(R, t) of the vector cosine:

[0049]

[0050] Step S405: By minimizing the cost function Solve for the external parameters, calculate the partial derivatives of the error with respect to the external parameters, construct the Jacobian matrix J, and iteratively update the rotation matrix R and the translation vector t:

[0051] Δθ=-(J T J) -1 J T e

[0052] R←R·exp(Δθ)

[0053] t←t+Δt

[0054] Where e is the error vector and e = 1-cos(θ i ), Δθ is the small change of the rotation vector, exp(Δθ) is the exponential mapping of the small change to the rotation matrix, and Δt is the small change of the translation vector;

[0055] Step S406: Determine whether the error has converged or whether the preset maximum number of iterations has been reached.

[0056] If the error converges or the maximum number of iterations has been reached, the iteration is stopped;

[0057] Otherwise, return to step S405 to continue iteration;

[0058] Step S407: Output the optimized extrinsic parameters, including the rotation matrix R and the translation vector t.

[0059] The laser radar vision joint calibration method based on lane line extraction provided by the present invention can achieve multiple beneficial effects such as fully automatic calibration, no need for manual intervention, strong anti-interference ability and low initial value dependence by reducing the dependence on calibration conditions. It solves the main problems in the prior art, improves the efficiency and accuracy of laser radar and camera extrinsic calibration, and significantly reduces the difficulty of laser radar and camera sensor calibration, which has important technical value and application prospects. Specifically including:

[0060] 1) Reduce the dependence on calibration conditions: By utilizing the common road marking features in road traffic scenarios, the present invention effectively reduces the dependence on infrastructure such as dedicated calibration plates, which not only simplifies the calibration process, but also significantly reduces the cost and complexity of calibration, making it unnecessary to prepare high-precision calibration plates in practical applications; at the same time, high-precision calibration of sensors can be achieved in a wider range of scenarios, enhancing the generalization ability of the system.

[0061] 2) No need for human intervention: The present invention realizes automatic external parameter calibration of LiDAR and camera by automatically detecting and matching natural features in the environment, reducing human intervention. It not only improves the calibration efficiency, but also avoids errors caused by human factors, thereby improving the stability and reliability of the calibration results. It is of particular significance for application scenarios that require frequent calibration, such as driverless cars and mobile robots.

[0062] 3) Fully automatic calibration: By introducing an improved optimization algorithm and multi-feature fusion technology, a fully automatic calibration process is realized. It can autonomously identify and extract natural features in the environment, perform feature matching and optimization calculations, and ultimately obtain high-precision external parameters. This can eliminate the need for manual operations and further improve the efficiency and accuracy of calibration. It is suitable for dynamic environments and large-scale deployment scenarios.

[0063] 4) Strong anti-interference ability: The present invention adopts a robust feature extraction and matching algorithm, which can effectively filter out environmental noise and interference, ensuring the accuracy and stability of the calibration results; through multiple measurements and data fusion technology, the anti-interference ability of the calibration is further improved, ensuring that high-precision external parameter calibration results can still be obtained in complex environments. It has important application value for application scenarios where sensor data is easily interfered, such as autonomous driving in outdoor environments.

[0064] 5) Low initial value dependence: Through an improved optimization algorithm, the present invention can automatically estimate and optimize the extrinsic parameters of the sensor in the absence of precise initial values, thereby reducing the dependence on the initial values. Even when the initial values ​​are inaccurate, accurate calibration results can be obtained through the autonomous optimization process, thereby improving the robustness and reliability of the calibration process, and providing important support for rapid deployment and adjustment in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 FIG. 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The present invention aims to reduce the difficulty of implementing laser radar and camera sensor calibration, thereby making it more practical and competitive in the field of multi-sensor fusion. The following is a description of the complete technical solution of the present invention, including examples of technical implementation methods. Figure 1 FIG. 1 is a flow chart of an embodiment of the present invention, as shown in FIG. Figure 1 As shown, this embodiment provides a laser radar vision joint calibration method based on lane line extraction, which specifically includes:

[0069] Step S1: LiDAR point cloud extraction and processing: firstly, environmental point cloud data is obtained from the LiDAR sensor, the obtained data is subjected to plane fitting by the RANSAC algorithm, then lane line point cloud is extracted, and finally the extracted lane line point cloud is clustered to form a line feature point cloud;

[0070] In this embodiment, step S1 is specifically as follows:

[0071] Step S101: Use the RANSAC algorithm (Random Sample Consensus) to perform plane fitting on the acquired environmental point cloud data to separate the ground plane point cloud P ground and the non-ground plane point cloud P object , and set the thickness of the ground plane; the thickness of the ground plane can be set to 0.2m, which is the maximum elevation offset between any two points in the point cloud;

[0072] Step S102: Since the road marking contains glass bead reflective material, based on the intensity value of the point cloud fed back by the laser radar, the ground plane point cloud P ground Extract lane line point cloud P lane , and by setting the intensity threshold I thresh , retain the intensity value greater than the intensity threshold I thresh Points are used to extract the lane line part;

[0073] Step S103: using the RANSAC line fitting algorithm (i.e., a method of fitting a straight line using the RANSAC algorithm) to fit a straight line to the remaining points, removing outliers therein, and merging the dotted line segments into a continuous line segment;

[0074] Step S104: extract the lane line point cloud P laneClustering is performed to form a line feature point cloud, and then abnormal points are removed again.

[0075] Among them, the removal of outliers and the secondary elimination of abnormal noise points can adopt any existing point cloud outlier removal method, which is not limited by the present invention.

[0076] Step S2: Image processing: First, obtain the environment image data from the camera, perform image semantic segmentation on the environment image data, use the DeepLab-V3 network to extract the pixel area of ​​the lane line in the environment image data, and finally perform feature mask binarization processing;

[0077] In other embodiments, in addition to using the DeepLab-V3 network to implement semantic segmentation, regular lane line targets can also be identified through edge detection, replacing the lane line semantic blocks implemented by the DeepLab-V3 network. Semantic segmentation algorithms such as FCN and SegNet can also be used. These methods can all achieve the ability to separate lane line targets.

[0078] In this embodiment, the binarization process in step S2 is specifically as follows:

[0079] Generate a binary mask M based on the semantic segmentation result line , where the lane line part is set to 1 and the other parts are set to 0, then the part with a value of 1 in the binary mask represents the pixel area of ​​the lane line.

[0080] The laser radar lane line features and the camera image binary mask lane line features together constitute the roadside environment features. Only when the features are fully extracted can subsequent calibration calculations be performed.

[0081] Step S3: Find line features in the environmental point cloud data and the environmental image data respectively, use the line features in the environmental image data and the line features in the environmental point cloud data for consistency matching (which can be regarded as coarse registration), solve the P3L (Perspective-Three-Line) problem, and obtain a preliminary external parameter estimation by establishing the correspondence between the three line features;

[0082] In this embodiment, step S3 specifically includes:

[0083] Step S301: In the environment image data, extract the line features of the road markings (lane lines) through Canny edge detection and Hough Transform It is a binary mask M generated by the semantic segmentation result line , obtained by performing Hough transform on the detected pixel-level edges after Canny edge detection;

[0084] In the environmental point cloud data, the RANSAC algorithm is used to perform straight line fitting to extract line features

[0085] If applied in a scene without road markings, it is also possible to use pre-manually set feature points for calibration. These feature points can be marking points pre-set on the road or in the environment. Although this method relies on manual settings, it can complete calibration in scenes without road markings. At the same time, in other embodiments, other natural features in the road, such as road edges, street light poles, traffic signs, road cracks, etc., can also be used as features to replace the lane lines of the above embodiments for calibration. These features can also be extracted by lidar and cameras and used for calibration. Although these alternatives do not have obvious advantages over the methods of the preferred embodiments, they still have obvious advantages over the prior art, including lower calibration costs, stronger robustness, lower dependence on the environment, and higher accuracy. Therefore, the technical solution of the present invention provides a more superior solution, which helps to improve the performance and practicality of lidar and camera calibration.

[0086] Step S302: extract line features from the environmental point cloud data Transform to the world coordinate system W(World):

[0087]

[0088] Where I represents the image coordinate system (Image) in the environmental image data, P represents the point cloud coordinate system (Pointcloud) in the environmental point cloud data, and p I is the point cloud in the image coordinate system, p P is the point cloud in the point cloud coordinate system, is the transformation matrix from the world coordinate system to the image coordinate system, is the transformation matrix from the point cloud coordinate system to the world coordinate system. Since the LiDAR point cloud naturally has three-dimensional coordinate characteristics, the transformation matrix from the LiDAR point cloud coordinate system to the world coordinate system is is known; is the translation vector from the world coordinate system to the image coordinate system;

[0089] Step S303: Obtain the parameters of preliminary extrinsic parameter estimation:

[0090]

[0091] Among them, R is the rotation matrix of the extrinsic parameter, and t is the translation vector of the extrinsic parameter. Taking the camera as an example, the extrinsic parameters of the camera mainly describe the position of the camera in the three-dimensional scene and the direction of the lens, which are usually represented by a rotation matrix R and a translation vector t to describe the position, direction and observation angle of the camera, thereby determining the angle from which the camera observes the scene. Therefore, in step S3, the preliminary calibrated extrinsic parameters are first obtained for subsequent precise alignment.

[0092] Step S4: Use the optimization algorithm to refine and calibrate the preliminary matching results, improve the calibration accuracy through multiple measurements and data fusion, and output the external parameter calibration results of the lidar and camera, including the rotation matrix and translation vector.

[0093] In this embodiment, step S4 is specifically as follows:

[0094] Step S401: The purpose of the refinement calibration is to minimize the angle error between the projection vector of the lane line vector in the point cloud on the image plane and the actual lane line vector. Therefore, the lane line features in the environment point cloud data are projected onto the image plane using the following formula:

[0095]

[0096] Among them, K is the camera intrinsic parameter matrix, is the i-th lane feature point in the environmental point cloud data, is the i-th lane feature point projected onto the image plane;

[0097] Step S402: Since this angle error can be measured by calculating the cosine similarity of two vectors, the projection of each lane line vector in the environment point cloud data is calculated according to each lane line feature point projected onto the image plane, and each lane line vector in the environment image data is obtained at the same time, and the projection of each lane line vector in the environment point cloud data and the cosine cosine (θ i );

[0098]

[0099] in, is the i-th lane line vector in the environment image data, is the projection of the lane line vector in the corresponding environment point cloud data on the image plane; since the present embodiment adopts the three-line perspective method, i here is the number of the extracted lane line, and the maximum value of i here is the number of the extracted lane lines, so as to use the cost function and iterate using all lane lines;

[0100] Step S403: Calculate the cost function J(R, t) of the vector cosine:

[0101]

[0102] Step S405: By minimizing the cost function Solve for the external parameters, calculate the partial derivatives of the error with respect to the external parameters, construct the Jacobian matrix J, and iteratively update the rotation matrix R and the translation vector t:

[0103] Δθ=-(J T J) -1 J T e

[0104] R←R·exp(Δθ)

[0105] t←t+Δt

[0106] Where e is the error vector and e = 1-cos(θ i ), Δθ is the small change of the rotation vector, exp(Δθ) is the exponential mapping of the small change to the rotation matrix, and Δt is the small change of the translation vector;

[0107] Step S406: Determine whether the error has converged or whether the preset maximum number of iterations has been reached.

[0108] If the error converges or the maximum number of iterations has been reached, the iteration is stopped;

[0109] Otherwise, return to step S405 to continue iteration;

[0110] Step S407: After iterative optimization, the final optimized high-precision extrinsic parameters are output, including the rotation matrix R and the translation vector t.

[0111] Through the above iterative optimization process, the accuracy of LiDAR and camera extrinsic calibration can be significantly improved, ensuring that stable and reliable calibration results can be obtained even in complex environments.

[0112] The laser radar vision joint calibration method based on lane line extraction provided by the present invention can solve the following deficiencies of the prior art:

[0113] 1) The calibration conditions are high

[0114] The primary purpose of the present invention is to solve the strong dependence of the existing technology on infrastructure such as calibration plates. The existing technology usually requires infrastructure such as calibration plates and feature recognition points. In the practical engineering process, the deployment of high-precision calibration plates will significantly increase the workload and difficulty of calibration, and increase the calibration cost. The present invention completes the external parameter calibration of the lidar and camera by identifying the common road marking features in road traffic scenes, avoiding the dependence on the calibration infrastructure in the calibration process.

[0115] 2) Human intervention is required

[0116] Existing calibration methods usually require a lot of manual intervention, such as manually placing calibration plates and manually marking feature points with the same name, which not only increases the workload, but also may introduce human errors, affecting the accuracy and stability of the calibration. The present invention realizes automatic external parameter calibration of LiDAR and camera by automatically detecting and matching natural features in the environment, such as road markings, etc., reducing manual intervention and improving calibration efficiency and reliability.

[0117] 3) Difficulty in repeated calibration

[0118] When the relative positions of the two sensors move, the existing calibration method is very user-unfriendly. Usually, the manufacturer needs to send staff to carry calibration equipment for on-site secondary calibration, which is not only time-consuming and labor-intensive, but also difficult to ensure the consistency and accuracy of each calibration result. The present invention can automatically complete secondary calibration in most scenarios by automatically identifying the characteristics of lane lines on the road, reducing the difficulty of repeated calibration.

[0119] 4) The measured value is susceptible to interference

[0120] In the prior art, during the calibration process, sensor data is easily affected by environmental noise and interference, resulting in a decrease in calibration accuracy. The present invention can effectively filter noise and interference by introducing a robust feature extraction and matching algorithm, thereby ensuring the stability and accuracy of the calibration results. In addition, the present invention further improves the anti-interference ability of the calibration through coarse alignment, fine alignment, multiple measurements, and data fusion, ensuring that a high-precision external parameter calibration result can still be obtained in a complex environment.

[0121] 5) Strong dependence on initial value

[0122] Existing calibration methods usually rely on the accuracy of the initial values ​​of the camera intrinsic parameters. If the calibration error of the camera intrinsic parameters is too large, it will directly affect the accuracy and convergence of the calibration results. The present invention uses an improved optimization algorithm to automatically estimate and optimize relevant parameters based on radar point cloud and image recognition features when there is a certain error in the initial value, thereby reducing the dependence on the initial value and improving the robustness and accuracy of the calibration process.

[0123] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0124] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser radar vision joint calibration method based on lane line extraction, characterized in that: include: Step S1: LiDAR point cloud extraction and processing: firstly, environmental point cloud data is obtained from the LiDAR sensor, the obtained data is subjected to plane fitting by the RANSAC algorithm, then lane line point cloud is extracted, and finally the extracted lane line point cloud is clustered to form a line feature point cloud; Step S2: Image processing: First, obtain the environment image data from the camera, perform image semantic segmentation on the environment image data, use the DeepLab-V3 network to extract the pixel area of ​​the lane line in the environment image data, and finally perform feature mask binarization processing; Step S3: Find line features in the environmental point cloud data and the environmental image data respectively, use the line features in the environmental image data and the line features in the environmental point cloud data for consistency matching, solve the three-line perspective problem, and obtain a preliminary external parameter estimation by establishing a corresponding relationship between three of the line features; Step S4: Use the optimization algorithm to refine and calibrate the preliminary matching results, improve the calibration accuracy through multiple measurements and data fusion, and output the external parameter calibration results of the lidar and camera, including the rotation matrix and translation vector.

2. The laser radar vision joint calibration method based on lane line extraction according to claim 1 is characterized in that: Step S1 is specifically as follows: Step S101: Use the RANSAC algorithm to perform plane fitting on the acquired environmental point cloud data to separate the ground plane point cloud P ground and the non-ground plane point cloud P object , and set the thickness of the ground plane; Step S102: Based on the intensity value of the point cloud fed back by the laser radar, ground Extract lane line point cloud P lane , and extract the lane line part by setting the intensity threshold and retaining the points whose intensity value is greater than the intensity threshold; Step S103: Use the RANSAC line fitting algorithm to fit a straight line in the remaining points, remove outliers, and merge the dashed line segments into a continuous line segment; Step S104: extract the lane line point cloud P lane Clustering is performed to form a line feature point cloud, and then abnormal points are removed again.

3. The laser radar vision joint calibration method based on lane line extraction according to claim 2 is characterized in that: In step S101 , the thickness of the ground plane is set to 0.2 m.

4. The laser radar vision joint calibration method based on lane line extraction according to claim 1 is characterized in that: The binarization process in step S2 is specifically as follows: Generate a binary mask M based on the semantic segmentation result line , where the lane line part is set to 1 and the other parts are set to 0, then the part with a value of 1 in the binary mask represents the pixel area of ​​the lane line.

5. The laser radar vision joint calibration method based on lane line extraction according to claim 4 is characterized in that: Step S3 specifically includes: Step S301: In the environment image data, the line features of the lane lines are extracted by using Canny edge detection and Hough transform It is a binary mask M generated by the semantic segmentation result line , obtained by performing Hough transform on the detected pixel-level edges after Canny edge detection; In the environmental point cloud data, the RANSAC algorithm is used to perform straight line fitting to extract line features Step S302: extracting line features from the environmental point cloud data Transform to world coordinate system W: Among them, I represents the image coordinate system in the environmental image data, P represents the point cloud coordinate system in the environmental point cloud data, and p I is the point cloud in the image coordinate system, p P is the point cloud in the point cloud coordinate system, is the transformation matrix from the world coordinate system to the image coordinate system, is the transformation matrix from the point cloud coordinate system to the world coordinate system, is the translation vector from the world coordinate system to the image coordinate system; Step S303: Obtain the parameters of preliminary extrinsic parameter estimation: Among them, R is the rotation matrix of the external parameter, and t is the translation vector of the external parameter.

6. The laser radar vision joint calibration method based on lane line extraction according to claim 1 is characterized in that: Step S4 is specifically as follows: Step S401: Project the lane line features in the environmental point cloud data onto the image plane using the following formula: Among them, K is the camera intrinsic parameter matrix, is the i-th lane feature point in the environmental point cloud data, is the i-th lane feature point projected onto the image plane; Step S402: Calculate the projection of each lane line vector in the environment point cloud data according to each lane line feature point projected onto the image plane, and simultaneously obtain each lane line vector in the environment image data, and calculate the projection of each lane line vector in the environment point cloud data and the cosine cosine (θ i ): in, is the i-th lane line vector in the environment image data, is the projection of the lane line vector in the corresponding environment point cloud data on the image plane; Step S403: Calculate the cost function J(R, t) of the vector cosine: Step S405: By minimizing the cost function Solve for the external parameters, calculate the partial derivatives of the error with respect to the external parameters, construct the Jacobian matrix J, and iteratively update the rotation matrix R and the translation vector t: Δθ=-(J T J) -1 J T e R←R·exp(Δθ) t←t+Δt Where e is the error vector and e = 1-cos(θ i ), Δθ is the small change of the rotation vector, exp(Δθ) is the exponential mapping of the small change to the rotation matrix, and Δt is the small change of the translation vector; Step S406: Determine whether the error has converged or whether the preset maximum number of iterations has been reached. If the error converges or the maximum number of iterations has been reached, the iteration is stopped; Otherwise, return to step S405 to continue iteration; Step S407: Output the optimized extrinsic parameters, including the rotation matrix R and the translation vector t.

Images