Laser radar and camera joint calibration method and system based on pixel point alignment

Abstract

The invention provides a laser radar and camera joint calibration method based on pixel point alignment, and the method comprises the steps: carrying out the preprocessing of point cloud data and an RGB image, and obtaining an optimized point cloud and a corrected image; carrying out edge feature recognition and extraction to obtain an edge pixel point set; projecting the optimized point cloud to a plane of the corrected image, and extracting an intensity gradient salient point set of the virtual intensity image; establishing a mapping relation of the two point sets through bidirectional nearest neighbor search, calculating similarity of mapping points, and performing data screening through the similarity to obtain an optimized data set; and constructing a joint loss function of the optimized data set according to a conversion relation among the calibration plate, the laser and the camera coordinate system, and solving through a least square method to obtain an external parameter matrix. According to the method, through intensity correction, image distortion compensation, cross-modal feature manifold mapping, bidirectional nearest neighbor matching and multi-constraint joint optimization, high-precision external parameter calibration is realized, and the robustness and real-time performance of calibration in a dynamic scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of sensor calibration technology, and in particular to a laser radar and camera joint calibration method and system based on pixel alignment. Background Art

[0002] With the rapid development of autonomous driving and robotic perception technologies, multimodal sensor fusion has become a core approach to environmental perception. As the most complementary sensor combination, lidar and cameras have a highly reliable joint calibration, which directly determines the reliability of the fused data. Lidar provides three-dimensional spatial geometry, while cameras provide two-dimensional texture details. Accurately determining the rigid transformation parameters (extrinsic matrix) between their coordinate systems is a fundamental prerequisite for spatial alignment of multi-source data.

[0003] Current calibration methods fall into three main categories. The first is traditional methods based on calibration plates, such as checkerboards and ArUco codes. These methods establish a correspondence between the point cloud and the image by detecting the corners or edges of the calibration plate, but are susceptible to changes in illumination, occlusion, and calibration plate manufacturing errors. The second is target-free mutual information optimization, which uses the statistical correlation of sensor data to optimize parameters. However, these methods are sensitive to initial values ​​and have high computational complexity. The third is edge feature alignment, which matches the intensity edges of the point cloud with the visual edges of the image. However, because laser intensity is affected by distance attenuation and incident angle interference, image edges are susceptible to distortion and texture interference, which can easily lead to insufficient feature consistency. Therefore, it is necessary to design a joint LiDAR and camera calibration method and system based on pixel alignment. Summary of the Invention

[0004] The purpose of the present invention is to provide a joint calibration method and system for lidar and camera based on pixel alignment, which realizes high-precision extrinsic parameter calibration through physically perceived intensity correction and image distortion compensation, cross-modal feature manifold mapping, bidirectional nearest neighbor matching and multi-constraint joint optimization, so as to improve the robustness and real-time performance of calibration in dynamic scenes.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A laser radar and camera joint calibration method based on pixel alignment includes the following steps:

[0007] Acquire the LiDAR point cloud data and the camera's RGB image in real time and perform preprocessing operations to obtain preprocessed data; the preprocessing data includes: optimizing the point cloud and correcting the image; the preprocessing operations include: intensity normalization and image distortion correction;

[0008] Perform edge feature recognition and extraction on the corrected image to obtain an edge pixel point set;

[0009] Project the optimized point cloud onto the plane of the rectified image to generate a virtual intensity map, and extract the set of intensity gradient significant points in the virtual intensity map;

[0010] The mapping relationship between the edge pixel point set and the intensity gradient salient point set is established through bidirectional nearest neighbor search;

[0011] The pre-processed data is dynamically screened based on the mapping point similarity calculated from the mapping relationship to obtain an optimized data set;

[0012] According to the transformation relationship between the calibration plate coordinate system, the laser coordinate system and the camera coordinate system, the optimization data set is jointly constrained to obtain the joint loss function;

[0013] The joint loss function is solved by the least squares method to obtain the external parameter matrix.

[0014] Optionally, the point cloud data of the lidar and the RGB image of the camera are acquired in real time, and preprocessed to obtain preprocessed data, including:

[0015] Perform distance attenuation compensation on the point cloud intensity of the point cloud data to obtain compensated intensity;

[0016] The incident angle is calculated by the surface normal vector of the point cloud data, and the compensation intensity is angle-corrected according to the incident angle to obtain the angle intensity;

[0017] The angle intensity is normalized by the Sigmoid function to obtain the normalized intensity;

[0018] A two-dimensional histogram is constructed according to the curvature of the point cloud data, and the point cloud data is jointly filtered by the normalized intensity and the two-dimensional histogram to obtain an intensity subset;

[0019] The intensity gradient amplitude of the intensity subset is calculated by the Sobel operator, and the intensity subset is screened according to the intensity gradient amplitude to obtain the optimized point cloud.

[0020] Optionally, the point cloud data of the lidar and the RGB image of the camera are acquired in real time, and preprocessed to obtain preprocessed data, including:

[0021] The calibration plate area in the RGB image is identified by the illumination-adaptive threshold segmentation method to obtain a binary mask;

[0022] Pixel corner detection is performed on the binary mask through gradient interpolation method, and a dynamic distortion field is generated based on the detected natural feature points;

[0023] The nonlinear distortion parameter inversion of the dynamic distortion field is performed according to the theoretical grid coordinates of the calibration plate area to obtain the distorted image;

[0024] The distortion component of the distorted image and the real scene information are separated by wavelet transform to obtain a separated image;

[0025] The RGB image is projected onto the plane of the separated image through forward mapping, and the projection result is blank-filled by bilinear interpolation to obtain the corrected image.

[0026] Optionally, edge feature recognition and extraction are performed on the corrected image to obtain an edge pixel point set, including:

[0027] Perform Retinex decomposition on the corrected image to obtain the enhanced image;

[0028] The gradient of the enhanced image is calculated using Laplace Gaussian kernels of different sizes to obtain a multi-scale gradient field;

[0029] The isolated noise points and texture pseudo edges of the multi-scale gradient field are removed by the curvature of adjacent pixels to obtain a set of candidate edge points.

[0030] The candidate edge point set is restructured through the gradient histogram to obtain a significant edge skeleton;

[0031] The edge offset of the salient edge skeleton is calculated, and the key points of the salient edge skeleton are screened according to the edge offset to obtain the edge pixel point set.

[0032] Optionally, the optimized point cloud is projected onto the plane of the rectified image to generate a virtual intensity map, and a set of intensity gradient salient points in the virtual intensity map is extracted, including:

[0033] Perform occlusion-aware projection of the optimized point cloud via ray tracing to obtain a virtual intensity map;

[0034] The virtual intensity map is subjected to noise suppression according to the projection point signal-to-noise ratio and the theoretical intensity upper limit to obtain a noise-reduced intensity map;

[0035] The denoised intensity map is mapped to a two-dimensional Riemannian manifold and then aggregated through exponential mapping to obtain the gradient amplitude map;

[0036] The optical flow field of the gradient amplitude map is subjected to non-maximum suppression to obtain a set of intensity gradient salient points.

[0037] Optionally, a mapping relationship between the edge pixel point set and the intensity gradient salient point set is established through a bidirectional nearest neighbor search, including:

[0038] The edge pixel point set and the intensity gradient salient point set are mapped to the unit circle manifold and the three-dimensional sphere respectively to obtain the edge manifold and intensity manifold;

[0039] Construct spatiotemporal constraints based on the optical flow displacement field of the edge manifold and the point cloud scene flow of the intensity manifold, and integrate the mapping point pairs that meet the spatiotemporal constraints into a candidate matching set;

[0040] Calculate the similarity matrix of the candidate matching set through a multi-layer perceptron;

[0041] An energy function is constructed according to the similarity matrix, and the energy function is used as a mapping relationship.

[0042] Optionally, the pre-processed data is dynamically screened based on the mapping point similarity calculated from the mapping relationship to obtain an optimized data set, including:

[0043] The geometric residual is calculated based on the physical properties of the mapped point pairs;

[0044] The similarity of the mapping points is obtained by calculating the geometric residual;

[0045] The data with mapping point similarity less than the preset similarity threshold in the preprocessed data are eliminated to obtain the optimized data set.

[0046] Optionally, the optimization dataset is jointly constrained according to the transformation relationship between the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system to obtain a joint loss function, including:

[0047] The transformation relationship between the calibration plate coordinate system, laser coordinate system and camera coordinate system is uniformly expressed to obtain the rotation matrix;

[0048] The mutual projection error between coordinate systems is obtained through the rotation matrix, and the mutual projection error is integrated into a multi-constraint error vector;

[0049] A joint loss function is constructed through multiple constrained error vectors.

[0050] A laser radar and camera joint calibration system based on pixel alignment, including:

[0051] The data acquisition module is used to acquire the point cloud data of the lidar and the RGB image of the camera in real time, and perform preprocessing operations to obtain preprocessed data; the preprocessing data includes: optimizing the point cloud and correcting the image; the preprocessing operations include: intensity normalization and image distortion correction;

[0052] Image feature extraction module, used to identify and extract edge features of the corrected image to obtain edge pixel point sets;

[0053] A point cloud feature extraction module is used to project the optimized point cloud onto the plane of the corrected image, generate a virtual intensity map, and extract a set of intensity gradient significant points in the virtual intensity map;

[0054] Feature association module, used to establish the mapping relationship between edge pixel point sets and intensity gradient salient point sets through bidirectional nearest neighbor search;

[0055] The data screening module is used to dynamically screen the pre-processed data according to the mapping point similarity calculated by the mapping relationship to obtain an optimized data set;

[0056] The multi-constraint joint optimization module is used to jointly constrain the optimization data set according to the transformation relationship between the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system to obtain a joint loss function;

[0057] The calibration output module is used to solve the joint loss function through the least squares method to obtain the external parameter matrix.

[0058] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention provides a joint calibration method for laser radar and camera based on pixel alignment, the method comprising: acquiring point cloud data of the laser radar and the RGB image of the camera in real time, and performing preprocessing operations to obtain preprocessed data; the preprocessed data comprises: optimized point cloud and corrected image; the preprocessing operations comprise: intensity normalization processing and image distortion correction; edge feature recognition and extraction of the corrected image to obtain an edge pixel point set; projecting the optimized point cloud onto the plane of the corrected image to generate a virtual intensity map, and extracting the intensity gradient significant point set in the virtual intensity map; establishing a mapping relationship between the edge pixel point set and the intensity gradient significant point set through bidirectional nearest neighbor search; dynamically screening the preprocessed data according to the mapping point similarity calculated according to the mapping relationship to obtain an optimized data set; jointly constraining the optimized data set according to the transformation relationship between the calibration plate coordinate system, the laser coordinate system and the camera coordinate system to obtain a joint loss function; solving the joint loss function through the least squares method to obtain an external parameter matrix. This method achieves high-precision extrinsic parameter calibration through physically perceived intensity correction and image distortion compensation, cross-modal feature manifold mapping, bidirectional nearest neighbor matching and multi-constraint joint optimization, improving the robustness and real-time performance of calibration in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 This is a flow chart of the laser radar and camera joint calibration method of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1 As shown, the present invention provides a laser radar and camera joint calibration method based on pixel alignment, comprising the following steps:

[0064] Step 100: Acquire the point cloud data of the lidar and the RGB image of the camera in real time, and perform preprocessing operations to obtain preprocessed data; the preprocessing data includes: optimizing the point cloud and correcting the image; the preprocessing operations include: intensity normalization and image distortion correction;

[0065] Specifically, the original point cloud intensity of the point cloud data is represented as I r The Euclidean distance between the target point and the radar center is d1. The compensation intensity is calculated according to the inverse square law of energy attenuation. The calculation formula is: where d r is the calibration reference distance, I c To compensate for intensity, the algorithm eliminates intensity distribution deviations caused by distance differences, thereby achieving distance attenuation compensation. The incident angle is then calculated using the geometric relationship between the point cloud surface normal and the laser incident direction. The incident angle is then used to perform angular correction on the compensated intensity according to Lambert's cosine law, yielding the angular intensity. This eliminates the attenuation of reflected energy caused by differences in the object's surface orientation. The angular intensity is then mapped to the [0, 1] interval using a Sigmoid function. The slope parameter of the function controls the normalization sensitivity, and the center point is the median of the current frame intensity. This achieves global normalization of the intensity distribution and enhances signal separability in low-contrast areas, resulting in the normalized intensity. The curvature value of each point is then calculated based on the local surface of the point cloud. A two-dimensional histogram is constructed with the curvature and normalized intensity as axes. Density estimation is then performed on the two-dimensional histogram using a bivariate Gaussian kernel. The subset of point clouds located within the probability density peak interval is retained to form the intensity subset. Finally, the intensity gradient amplitude of the intensity subset is calculated by the Sobel operator, and the points whose intensity gradient amplitude is greater than the preset dynamic threshold are screened out to generate the optimized point cloud. The dynamic threshold is determined by the mean and standard deviation of the intensity gradient amplitude.

[0066] More specifically, the cumulative distribution function of each local area in the RGB image is calculated using an 8×8 pixel grid in the V channel of the HSV color space to achieve V channel equalization. The preset size ratio of the calibration plate is used as a constraint, and the equalized V channel is binarized using the Otsu threshold method. All closed contour candidates are extracted through connected domain analysis to obtain a binary mask. The Shi-Tomasi corner detection algorithm is then used within the mask area to obtain discrete corner points. A displacement field is established based on the theoretical grid coordinates of the calibration plate, and thin plate spline interpolation is used to diffuse the discrete corner point displacements to the entire image. This forms a dynamic distortion field, which is then modeled as an overlay model, expressed as: where r 2 =x 2 +y 2 , (x,y) are the normalized coordinates in the image, r is the normalized radial distance from the current pixel to the optical center, k1 is the quadratic radial distortion coefficient, k2 is the quartic radial distortion coefficient, p1 and p2 are the horizontal and vertical shear distortions, respectively, and Δx and Δy are the distortion corrections. The least squares method is used to solve the distortion parameter vector [k1, k2, p1, p2] to generate the distorted image. A two-dimensional discrete wavelet transform is performed on the distorted image, decomposing it into low-frequency approximation components (LL), horizontal high-frequency components (LH), vertical high-frequency components (HL), and diagonal high-frequency components (HH). Coefficients in the LH, HL, and HH subbands with amplitudes below a preset adaptive threshold are reset to zero, resulting in a separated image containing both the distortion components and the true scene information. Finally, the RGB image is projected onto the separated image plane via forward mapping. Bilinear interpolation is performed on the projected image, and the blank areas after the projection are filled with morphological closing operations to obtain the corrected image.

[0067] It should be noted that intensity correction through distance attenuation compensation and incident angle correction, combined with image distortion removal using a dynamic distortion field and wavelet separation, improves the physical consistency between point cloud intensity features and image edge features. Combined with joint filtering and gradient amplitude screening, noise interference is eliminated, ensuring the quality of input data for subsequent cross-modal alignment.

[0068] Step 200: performing edge feature recognition and extraction on the corrected image to obtain an edge pixel point set;

[0069] Specifically, a multi-scale Gaussian wrapping strategy is employed to convolve the rectified image with three Gaussian kernels of varying standard deviations to obtain an illumination estimate for the illumination component. Based on the illumination estimate, the reflectance component is calculated via logarithmic domain differencing. The illumination and reflectance components are then weightedly fused to produce an enhanced image. Three different-sized Laplacian of Gaussian kernels (small, medium, and large) are then employed to capture fine edges in the enhanced image, balance noise suppression with edge continuity, and extract the macrostructure of the enhanced image. The gradient response maps obtained through the three Gaussian convolutions are fused into a multi-scale gradient field. The gradient magnitudes within adjacent 5×5 windows are sampled along the normal direction of the gradient field, and a quadratic curve is fitted, where the curvature of the curve is characterized by its second-order derivative. Points that satisfy the following two conditions—a gradient magnitude greater than 0.1 times the maximum gradient magnitude and a curvature less than a preset curvature threshold—are classified as candidate edge points. This approach eliminates high-frequency texture oscillations and isolated noise in the gradient field. The candidate point set is then divided into 8×8 pixel local windows. A gradient direction histogram is constructed within each window, and peaks with amplitudes exceeding 25% of the global maximum are retained as dominant directions for structural reconstruction, resulting in a salient edge skeleton. Finally, with the current point as the center, grayscale profiles of 3 pixels on each side are sampled along the gradient perpendicular to the edge skeleton. The offset is calculated, and points with an offset of less than 1.5 pixels are selected as key points, thus forming a set of edge pixels.

[0070] It should be noted that the multi-scale gradient fusion method is used to eliminate the thickness differences between edges, and the dual threshold filtering of curvature and amplitude is used to suppress the interference of texture and noise. The gradient histogram is combined for reconstruction to ensure the topological integrity of the structure, which greatly improves the positioning accuracy of pixel points.

[0071] Step 300: Projecting the optimized point cloud onto the plane of the corrected image to generate a virtual intensity map, and extracting a set of intensity gradient significant points in the virtual intensity map;

[0072] Specifically, the depth traversal is performed along the ray direction of the optimized point cloud with a step size of 0.5 pixels. When the ray enters the local neighborhood of the point cloud, the depth variance of the points in the neighborhood is calculated. If the depth variance is less than 0.05m, the minimum depth point is recorded. If there are multiple valid neighborhoods, only the minimum depth point of the nearest neighborhood is retained as the projection point. For the successfully projected pixels, the median of the intensity values ​​in the neighborhood is assigned to obtain a virtual intensity map. The virtual intensity map is then subjected to two levels of noise suppression. The first level is soft threshold filtering based on the signal-to-noise ratio. The signal-to-noise ratio is calculated based on the intensity mean and standard deviation in the 3×3 neighborhood of each projection point. When the signal-to-noise ratio is greater than 2.5, the intensity of the current projection point remains unchanged, otherwise the intensity is suppressed to μ I tanh(0.5·SNR), where μ Iis the neighborhood intensity mean, tanh(·) is the hyperbolic tangent function, and SNR is the signal-to-noise ratio. The second level exponentially truncates the abnormal highlight value according to the maximum reflection intensity of the lidar, and the expression is: I D =I max (1-e -kI’ ), where I max is the upper limit of the sensor's theoretical strength, k is the attenuation coefficient, I' is the intensity value of the first level, and the denoised intensity map is output. The denoised intensity map is then treated as a scalar field on a two-dimensional Riemannian manifold. Starting from the current point, geodesic propagation along the gradient direction vector is performed via an exponential mapping to achieve information aggregation and generate a gradient magnitude map. A pixel detection band is constructed along the optical flow direction and normal of the gradient magnitude map. Points are retained only if the center point amplitude is the maximum within the detection band and satisfies the dual threshold conditions of 0.2×global maximum gradient amplitude > center point amplitude > 0.05×global maximum gradient amplitude. This ultimately results in a set of salient intensity gradient points.

[0073] It should be noted that the projection mechanism combining ray tracing with occlusion processing ensures the consistency of the point cloud and image space. At the same time, the two-level noise suppression using SNR filtering and intensity truncation improves the reliability of the data, and the calculation error is reduced through non-maximum suppression, which significantly improves the robustness of the overall method in motion blurred scenes.

[0074] Step 400: establishing a mapping relationship between an edge pixel point set and an intensity gradient significant point set through a bidirectional nearest neighbor search;

[0075] Specifically, the gradient direction angle of each edge point is calculated and double-angle mapping is performed to map the edge pixel point set to the unit circle manifold to form the edge manifold. The intensity gradient salient point set is mapped to a three-dimensional sphere through Gaussian kernel smoothing operation to form the intensity manifold. Then, the spatiotemporal constraint condition is constructed based on the optical flow displacement field of the edge manifold and the point cloud scene flow of the intensity manifold, which is expressed as arccos(φ(p i )·ψ(q j ))<δ φ , where p i is a point in the optical flow displacement field, q j is a point in the point cloud scene flow, φ(p i ) is p i Coordinates on the unit circle manifold, ψ(q j ) is q j Coordinates on the three-dimensional spherical manifold, δ φ is the manifold distance threshold, which is 0.7 in some embodiments. The mapping point pairs (p i ,q j) are integrated into a candidate matching set. Then, the similarity matrix of the candidate matching set is calculated by a multi-layer perceptron; the multi-layer perceptron consists of an input layer, a hidden layer, and an output layer. The input layer extracts the 12-dimensional feature vector of the candidate matching set, and after the standard normal accumulation is performed by the GeLU activation function in the hidden layer with 512 nodes, the single node activation is performed by the Sigmoid function of the output layer, thereby outputting the similarity matrix. Finally, the energy function is constructed based on the similarity matrix. And the energy function is used as the mapping relationship, where M is the binary matching matrix, m ij is the image point, s ij is the similarity score of the mapping point pair, calculated by a multi-layer perceptron, and λ is the constraint weight coefficient, which is 0.7 in some embodiments.

[0076] Step 500: Dynamically screen the pre-processed data based on the mapping point similarity calculated from the mapping relationship to obtain an optimized data set;

[0077] Specifically, the geometric residual is first calculated based on the physical properties of the mapped point pairs. The calculation formula is: Wherein, α is a weight coefficient, which is 0.6 in some embodiments, is the rigid body transformation matrix from the laser radar to the camera, Q j is the three-dimensional coordinate of the point cloud intensity salient point, K is the camera intrinsic parameter matrix, is the homogeneous coordinate of the edge point of the image, × is the vector cross product operation, and ||·|| is the Euclidean norm. The similarity B of the mapping point is then calculated by geometric residual ij , the calculation formula is: Where w is the blending weight, which is 0.7 in some embodiments, and σ g is the Gaussian kernel standard deviation, σ l is the Laplace kernel scale parameter, r ij is the geometric residual. ij The data with a value less than 0.75 are used to obtain the optimized data set.

[0078] Step 600: Jointly constraining the optimized data set according to the conversion relationship between the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system to obtain a joint loss function;

[0079] Specifically, based on the transformation relationship between the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system, the RANSAC algorithm is used to fit the plane equations of the three coordinate systems, and the expression is: in is the plane unit normal vector, is the distance from the plane to the origin. Take the plane normal vector as the Z-axis direction of the calibration plate coordinate system, and get the Z-axis vector Normalize the principal component vectors along the edge of the calibration plate to obtain the X-axis vector. Calculate the cross product of the Z-axis vector and the X-axis vector to obtain the Y-axis vector. Concatenate the X-, Y-, and Z-axis vectors into a rotation matrix. Use the rotation matrix to project the coordinate systems against each other, obtaining three projection errors that are then concatenated into a multi-constrained error vector. Finally, construct a joint loss function based on the multi-constrained error vector, expressed as: Among them, ε k is the multi-constraint error vector, ρ(·) is the Geman-McClure kernel function, is the rotation matrix.

[0080] Step 700: Solve the joint loss function using the least squares method to obtain an extrinsic parameter matrix.

[0081] Specifically, the rotation matrix and translation vector in the joint loss function are first parameterized into Lie algebraic form. Then, the Jacobian matrix of the loss function with respect to the Lie algebra parameters is calculated using the Gauss-Newton iteration method, and the normal equations are solved in each iteration. During the iteration, the LM algorithm is used to adaptively adjust the damping factor to balance the convergence speed and stability. The algorithm terminates when the parameter update amount is less than the threshold or the maximum number of iterations is reached. Finally, the optimized Lie algebra parameters are converted into the extrinsic parameter matrix in the SE(3) group through exponential mapping.

[0082] The present invention also provides a laser radar and camera joint calibration system based on pixel alignment, comprising:

[0083] The data acquisition module is used to acquire the point cloud data of the lidar and the RGB image of the camera in real time, and perform preprocessing operations to obtain preprocessed data; the preprocessing data includes: optimizing the point cloud and correcting the image; the preprocessing operations include: intensity normalization and image distortion correction;

[0084] Image feature extraction module, used to identify and extract edge features of the corrected image to obtain edge pixel point sets;

[0085] A point cloud feature extraction module is used to project the optimized point cloud onto the plane of the corrected image, generate a virtual intensity map, and extract a set of intensity gradient significant points in the virtual intensity map;

[0086] Feature association module, used to establish the mapping relationship between edge pixel point sets and intensity gradient salient point sets through bidirectional nearest neighbor search;

[0087] The data screening module is used to dynamically screen the pre-processed data according to the mapping point similarity calculated by the mapping relationship to obtain an optimized data set;

[0088] The multi-constraint joint optimization module is used to jointly constrain the optimization data set according to the transformation relationship between the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system to obtain a joint loss function;

[0089] The calibration output module is used to solve the joint loss function through the least squares method to obtain the external parameter matrix.

[0090] The beneficial effects of the present invention are as follows:

[0091] 1) Through distance attenuation compensation, incident angle correction, and Sigmoid normalization, the influence of environmental interference on laser intensity is eliminated, and the physical authenticity of the point cloud intensity characteristics is improved;

[0092] 2) Through illumination adaptive segmentation, gradient interpolation distortion field inversion and wavelet separation technology, image distortion and texture noise are effectively suppressed, ensuring the spatial accuracy of image edge features;

[0093] 3) The mutual projection errors among the calibration plate, laser, and camera coordinate systems are integrated and optimized through Lie algebra parameterization and LM algorithm, which significantly improves the calibration accuracy of the extrinsic parameter matrix.

[0094] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0095] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A laser radar and camera joint calibration method based on pixel alignment, characterized in that: The steps include: Acquire the point cloud data of the lidar and the RGB image of the camera in real time, and perform preprocessing operations to obtain preprocessed data; The preprocessing data includes: optimizing point clouds and correcting images; the preprocessing operations include: intensity normalization and image distortion correction; Performing edge feature recognition and extraction on the corrected image to obtain an edge pixel point set; Projecting the optimized point cloud onto the plane of the corrected image to generate a virtual intensity map, and extracting a set of intensity gradient significant points in the virtual intensity map; Establishing a mapping relationship between the edge pixel point set and the intensity gradient significant point set through a bidirectional nearest neighbor search; Dynamically screening the pre-processed data according to the mapping point similarity calculated from the mapping relationship to obtain an optimized data set; The optimized data set is jointly constrained according to the conversion relationship between the calibration plate coordinate system, the laser coordinate system and the camera coordinate system to obtain a joint loss function; The joint loss function is solved by the least square method to obtain an extrinsic parameter matrix.

2. The laser radar and camera joint calibration method based on pixel alignment according to claim 1 is characterized in that: Acquire the LiDAR point cloud data and the camera RGB image in real time and perform preprocessing operations to obtain preprocessed data, including: Performing distance attenuation compensation on the point cloud intensity of the point cloud data to obtain a compensated intensity; Calculating an incident angle using a surface normal vector of the point cloud data, and performing angle correction on the compensation intensity according to the incident angle to obtain an angle intensity; Normalizing the angle intensity using a Sigmoid function to obtain a normalized intensity; constructing a two-dimensional histogram according to the curvature of the point cloud data, and jointly filtering the point cloud data using the normalized intensity and the two-dimensional histogram to obtain an intensity subset; The intensity gradient amplitude of the intensity subset is calculated using a Sobel operator, and data of the intensity subset is screened according to the intensity gradient amplitude to obtain the optimized point cloud.

3. The laser radar and camera joint calibration method based on pixel alignment according to claim 1, characterized in that: Acquire the LiDAR point cloud data and the camera RGB image in real time and perform preprocessing operations to obtain preprocessed data, including: Identify the calibration plate area in the RGB image by using an illumination-adaptive threshold segmentation method to obtain a binary mask; Performing pixel corner detection on the binary mask by using a gradient interpolation method, and generating a dynamic distortion field based on the detected natural feature points; performing nonlinear distortion parameter inversion on the dynamic distortion field according to theoretical grid coordinates of the calibration plate area to obtain a distorted image; Separating the distortion component of the distorted image and the real scene information by wavelet transform to obtain a separated image; The RGB image is projected onto the plane of the separated image by forward mapping, and the projection result is blank-filled by bilinear interpolation to obtain the corrected image.

4. The laser radar and camera joint calibration method based on pixel alignment according to claim 1, characterized in that: Performing edge feature recognition and extraction on the corrected image to obtain an edge pixel point set, including: Performing Retinex decomposition on the corrected image to obtain an enhanced image; performing gradient calculation on the enhanced image using Laplacian-Gaussian kernels of different sizes to obtain a multi-scale gradient field; Remove isolated noise points and texture pseudo edges of the multi-scale gradient field by the curvature of adjacent pixel points to obtain a set of candidate edge points; Reconstructing the structure of the candidate edge point set through a gradient histogram to obtain a significant edge skeleton; An edge offset of the significant edge skeleton is calculated, and key points of the significant edge skeleton are screened using the edge offset to obtain the edge pixel point set.

5. The laser radar and camera joint calibration method based on pixel alignment according to claim 1, characterized in that: Projecting the optimized point cloud onto the plane of the corrected image to generate a virtual intensity map, and extracting a set of intensity gradient significant points in the virtual intensity map, including: performing occlusion-aware projection on the optimized point cloud by ray tracing to obtain the virtual intensity map; performing noise suppression on the virtual intensity map according to the projection point signal-to-noise ratio and the theoretical intensity upper limit to obtain a noise-reduced intensity map; Mapping the denoised intensity map to a two-dimensional Riemannian manifold and then aggregating the result through exponential mapping to obtain a gradient amplitude map; Non-maximum suppression is performed on the optical flow field of the gradient magnitude map to obtain the intensity gradient salient point set.

6. The laser radar and camera joint calibration method based on pixel alignment according to claim 1, characterized in that: Establishing a mapping relationship between the edge pixel point set and the intensity gradient significant point set through a bidirectional nearest neighbor search includes: Mapping the edge pixel point set and the intensity gradient salient point set to a unit circle manifold and a three-dimensional sphere, respectively, to obtain an edge manifold and an intensity manifold; Constructing spatiotemporal constraints based on the optical flow displacement field of the edge manifold and the point cloud scene flow of the intensity manifold, and integrating mapping point pairs that satisfy the spatiotemporal constraints into a candidate matching set; Calculate the similarity matrix of the candidate matching set by a multi-layer perceptron; An energy function is constructed according to the similarity matrix, and the energy function is used as the mapping relationship.

7. The laser radar and camera joint calibration method based on pixel alignment according to claim 6, characterized in that: Dynamically screening the pre-processed data according to the mapping point similarity calculated from the mapping relationship to obtain an optimized data set includes: Calculating a geometric residual according to the physical properties of the mapping point pair; Obtaining the mapping point similarity by calculating the geometric residual; The data whose mapping point similarity in the preprocessed data is less than a preset similarity threshold is eliminated to obtain the optimized data set.

8. The laser radar and camera joint calibration method based on pixel alignment according to claim 1, characterized in that: The optimized data set is jointly constrained according to the conversion relationship between the calibration plate coordinate system, the laser coordinate system and the camera coordinate system to obtain a joint loss function, including: The conversion relationship among the calibration plate coordinate system, the laser coordinate system and the camera coordinate system is uniformly expressed to obtain a rotation matrix; Obtaining the mutual projection error between the coordinate systems through the rotation matrix, and integrating the mutual projection error into a multi-constraint error vector; The joint loss function is constructed using the multiple constraint error vectors.

9. A laser radar and camera joint calibration system based on pixel alignment, characterized in that: include: The data acquisition module is used to obtain the point cloud data of the lidar and the RGB image of the camera in real time, and perform preprocessing operations to obtain preprocessed data; The preprocessing data includes: optimizing point clouds and correcting images; the preprocessing operations include: intensity normalization and image distortion correction; An image feature extraction module is used to identify and extract edge features of the corrected image to obtain an edge pixel point set; a point cloud feature extraction module, configured to project the optimized point cloud onto the plane of the corrected image, generate a virtual intensity map, and extract a set of intensity gradient significant points in the virtual intensity map; A feature association module, configured to establish a mapping relationship between the edge pixel point set and the intensity gradient significant point set through a bidirectional nearest neighbor search; A data screening module, configured to dynamically screen the pre-processed data based on the mapping point similarity calculated from the mapping relationship to obtain an optimized data set; a multi-constraint joint optimization module, configured to perform joint constraints on the optimization data set according to the conversion relationship among the calibration plate coordinate system, the laser coordinate system, and the camera coordinate system, to obtain a joint loss function; The calibration output module is used to solve the joint loss function by the least square method to obtain an external parameter matrix.

Images