Mobile vehicle point cloud automatic registration method fusing visual and geometric weights

Abstract

This invention relates to the fields of computer vision and lidar signal processing technology, and discloses an automatic registration method for moving vehicle point clouds that integrates visual and geometric weights. It utilizes a lidar mounted on a fixed base such as a gantry to work in conjunction with a camera to register multiple frames of point clouds of passing moving vehicles and reconstruct a complete 3D model. Specifically, it constructs a coupled weight matrix using visual feature confidence and local geometric curvature of the point cloud, solves the initial pose using a weighted SVD algorithm, and introduces ground plane constraints to optimize registration consistency, thereby achieving automatic reconstruction of multiple frames of sparse point clouds into a complete vehicle model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and lidar signal processing technology, specifically to a method for registering multi-frame point clouds and reconstructing a complete 3D model of a passing moving vehicle by using lidar and a camera in collaboration, mounted on a fixed base such as a gantry. Background Technology

[0002] In recent years, applications such as vehicle 3D reconstruction, fine-grained perception of traffic targets, and digital roads have developed rapidly. High-precision point cloud models of vehicle targets are of great value in scenarios such as autonomous driving simulation, traffic law enforcement evidence collection, vehicle shape measurement, and accident reconnaissance. LiDAR can provide accurate 3D geometric information, but for roadside fixed-deployment LiDAR, single-frame vehicle point clouds are often characterized by sparseness, partial visibility, severe occlusion, and large changes in viewpoint, resulting in incomplete vehicle geometric features. Cameras can provide clear target boundaries, texture, and semantic information, but they are easily affected by factors such as illumination, motion blur, and reflections, and cannot directly provide reliable depth. Integrating both for multi-frame vehicle registration can obtain a more complete vehicle point cloud model under single-viewpoint conditions. Therefore, how to achieve stable registration under sparse point cloud conditions has become a key challenge.

[0003] Existing point cloud registration and 3D reconstruction methods for mobile vehicles can be mainly divided into three categories: methods based on motion / pose priors, such as invention patent (publication number: CN114119886A, title: High-precision map point cloud reconstruction method, device, vehicle, equipment and storage medium), which rely on motion compensation and global pose optimization, but are prone to failure when roadside fixed acquisition, vehicles lack reliable pose priors and have low inter-frame overlap; point cloud registration methods based on geometric constraints, such as invention patent (publication number: CN104700451A, title: Point cloud registration method based on iterative nearest point algorithm) and invention patent (publication number: CN110443836A, title: A point cloud based on planar features). Automatic data registration methods and devices are prone to getting stuck in local optima or mismatching under conditions of sparse, occluded, and symmetrical vehicle point clouds. Semantic / fusion / learning-based methods, such as the invention patent (publication number: CN110264502A, title: Point Cloud Registration Method and Device) which introduces semantic-assisted matching, and the invention patents (publication number: CN113724379A, title: 3D Reconstruction Method and Device for Fusion Image and Laser Point Cloud) and (publication number: CN112950786A, title: A Vehicle 3D Reconstruction Method Based on Neural Network), are sensitive to texture, point cloud density, and cross-frame correlation stability, and the error increases in weak texture or sparse point cloud scenes.

[0004] In summary, current registration methods cannot achieve high-precision automatic registration of mobile vehicle point clouds under conditions of sparse point clouds, uneven density, repetitive textures, and lack of physical constraints. There is a need for a vehicle point cloud fusion registration method that is adapted to fixed deployment scenarios of roadside gantries, effectively integrating the geometric features of LiDAR and the visual features of cameras to improve the registration accuracy and integrity of multi-frame fragmented point clouds. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes an automatic registration method for mobile vehicle point clouds that integrates visual and geometric weights.

[0006] This invention relates to a method for registering point clouds of passing moving vehicles across multiple frames and reconstructing a complete 3D model, mounted on a fixed base such as a gantry and utilizing the collaborative work of a LiDAR and a camera. Specifically designed for gantry-based shooting scenarios, this invention constructs a coupled weight matrix using visual feature confidence and the local geometric curvature of the point cloud. The initial pose is solved using a weighted SVD algorithm, and ground plane constraints are introduced to optimize registration consistency, thereby achieving automatic reconstruction of a complete vehicle model from multiple frames of sparse point clouds.

[0007] The technical solution of this invention is as follows:

[0008] An automatic registration method for mobile vehicle point clouds that integrates visual and geometric weights includes the following steps:

[0009] Step 1: Acquire the multimodal sensor data sequence during the target vehicle's movement and the various calibration parameters that have been pre-calibrated by the system;

[0010] Step 2: Preprocess the lidar point cloud data and camera image data respectively to complete image distortion correction, background removal and vehicle target extraction, and realize the data association between point cloud and image. Then, perform inter-frame image feature matching and filtering to establish the corresponding matching relationship between two-dimensional image features and three-dimensional point cloud.

[0011] Step 3: Calculate the geometric significance σ of a single point based on the constructed 3D corresponding point pairs. j Combined with visual confidence level c j The normalized fusion registration weights are calculated, and then the rotation matrix obtained by the solution is constrained and corrected in combination with the ground plane motion characteristics to complete the inter-frame coarse registration pose solution with weight constraints.

[0012] Step 4: Use the two-stage ICP algorithm to sequentially complete the coarse and fine registration of adjacent frames and the global point cloud optimization registration, and iteratively optimize to obtain the global optimal pose;

[0013] Step 5: Based on the global optimal pose, complete the global fusion of vehicle point clouds in all frames to construct a complete 3D reconstructed vehicle model.

[0014] Furthermore, step S1 specifically includes:

[0015] The system acquires a continuous frame sequence of lidar point cloud and camera image with time synchronization and overlapping fields of view while the target vehicle is in motion. At the same time, it retrieves a pre-calibrated set of system parameters, which includes the camera intrinsic parameter matrix, image distortion coefficients, and extrinsic parameter transformation matrix for transforming the lidar coordinate system to the camera coordinate system.

[0016] Furthermore, step S2 specifically includes:

[0017] Step 2.1: Perform ground plane fitting on each frame of LiDAR point cloud, filter out ground point clouds according to the set distance threshold, and separate and extract vehicle candidate point clouds;

[0018] Step 2.2: Correct the distortion of the original image by using the distortion model coordinate remapping method, generate a vehicle mask by binarization segmentation, and use the mask to remove the background of the image, retaining only the effective vehicle image area;

[0019] Step 2.3: Perform coordinate system transformation and pixel projection on the vehicle candidate point cloud to generate the corresponding frame projection point set;

[0020] Step 2.4: Perform depth feature matching on adjacent frame images, and successively filter by visual confidence threshold, polar geometric consistency, effective inlier count, and inter-frame motion displacement to remove mismatched points and invalid image frames.

[0021] Step 2.5: Based on the effective matching feature points, candidate 3D points are selected by combining pixel distance and depth information, the optimal matching relationship is determined, and a sparse 3D corresponding point pair set of adjacent frames is constructed.

[0022] Furthermore, step S3 specifically includes:

[0023] Step 3.1: Delineate the neighborhood range centered on the corresponding three-dimensional points, solve for the eigenvalues ​​of the neighborhood covariance matrix, and calculate the geometric significance σ corresponding to each matching point. j ;

[0024]

[0025] Among them, 𝜆 i Represents the covariance matrix Σ j The eigenvalues ​​obtained by eigenvalue decomposition satisfy Σ1≥Σ2≥Σ3≥0; j It is the point cloud in the current frame. The neighborhood point set is obtained by searching with center and radius r. ;

[0026] Step 3.2: Fuse the visual confidence c of the matching points j Calculate the initial registration weights based on geometric significance:

[0027]

[0028] Then, the weight normalization process is completed to obtain a unified and usable fusion registration weight;

[0029]

[0030] in, Represents the geometric gain coefficient. This represents the maximum geometric significance among all matching points. This represents the maximum value among all the initial registration weights for the matched points. This represents a very small positive number that prevents the denominator from being zero;

[0031] Step 3.3: Based on the fusion weights, construct the corresponding matrix to solve for the original rotation matrix and translation vector between frames, combined with the characteristics of the vehicle's horizontal motion scene:

[0032]

[0033] Where R(2,1) and R(1,1) represent the elements in the 2nd row and 1st column of the original rotation matrix R, respectively;

[0034] The original rotation matrix is ​​subjected to orthogonal projection constraint processing to retain the rotation components required for the horizontal motion of the ground and remove redundant rotation components, finally obtaining the inter-frame coarse registration homogeneous transformation matrix.

[0035] Furthermore, step S4 specifically includes:

[0036] Step 4.1: First, use the coarse registration pose obtained in step S3 as the initial value, and perform point-to-point ICP registration on the point clouds of adjacent frames;

[0037] Step 4.2: Using the existing global pose as the initial prediction value, perform point-to-plane ICP global fine registration between the current frame point cloud and the global accumulated point cloud. After iterative convergence, determine the final global optimal spatial pose.

[0038] Furthermore, step S5 specifically includes:

[0039] The vehicle target point clouds extracted from all time frames are uniformly transformed to the same global coordinate system for stitching and integration, completing the global fusion of point clouds and generating a complete 3D reconstruction model of the target vehicle.

[0040] Compared with existing methods, the beneficial effects of the present invention through the above method are as follows:

[0041] 1) A weighting mechanism coupling visual confidence and geometric curvature significance is proposed, which effectively combines image texture and point cloud geometry, and can maintain high-precision registration even when geometric features are missing, such as smooth car body or visual features are missing, such as low light environment.

[0042] 2) This method uses a weighted coupling of geometric saliency measure based on PCA eigenvalue decomposition and visual confidence. It assigns reasonable weights to regions with significant geometric features but poor visual texture, such as smooth car roofs, and regions with rich visual features but flat geometry, such as car body stickers, thereby improving the robustness of registration.

[0043] 3) By introducing ground plane constraints in the coarse registration stage, the problems of "trajectory drift" and "model warping" that are very easy to occur when moving vehicles are superimposed in multiple frames are effectively solved;

[0044] 4) By utilizing the significance of theoretical curvature, this method still has a strong reconstruction capability for LiDAR data with low beamwidth and sparse scanning, thus reducing hardware costs. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the overall process of the automatic registration and reconstruction method for mobile vehicle point clouds according to the present invention.

[0046] Figure 2 This is a top view diagram of point cloud ground filtering and vehicle extraction in step 2.1 of the present invention;

[0047] Figure 3 This is the result image after image distortion correction and background removal in step 2.2 of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the effect of reprojecting the lidar point cloud onto the vehicle image in step 2.3 of the present invention.

[0049] Figure 5 This is a diagram of the original feature matching results based on the LoFTR network in step 2.4.1 of this invention;

[0050] Figure 6 This is a schematic diagram of the final effective feature matching pairs after screening in step 2.4.3 of the present invention;

[0051] Figure 7 This is a schematic diagram illustrating the selection of typical points for calculating geometric saliency in step 3.1 of the present invention;

[0052] Figure 8 This is a visualization of the registration weight distribution of all feature points in step 3.2 of this invention;

[0053] Figure 9 This is a schematic diagram of the complete three-dimensional point cloud of the vehicle after the stitching is completed in step 5 of the present invention. Detailed Implementation

[0054] The following detailed embodiments illustrate the implementation of the automatic registration and reconstruction method for moving vehicle point clouds of the present invention. This embodiment uses a set of synchronously acquired multi-frame sparse and unevenly dense point cloud sequences and image sequences as input data. It automatically registers and fuses the incomplete vehicle point clouds from multiple frames during the vehicle's passage through the field of view, outputting a complete vehicle point cloud model and automatically measuring the vehicle's dimensions. The overall process is as follows: Figure 1 As shown, the specific steps include the following:

[0055] Step 1: Obtain the multimodal sensor data sequence and calibration parameters of the target vehicle, specifically as follows: Obtain the continuous frame lidar point cloud sequence when the target vehicle moves within the sensor's field of view. With camera image sequence ,in, This represents the point cloud of the LiDAR in the i-th frame. Let N represent the i-th frame of the camera image sequence, and let N represent the number of frames in the LiDAR and camera image sequences. The point cloud sequence and the image sequence are synchronized in time, and the LiDAR and camera have overlapping fields of view. Simultaneously, a pre-calibrated set of system parameters Q is acquired, which includes at least the camera intrinsic parameter matrix K and distortion coefficients D. o And the extrinsic transformation matrix T from the lidar coordinate system to the camera coordinate system L In this embodiment, the data acquisition system is built on a traffic gantry, and both share a common field of view. The camera's intrinsic parameters were pre-calibrated and distortion correction was performed, and joint extrinsic parameter calibration of the lidar and camera was completed. The intrinsic parameter matrix K is [[1339.36, 0, 811.46], [0, 1339.92, 623.10], [0, 0, 1]], and the distortion coefficient D... o The inequalities are [-0.0568, 0.1190, 0.00048, -0.0011, -0.0625], and the extrinsic transformation matrix T from the lidar to the camera is... L The values ​​are [[0.9999, 0.0019, 0.0101, 0.0895], [0.0101, -0.0017, -0.9999, 0.1457], [-0.0018, 0.9999, -0.0017, 0.1204], [0, 0, 0, 1]]. Point clouds and image sequences were continuously collected as the vehicle passed at 20 km / h, for a total of 26 frames. For ease of explanation, frames 4 and 16 of the sequence are selected for registration illustration.

[0056] Step 2: Preprocess and correlate the acquired point cloud and image data respectively. The specific steps are as follows:

[0057] Step 2.1: Ground filtering and vehicle point cloud extraction operations are performed on each point cloud frame. Fitting the ground plane And extract the candidate point cloud of vehicles that satisfy equation (1). ;

[0058] (1)

[0059] Where (x, y, z) represents the three-dimensional coordinates of the point cloud, and d represents the ground distance threshold; for The result of the preprocessing operation is a top view of the point cloud, as shown below. Figure 2 As shown;

[0060] Step 2.2: Use a coordinate remapping method based on radial and tangential distortion models to remap the original image. Distortion correction is performed to obtain a distortion-free image. According to equation (2), a mask is used. i Background removal is performed on the image to obtain the vehicle image. :

[0061] (2)

[0062] Where × represents pixel-wise multiplication, Mask i Using the YOLOv8 algorithm for distortion-free images Binarization segmentation is performed to obtain; for The results of the preprocessing operation are as follows Figure 3 As shown;

[0063] Step 2.3: Generate the vehicle candidate point cloud according to equation (3). Transformed to camera coordinate system According to equation (4) Mapping to the image pixel coordinate system yields the two-dimensional projection point pixel coordinates (u, v), and all these projection points constitute the projection point set U of the current frame. i As shown in equation (4):

[0064] (3)

[0065] (4)

[0066] in, This represents the projection function; in this embodiment, the point cloud... Projected onto image like Figure 4 As shown;

[0067] Step 2.4: Perform feature matching and filtering for adjacent frames. The specific steps are as follows:

[0068] Step 2.4.1: Use a deep feature matching method to match adjacent frames. , Perform feature matching and output the coordinates and confidence scores of the matching key points:

[0069] (5)

[0070] in, Let c represent the pixel coordinates of the j-th pair of matching points in frame i-1 and frame i, respectively. j Let represent the visual confidence level of the j-th pair of matching points; in this embodiment, and Inputting the LoFTR network yields an initial number of 936 matching pairs, as shown below. Figure 5 As shown;

[0071] Step 2.4.2: Set the confidence threshold ,right All matching pairs are filtered, and those that satisfy equation (6) are retained to obtain the confidence-filtered set. ;

[0072] (6)

[0073] In this embodiment, a confidence threshold is set. The value was 0.45, retaining 329 valid matching points;

[0074] Step 2.4.3: Apply equation (7) to the set The matching points in the model are subjected to extreme geometric consistency screening to obtain a set of interior points that conform to the geometric model. , where e j The epipolar error F is calculated according to equation (8). i The fundamental matrix is ​​calculated using the RANSAC algorithm:

[0075] (7)

[0076] (8)

[0077] in, Indicates the extreme threshold. and These represent the homogeneous form of the corresponding pixel coordinates; in this embodiment, after RANSAC geometric verification, 188 pairs of final valid matches are retained, as shown in the following figures. Figure 6 As shown;

[0078] Step 2.4.4: When If the frame cannot be stably registered, then the frame is skipped. M min This represents the minimum number of interior points required for stable registration; in this embodiment, it refers to the number of effective matches. Continue with the registration process;

[0079] Step 2.4.5: Calculate the average displacement modulus D of all retained matching pairs according to equation (9). avg When D avg Less than the threshold If the inter-frame motion information is insufficient, the frame will be skipped.

[0080] (9)

[0081] Where n represents the total number of retained matching pairs; This represents the displacement modulus of the j-th matching pair; in this embodiment, the minimum displacement modulus is 22, and the final average displacement modulus is 384.74, indicating that the vehicle is in motion, and registration continues.

[0082] Step 2.5: Establish 2D-3D correspondence:

[0083] Step 2.5.1: For the valid matching set Each pair of matching points in the image is projected onto the previous frame's point set U. i-1 With the current frame projection point set U i The retention distance is no greater than Based on equation (10), a set of candidate projection points is constructed. ,in This represents the pixel distance between the image matching point and the projection point calculated according to equation (11):

[0084] (10)

[0085] (11)

[0086] Where 'a' represents the traversal index of the element within the projection-associated set. Represents the projection point set U i Two-dimensional pixel coordinates of the projection point in the middle. express The corresponding candidate 3D point coordinates, express Depth in the camera coordinate system;

[0087] Step 2.5.2: According to equation (12), select from the candidate set Determine the optimal matching 3D point in the current frame. When multiple candidate projection points exist, the projection point with the smallest pixel distance is selected; when the difference in pixel distance among multiple candidate projection points is not greater than a threshold... When selecting the projection point with the smallest depth, construct the set C of sparse three-dimensional corresponding point pairs of adjacent frames according to equation (13). i :

[0088] (12)

[0089] (13)

[0090] In this embodiment, the search pixel radius The value is 0.4m, the nearest neighbor number k is 5, and a KD-Tree search is used. 188 feature points were successfully associated with the 3D point cloud.

[0091] Step 3: Coarse inter-frame registration based on visual confidence and geometric saliency weighting, the specific steps are as follows:

[0092] Step 3.1:

[0093] Calculate set C according to equation (14) i Each Geometric significance σ j ,in, Represents the covariance matrix Σ j The eigenvalues ​​obtained by eigenvalue decomposition satisfy the following conditions: , Σ j It is the point cloud in the current frame. The neighborhood point set is obtained by searching with center and radius r. According to equation (15), the following can be calculated:

[0094] (14)

[0095] (15)

[0096] in, Let σ represent the neighborhood centroid, where σ is the centroid of the neighborhood. j A larger value indicates stronger local curvature / non-planarity, making it more likely to be located in areas of geometric abrupt change such as edges or corners; a smaller value indicates a surface closer to a flat surface. In this embodiment, to illustrate the effectiveness of the weights, [the following values ​​are selected]. Two typical points on the above are as follows Figure 7 As shown, the PCA eigenvalues ​​of point A within a neighborhood of radius R of 0.4m are calculated, and the results are obtained. It is 0.0180. It is 0.0096. The value is 0.0033, and point B is at the PCA eigenvalue. It is 0.0492. It is 0.0149. It is 0.0001;

[0097] Step 3.2: Construct the fusion weight matrix, combining it with the visual confidence score c. j Geometric significance The final registration weight w is calculated according to equation (16). j And normalize according to equation (17):

[0098] (16)

[0099] (17)

[0100] in, This represents the geometric gain coefficient, used to adjust the contribution of edge / corner features in registration. This represents the maximum geometric significance among all matching points. This represents the maximum value among all the initial registration weights for the matched points. This represents a minimal positive number that prevents the denominator from being zero; in this embodiment, the geometric significance of point A is... The visual confidence score is 0.1068, and the visual confidence score is c. j The final weight w is 0.90. j The geometric significance of point B is 0.38. The visual confidence level is 0.0012. j The final weight w is 0.46. j The value is 0.46. The calculation is performed on all feature points as follows: Figure 8 As shown;

[0101] Step 3.3: Solving the relative pose based on ground plane constraints. The specific steps are as follows:

[0102] Step 3.3.1: Construct the weighted covariance matrix H according to equation (18), where μ s and μ t The weighted centroids of the effective matching 3D point sets in the current frame and the previous frame are calculated according to equation (19); SVD decomposition is performed on H. The original rotation matrix is ​​obtained. :

[0103] (18)

[0104] (19)

[0105] Where U, S, and V represent the singular value decomposition results of matrix H; in this embodiment, the current frame The weighted centroid is [0.5183, 7.5691, 1.3906], target frame. The weighted centroid of the corresponding point set is [0.3808, 5.5043, -0.4841], the weighted covariance matrix is ​​[[0.3005, 0.0107, 0.0548], [0.0266, 0.0587, 0.1343], [0.0639, 0.2504, 0.5744]], and the original rotation matrix is ​​[[0.9968, 0.0804, -0.0035], [-0.0783, 0.9790, 0.1880], [0.0185, -0.1871, 0.9822]];

[0106] Step 3.3.2: Considering the near-horizontal motion of the vehicle in the gantry shooting scene, R is orthogonally projected according to equation (20), forcibly retaining the yaw angle around the Z-axis perpendicular to the ground. Construct the homogeneous transformation matrix according to equation (21) As a result of coarse registration, among which The ground plane constraint rotation matrix is ​​constructed according to equation (22), and t is the translation vector calculated according to equation (23):

[0107] (20)

[0108] (twenty one)

[0109] (twenty two)

[0110] (twenty three)

[0111] Where R(2,1) and R(1,1) represent the elements in the 2nd row and 1st column of the original rotation matrix R, respectively; in this embodiment, the original Roll is -10.7859°, Pitch is -1.0615°, and Yaw is -4.4933°. After attitude constraint, Roll is 0.0000°, Pitch is 0.0000°, and Yaw is -4.4933°, corresponding to the constraint rotation matrix [[0.9969,0.0783, 0.0000], [-0.0783, 0.9969, 0.0000], [0.0000, 0.0000, 1.0000]]; the translation vector is [-0.7289, -2.0009, -1.8746]; the coarse registration transformation matrix is ​​[[0.9969, [0.0783, 0.0000, -0.7289], [-0.0783, 0.9969, 0.0000, -2.0009], [0.0000, 0.0000, 1.0000, -1.8746], [0.0000, 0.0000, 0.0000, 1.0000]];

[0112] Step 4: Two-stage ICP fine registration and global fusion, the specific steps are as follows:

[0113] Step 4.1: Point-to-point ICP coarse registration, using the data obtained in Step 3... As initial values, the current frame point cloud Point cloud from the previous frame Perform point-to-point ICP to obtain coarse registration. In this embodiment, point-to-point ICP is performed, the threshold is set to 0.2m, and the process is completed after 5 iterations. The coarse registration overlap is 1.0000 and the root mean square error of the coarse registration is 0.0835.

[0114] Step 4.2: Point-to-plane ICP global fine registration. Based on Step 4.1, and using the current global pose as the initial prediction value, the current frame point cloud is... With global point cloud P f Perform point-to-plane ICP to obtain the final pose. Update the global point cloud according to the final pose:

[0115] (twenty four)

[0116] In this embodiment, point-to-surface ICP is performed with a threshold of 0.1m. After 8 iterations and convergence, the fine registration overlap is 0.8413, the root mean square error is 0.0456, and the final fine registration transformation matrix is ​​[[0.9094,0.2569, 0.3271, -2.7873], [-0.2531, 0.9659, -0.0548, -1.7577], [-0.3300, -0.0330, 0.9434, -1.7946], [0.0000, 0.0000, 0.0000, 1.0000]]. P... 16 Integrating into the global map; after stitching together 26 frames of data, this embodiment yields a complete 3D point cloud model of the vehicle, as shown below. Figure 9 As shown. Due to the lack of the actual trajectory of the moving vehicle, we quantitatively back-calculated the cumulative longitudinal translation error by the shrinkage of the physical dimensions of the final reconstructed model, and conducted a comparative experiment with existing similar registration methods. Compared with existing similar methods at home and abroad, the present invention has the smallest relative errors in length, width, and height of the reconstructed vehicle 3D point cloud. The comparison results of the relative errors in the reconstruction dimensions of each method are shown in Table 1:

[0117] Table 1

[0118] ICP 1.2 17 9.2 GICP 2.4 13.4 8 CN111322964A 1 0.5 6 CN118247295A 1.9 1 2.5 This method 0.6 0.2 0.8

[0119] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for automatic registration of mobile vehicle point clouds by integrating visual and geometric weights, characterized in that, Includes the following steps: Step 1: Acquire the multimodal sensor data sequence during the target vehicle's movement and the various calibration parameters that have been pre-calibrated by the system; Step 2: Preprocess the lidar point cloud data and camera image data respectively to complete image distortion correction, background removal and vehicle target extraction, and realize the data association between point cloud and image. Then, perform inter-frame image feature matching and filtering to establish the corresponding matching relationship between two-dimensional image features and three-dimensional point cloud. Step 3: Calculate the geometric significance σ of a single point based on the constructed 3D corresponding point pairs. j Combined with visual confidence level c j The normalized fusion registration weights are calculated, and then the rotation matrix obtained by the solution is constrained and corrected in combination with the ground plane motion characteristics to complete the inter-frame coarse registration pose solution with weight constraints. Step 4: Use the two-stage ICP algorithm to sequentially complete the coarse and fine registration of adjacent frames and the global point cloud optimization registration, and iteratively optimize to obtain the global optimal pose; Step 5: Based on the global optimal pose, complete the global fusion of vehicle point clouds in all frames to construct a complete 3D reconstructed vehicle model.

2. The automatic registration method for mobile vehicle point clouds integrating visual and geometric weights according to claim 1, characterized in that, Step S1 is as follows: The system acquires a continuous frame sequence of lidar point cloud and camera image with time synchronization and overlapping fields of view while the target vehicle is in motion. At the same time, it retrieves a pre-calibrated set of system parameters, which includes the camera intrinsic parameter matrix, image distortion coefficients, and extrinsic parameter transformation matrix for transforming the lidar coordinate system to the camera coordinate system.

3. The automatic registration method for mobile vehicle point clouds integrating visual and geometric weights according to claim 1, characterized in that, Step S2 is as follows: Step 2.1: Perform ground plane fitting on each frame of LiDAR point cloud, filter out ground point clouds according to the set distance threshold, and separate and extract vehicle candidate point clouds; Step 2.2: Correct the distortion of the original image by using the distortion model coordinate remapping method, generate a vehicle mask by binarization segmentation, and use the mask to remove the background of the image, retaining only the effective vehicle image area; Step 2.3: Perform coordinate system transformation and pixel projection on the vehicle candidate point cloud to generate the corresponding frame projection point set; Step 2.4: Perform depth feature matching on adjacent frame images, and successively filter by visual confidence threshold, polar geometric consistency, effective inlier count, and inter-frame motion displacement to remove mismatched points and invalid image frames. Step 2.5: Based on the effective matching feature points, candidate 3D points are selected by combining pixel distance and depth information, the optimal matching relationship is determined, and a sparse 3D corresponding point pair set of adjacent frames is constructed.

4. The automatic registration method for mobile vehicle point clouds integrating visual and geometric weights according to claim 1, characterized in that, Step S3 is as follows: Step 3.1: Delineate the neighborhood range centered on the corresponding three-dimensional points, solve for the eigenvalues ​​of the neighborhood covariance matrix, and calculate the geometric significance σ corresponding to each matching point. j ; ； Among them, 𝜆 i Represents the covariance matrix Σ j The eigenvalues ​​obtained by eigenvalue decomposition satisfy Σ1≥Σ2≥Σ3≥0; j It is the point cloud in the current frame. The neighborhood point set is obtained by searching with center and radius r. ; Step 3.2: Fuse the visual confidence c of the matching points j Calculate the initial registration weights based on geometric significance: ； Then, the weight normalization process is completed to obtain a unified and usable fusion registration weight; ； in, Represents the geometric gain coefficient. This represents the maximum geometric significance among all matching points. This represents the maximum value among all the initial registration weights for the matched points. This represents a very small positive number that prevents the denominator from being zero; Step 3.3: Based on the fusion weights, perform weighted pose calculation on the corresponding 3D point pairs in adjacent frames to obtain the original rotation matrix and translation vector between frames, and extract the yaw angle by combining the characteristics of the vehicle's horizontal motion scene. ； Where R(2,1) and R(1,1) represent the elements in the second row and first column of the original rotation matrix R, respectively; The original rotation matrix is ​​constrained and corrected based on the yaw angle, retaining the rotation components required for ground horizontal motion and eliminating redundant rotation components, finally obtaining the inter-frame coarse registration homogeneous transformation matrix.

5. The automatic registration method for mobile vehicle point clouds integrating visual and geometric weights according to claim 1, characterized in that, Step S4 is as follows: Step 4.1: First, use the coarse registration pose obtained in step S3 as the initial value, and perform point-to-point ICP registration on the point clouds of adjacent frames; Step 4.2: Using the existing global pose as the initial prediction value, perform point-to-plane ICP global fine registration between the current frame point cloud and the global accumulated point cloud. After iterative convergence, determine the final global optimal spatial pose.

6. The automatic registration method for mobile vehicle point clouds integrating visual and geometric weights according to claim 1, characterized in that, Step S5 is as follows: The vehicle target point clouds extracted from all time frames are uniformly transformed to the same global coordinate system and stitched together to complete the global fusion of point clouds and generate a complete 3D reconstruction model of the target vehicle.

Images