Cross-source point cloud registration method based on adversarial neural network

Abstract

This application belongs to the field of computer vision technology and discloses a cross-source point cloud registration method based on adversarial neural networks. The method includes: inputting a multi-frame LiDAR point cloud sequence and a UAV video sequence; segmenting each frame of the LiDAR point cloud into ground points and non-ground points and registering them to obtain a global LiDAR point cloud; performing motion recovery structure reconstruction on the UAV video sequence to generate a dense color point cloud; inputting the global LiDAR point cloud and the SfM dense color point cloud into an MMAlignNet network to construct a shared latent feature space; solving for the rigid transformation matrix from the LiDAR point cloud to the SfM dense color point cloud; and aligning the cross-modal point clouds according to the rigid transformation matrix to obtain a multimodal 3D reconstruction result. This application improves the registration accuracy, completeness, and robustness of large-scale outdoor environment 3D reconstruction by fusing UAV imagery and mobile LiDAR data, combined with cross-modal feature alignment and geometric constraint optimization methods.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, specifically relating to a cross-source point cloud registration method based on adversarial neural networks. Background Technology

[0002] With the development of spatial perception and environmental modeling technologies, 3D reconstruction has become a crucial foundational technology in fields such as large-scale outdoor scene digitization, urban monitoring, and environmental change assessment. Traditional 3D reconstruction methods primarily rely on single data sources, such as aerial imagery, terrestrial laser scanning (TLS), or photogrammetric point clouds. However, due to the inherent limitations of different sensors in terms of spatial coverage and data density, single-modal methods often struggle to provide comprehensive and accurate models of large-scale, complex outdoor environments. Therefore, the fusion of heterogeneous sensor data, especially UAV imagery and terrestrial mobile lidar (LiDAR) point cloud data, has become an important direction for multimodal 3D reconstruction, leveraging the respective advantages of each to improve geometric accuracy and visual fidelity in complex scenes.

[0003] Currently, UAV imagery and mobile LiDAR mounted on unmanned ground vehicles (UGVs) demonstrate significant potential in outdoor 3D reconstruction, exhibiting strong complementarity. UAVs can efficiently acquire large-scale, high-resolution imagery containing rich semantic information, enabling detailed depiction of scene surfaces and materials; while ground-based mobile LiDAR provides high-precision geometric measurements of ground structures and environmental features, enhancing modeling accuracy, especially in complex terrain or low-altitude visibility areas. The combination of the two can achieve comprehensive spatial coverage and reconstruction adaptable to varying structural complexities. However, heterogeneous sensor data exhibits significant differences in imaging mechanisms, viewing angles, and sampling densities, posing considerable challenges to cross-source feature alignment and geometric registration.

[0004] Existing research attempts to fuse multimodal data by extracting features through handcrafted features or deep neural networks. Handcrafted feature methods rely on texture and geometric information and are only suitable for environments with simple structures; while deep learning methods, under supervised training, can extract robust and discriminative features, improving alignment accuracy in complex outdoor environments. However, most learning-based methods are still limited by the differences in heterogeneous modal feature spaces, and obtaining sufficient labeled multimodal data for training is costly and difficult to implement in practical engineering. Therefore, there is an urgent need to develop registration methods that can efficiently align multimodal point clouds in real outdoor environments to improve the practicality of 3D reconstruction.

[0005] Furthermore, large-scale and structurally complex outdoor environments often present severe occlusion and low-texture areas, further increasing the difficulty of 3D reconstruction tasks. Multimodal data fusion can improve the geometric accuracy and texture integrity of 3D reconstruction to some extent by integrating complementary information from different sensors. However, different sensors vary significantly in terms of spatial resolution, sampling density, observation perspective, and data structure, making effective alignment and fusion of multimodal data still a considerable challenge.

[0006] In summary, how to effectively reduce the feature differences between different modal data, achieve high-precision feature alignment and fusion of UAV imagery and ground-based LiDAR point clouds, and improve the accuracy and completeness of 3D reconstruction in complex outdoor environments remains a key research focus. Summary of the Invention

[0007] To address the aforementioned technical issues, this application provides a cross-source point cloud registration method based on adversarial neural networks. This method improves the registration accuracy, integrity, and robustness of large-scale outdoor environment 3D reconstruction by fusing UAV imagery and mobile LiDAR data, and combining cross-modal feature alignment and geometric constraint optimization methods.

[0008] To achieve the above objectives, this application employs the following technical solution:

[0009] This application presents a cross-source point cloud registration method based on adversarial neural networks, comprising the following steps:

[0010] Step 1: Input a multi-frame lidar point cloud sequence collected by an unmanned ground vehicle (UGV) and a drone video sequence collected by an unmanned aerial vehicle (UAV). Divide the multi-frame lidar point cloud sequence into a ground point cloud set and a non-ground point cloud set.

[0011] Step 2: Use a ground-to-non-ground dual-branch ICP registration strategy to register the ground point cloud set and the non-ground point cloud set respectively, and perform frame-by-frame registration and accumulation of multi-frame lidar point clouds to generate a global lidar point cloud.

[0012] Step 3: Perform Structure of Motion (SfM) reconstruction on the UAV video sequence acquired by the UAV to restore the camera attitude and generate SfM dense color point cloud;

[0013] Step 4: Input the global LiDAR point cloud generated in Step 2 and the SfM dense color point cloud generated in Step 3 into the cross-source point cloud registration network MMAlignNet. The cross-source point cloud registration network MMAlignNet includes a TGSC-Attn feature generation module, a GAN-FeaAlign feature alignment module, an RCF-RobustMatch robust matching module, and a weighted singular value decomposition (WSVD) solving module. The deep features of the global LiDAR point cloud and the SfM dense color point cloud are extracted through the TGSC-Attn feature generation module.

[0014] Step 5: Use the GAN-FeaAlign feature alignment module to perform adversarial alignment between the deep features of the global lidar point cloud and the deep features of the SfM dense color point cloud to construct a shared latent feature space.

[0015] Step 6: Calculate the similarity matrix between cross-modal point clouds using the RCF-RobustMatch robust matching module, and obtain the set of coarse matching corresponding point pairs based on the similarity matrix and bidirectional mutual consistency constraints. Then, combine the local neighborhood geometric consistency filtering to obtain the set of fine matching point pairs.

[0016] Step 7: Input the set of fine matching point pairs and its consistency score obtained in Step 6 into the weighted singular value decomposition (WSVD) solution module to solve the rigid transformation matrix from the lidar point cloud to the SfM dense color point cloud.

[0017] Step 8: Apply the rigid transformation matrix obtained in Step 7 to the global LiDAR point cloud, and register the transformed global LiDAR point cloud with the SfM dense color point cloud generated in Step 3 to a unified coordinate system to obtain the multimodal 3D reconstruction result of the outdoor scene.

[0018] A further improvement in this application is that, in step 1, the lidar point cloud set is divided into a ground point cloud set and a non-ground point cloud set, specifically:

[0019] Adopt along The adaptive ground height threshold for the current frame is calculated using histogram statistics along the axis. and the first Frame LiDAR point cloud collection Divided into ground point cloud sets Non-terrestrial point cloud collection :

[0020]

[0021] in, Indicates the first Frame-by-frame lidar point cloud aggregation Indicates by the first The set of ground point clouds obtained by dividing the point cloud from a frame of lidar. Indicates by the first The set of non-ground point clouds obtained by dividing the frame lidar point cloud. Indicates the point to be judged. The height coordinates, This represents the lowest altitude value in the current frame of the lidar point cloud. Indicates the first The ground adaptive height threshold corresponding to the frame lidar point cloud.

[0022] A further improvement in this application is that step 2 specifically includes the following steps:

[0023] Step 2.1: Based on the local planar structure characteristics of the ground point cloud, introduce a point-to-plane distance term and a normal vector consistency term, and apply them to the... Frame ground point cloud collection With the Frame ground point cloud collection The rotation matrix and displacement vector between the points are solved to obtain the ground point cloud registration result, and the optimization objective is:

[0024]

[0025] in, and They represent the first Frame ground point cloud collection With the Frame ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and Let them represent the rotation matrix and translation vector to be solved during the optimization process, respectively. Indicates the first Frame ground point cloud collection The coordinates of the ground points currently involved in the registration calculation. For the first Frame ground point cloud collection coordinates of the center and ground point The corresponding matching point coordinates, For the first Frame ground point cloud collection Central ground point coordinates The normal vector, The weight coefficients for the consistency term of the normal vector. For the first Frame ground point cloud collection Matching point coordinates The normal vector, Indicates the first Frame ground point cloud collection The number of ground points involved in the optimization calculation Represents the variable to be summed. Point clouds on the ground;

[0026] Step 2.2: For non-ground point clouds, solve the first step based on point-to-plane constraints. non-ground point cloud collection With the non-ground point cloud collection The rotation matrix and translation vector between them are used to obtain the non-ground point cloud registration result, and its optimization objective is:

[0027]

[0028] in, and They represent the first non-ground point cloud collection and the non-ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and This represents the rotation matrix and translation vector to be solved during the optimization process. Indicates the first non-ground point cloud collection The coordinates of the non-ground points currently involved in the registration calculation. Indicates the first non-ground point cloud collection Zhongyu The corresponding matching point coordinates, Represents the coordinates of non-ground points The normal vector at that point, Indicates the first non-ground point cloud collection The number of non-ground points involved in the optimization calculation These are non-ground point clouds;

[0029] Step 2.3, according to the first Frame ground point cloud collection , No. Frame ground point cloud collection Registration results and the first non-ground point cloud collection , No. non-ground point cloud collection The registration results are then fused to obtain the first... Frame and the The final registration transformation between frames is performed, and based on the final registration transformation, the multi-frame lidar point cloud is registered and accumulated frame by frame to generate a global lidar point cloud. The final registration formula is:

[0030]

[0031] in, This represents a transformation solver that fuses the registration results of ground point clouds with those of non-ground point clouds.

[0032] A further improvement of this application is that step 3 specifically includes the following steps:

[0033] Step 3.1: Extract keyframes from the drone video sequence captured by the drone, and use feature detection and feature matching methods to obtain two-dimensional keypoint pairs, i.e., the first keyframes. Frame Image and the Frame Image ;

[0034] Step 3.2: Based on the relative motion relationships between images, calculate the first [image] in the UAV video sequence. Frame image and the first The essential matrix between frames :

[0035]

[0036] in, Indicates the first Frame Image and the Frame Image The relative rotation matrix between them Indicates the first Frame Image With the Frame Image The relative translation vector between them Represents the relative translation vector The antisymmetric matrix formed;

[0037] Step 3.3, based on the essential matrix The camera pose is recovered and triangulated to obtain sparse 3D point cloud coordinates. The 3D point projection relationship is as follows:

[0038]

[0039] in, The coordinates of the sparse 3D point cloud are represented at the 1st position. Frame Image The depth scale factor in and Indicates the first Frame Image Center and three-dimensional points The corresponding two-dimensional pixel coordinates, where 1 represents the homogeneous coordinate component. Represents the camera intrinsic parameter matrix. and They represent the first Frame Image The corresponding camera rotation matrix and translation vector, The first term obtained by triangulation is represented as the third term. A sparse three-dimensional point cloud coordinate system;

[0040] Step 3.4: After obtaining the coordinates of the sparse 3D point cloud, the sparse 3D point cloud is densely reconstructed using a multi-view stereo reconstruction method to generate an SfM dense color point cloud with RGB color information.

[0041] A further improvement in this application is that step 4 specifically includes the following steps:

[0042] Step 4.1, let the global lidar point cloud be... SfM dense colored point cloud is , and These represent global lidar points and SfM dense color points in Euclidean space, respectively. This represents the total number of point clouds in the global LiDAR system. To determine the number of SfM dense color point clouds, the original 3D coordinates of the global LiDAR point cloud and the SfM dense color point cloud are embedded into a one-dimensional convolutional layer. In the initial feature space, the initial feature representations of the lidar point cloud and the initial feature representation of the SfM point cloud after initial encoding are obtained:

[0043]

[0044] in, This represents the dimension of the initial feature space. This represents the initial feature representation of the global lidar point cloud after initial encoding. This is the initial feature representation of the SfM dense color point cloud after initial encoding, with superscript indicating the initial feature representation. This represents the output of layer 0 in the shared feature encoder of the TGSC-Attn feature generation module;

[0045] Step 4.2: Initial feature representation of lidar point cloud. Initial feature representation of SfM point cloud Multiple self-attention layers are applied to update the initial feature representation of the lidar point cloud layer by layer. Initial feature representation of SfM point cloud ;

[0046]

[0047] in, This represents the total number of self-attention layers. This indicates the layer index of the current self-attention layer. and These represent the lidar point cloud feature representation and the SfM point cloud feature representation output from the previous layer, respectively.

[0048] Step 4.3: After the self-attention layer, intermodal information exchange is performed through the cross-attention layer:

[0049]

[0050]

[0051] in, This indicates the total number of cross-attention layers; the output of the last layer represents the deep features of the global LiDAR point cloud. Deep features of SfM dense colored point clouds .

[0052] A further improvement in this application is that: in step 5, the deep features of the global lidar point cloud and the deep features of the SfM dense color point cloud are adversarially aligned using the GAN-FeaAlign feature alignment module, and the objective function is:

[0053]

[0054] in, This refers to the shared feature encoder in the TGSC-Attn feature generation module. This represents the domain discriminator in the GAN-FeaAlign feature alignment module. This represents the input sample from the global LiDAR point cloud. This represents an input sample from a dense colored point cloud (SfM).

[0055] A further improvement in this application is that step 6 specifically includes the following steps:

[0056] Step 6.1: Based on the shared latent feature space obtained in Step 5, let the global LiDAR point cloud aligned feature set obtained after adversarial alignment processing by the TGSC-Attn feature generation module and the GAN-FeaAlign feature alignment module be (the set of features to be defined in the original text). The SfM dense color point cloud alignment feature set is Calculate the similarity score between the global lidar point cloud alignment feature set and the SfM dense color point cloud alignment feature set. And form a similarity matrix :

[0057]

[0058]

[0059] in, Represents the alignment feature vector of LiDAR point cloud Align feature vectors with SfM dense color point cloud The inner product, The norm of the aligned feature vectors of the lidar point cloud is represented. Let be the norm of the aligned feature vectors of the SfM dense color point cloud. This indicates the total number of point clouds in the global LiDAR system. Indicates the number of dense colored point clouds in SfM. , ;

[0060] Step 6.2: Based on the similarity matrix For each global lidar point Selecting from SfM dense color point cloud The candidate matching points with the highest similarity are the Top-k candidate points, and a coarse matching point pair set is obtained by applying bidirectional mutual consistency constraints. :

[0061]

[0062] in, Represented as the first in the global lidar point cloud One point, Represents the first in the SfM dense colored point cloud One point, Indicates the connection with the global lidar point The most similar A set of SfM candidate points This indicates the dense colored points of the SfM. The most similar A set of candidate points for lidar;

[0063] Step 6.3: For the coarse matching point pair set Candidate point pairs By comparing the first point in the global lidar point cloud Points With SfM dense colored dots Calculate the consistency score based on the local neighborhood geometry consistency in each point cloud, and then apply the consistency score threshold. Filter to obtain a set of fine matching point pairs The formula for calculating the consistency score is:

[0064]

[0065] in, and These represent the first and second points in the global lidar point cloud, respectively. Points With SfM dense colored dots The local neighborhood, Represents the first point in the global lidar point cloud. Points Its neighboring points The Euclidean distance between them Represents SfM dense colored dots Its neighboring points The Euclidean distance between them Indicates the scale parameter.

[0066] A further improvement in this application is that step 7 specifically includes the following steps:

[0067] Step 7.1: Filter the set of fine matching point pairs obtained in Step 6.3. and corresponding consistency score As weights, construct the rigid transformation optimization objective:

[0068]

[0069] in, Represents the rotation matrix. Represents the translation vector;

[0070] Step 7.2: Based on the set of fine-matching points The consistency score of each point pair in the set will be used to refine the set of matching point pairs. The endpoints of the lidar in the set constitute the lidar matching point set. The endpoints of the fine matching point pair set are then combined to form the SfM matching point set. The weighted centroids of the lidar matching point set and the SfM matching point set are calculated respectively.

[0071]

[0072] in, This represents the weighted centroid of the LiDAR matching point set in the fine matching point pair. This represents the weighted centroid of the SfM matching point set in a fine-matching point pair;

[0073] Step 7.3: Set the fine-matching point pairs Each matching point is centered relative to the weighted centroid of its respective LiDAR matching point set and the weighted centroid of the SfM matching point set, and a weighted cross-covariance matrix is ​​constructed. :

[0074]

[0075] Step 7.4: Calculate the weighted cross-covariance matrix. Perform singular value decomposition And calculate the optimal rotation matrix. Translation vector :

[0076]

[0077] in, This represents the orthogonal matrix of left singular vectors obtained by performing singular value decomposition on the weighted cross-covariance matrix. This represents the right singular vector orthogonal matrix obtained by performing singular value decomposition on the weighted cross-covariance matrix. Represents a singular value diagonal matrix. For matrix transpose, and Together they are used to calculate the optimal rotation matrix. And based on the rotation matrix Translation vector Constructing a rigid transformation matrix ;

[0078] Step 7.5, when When, the orthogonal matrix of the right singular vectors obtained by singular value decomposition is... The last column is corrected by sign flipping to ensure that it is used to construct the rigid transformation matrix. rotation matrix For a valid rotation matrix, where Represents a determinant.

[0079] A further improvement in this application is that step 8 specifically includes the following steps:

[0080] Step 8.1: Apply the rigid transformation matrix obtained in Step 7. Application to global lidar point cloud The transformed lidar point cloud is obtained from each point in the center. :

[0081]

[0082] in, This represents the original global lidar point cloud. This represents the lidar point cloud after rigid transformation. Represented by rotation matrix Translation vector The rigid transformation matrix formed;

[0083] Step 8.2: Transform the lidar point cloud SfM Dense Colored Point Clouds By registering to a unified coordinate system, the lidar point cloud and the SfM dense color point cloud are aligned and fused to obtain a fused point cloud:

[0084]

[0085] Step 8.3: Based on the fused point cloud Generate multimodal 3D reconstruction results for outdoor scenes.

[0086] This application provides a cross-source point cloud registration system based on adversarial neural networks. The system includes a lidar point cloud registration module, an SfM point cloud generation module, and a cross-modal point cloud alignment module.

[0087] The lidar point cloud registration module is used to segment the input multi-frame lidar point cloud sequence into ground point cloud and non-ground point cloud, and register them separately to generate a global lidar point cloud.

[0088] The SfM point cloud generation module is used to recover the camera pose and generate SfM dense color point clouds based on the UAV video sequence collected by the UAV.

[0089] The cross-modal point cloud alignment module uses the cross-source point cloud registration network MMAlignNet to perform cross-modal feature alignment and rigid registration between the global lidar point cloud and the SfM dense color point cloud;

[0090] The TGSC-Attn feature generation module performs multi-scale feature encoding on the global LiDAR point cloud and SfM dense color point cloud through one-dimensional convolutional layers, self-attention layers, and cross-attention layers to extract deep features from the global LiDAR point cloud. Deep features of SfM dense colored point clouds , among which, the The output of the self-attention layer is a feature representation of the LiDAR point cloud. SfM point cloud feature representation And through cross-attention layers in lidar point cloud feature representation The system integrates intramodal structural information and intermodal correlation information.

[0091] The GAN-FeaAlign feature alignment module uses a domain discriminator to analyze the deep features of the global LiDAR point cloud. Deep features of dense colored point clouds with SfM Adversarial learning is employed to reduce the feature distribution differences between lidar point clouds and SfM dense color point clouds, thereby achieving alignment of the two modal features in a shared latent feature space.

[0092] The RCF-RobustMatch robust matching module is used to perform coarse matching filtering based on the similarity matrix and bidirectional mutual consistency constraints, and to perform fine matching filtering by combining local neighborhood geometric consistency, so as to obtain a set of fine matching point pairs.

[0093] The weighted singular value decomposition (WSVD) solution module calculates the rotation matrix and translation vector between the lidar point cloud and the SfM dense color point cloud based on the set of fine matching point pairs and the consistency score corresponding to each fine matching point pair, thus completing the cross-modal rigid transformation estimation.

[0094] The beneficial effects of this application are:

[0095] This application models and registers ground point clouds and non-ground point clouds separately, making full use of the geometric features of different scene structures, effectively reducing the cumulative error and drift phenomenon in the multi-frame lidar point cloud registration process, and improving the stability and accuracy of large scene point cloud registration.

[0096] This application integrates ground-based lidar point clouds and UAV SfM dense color point clouds, achieving complementarity between geometric and texture information, and obtaining more complete and accurate 3D reconstruction results of outdoor scenes.

[0097] This application effectively reduces the feature differences between cross-modal point clouds by combining TGSC-Attn feature generation, GAN-FeaAlign adversarial alignment, and RCF-RobustMatch robust matching, thereby improving the reliability of the correspondence between lidar point clouds and SfM point clouds and the accuracy of multimodal registration.

[0098] This application adopts a WSVD rigid transformation solution method based on matching confidence weighting, which effectively enhances the robustness of rigid transformation estimation and is especially suitable for high-precision fusion reconstruction of multi-source heterogeneous point clouds in complex outdoor environments. Attached Figure Description

[0099] Figure 1 This is a flowchart of the method used in this application.

[0100] Figure 2 This is a schematic diagram of the MMAlignNet cross-source point cloud registration network proposed in this application.

[0101] Figure 3 This is a sample image of a representative outdoor scene from a multimodal sensor acquisition platform and the MMPC dataset.

[0102] Figure 4 The image shows the scene reconstruction results obtained by frame-by-frame registration and accumulation of single-frame and multi-frame LiDAR point clouds from different perspectives in the MMPC-LiDAR point cloud dataset.

[0103] Figure 5 This is a collection of multiple video frames from the MMPC-Color color point cloud dataset and SfM dense color point cloud maps reconstructed from SfM.

[0104] Figure 6 A visualization comparing different registration methods on the MMPC dataset.

[0105] Figure 7 A 3D surface mesh image of the point cloud reconstructed by fusing UAV imagery with LiDAR. Detailed Implementation

[0106] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0107] like Figure 1 and Figure 2 As shown, this application discloses a cross-source point cloud registration method based on adversarial neural networks. This method is implemented through a cross-source point cloud registration system, which includes a LiDAR point cloud registration module, a SfM point cloud generation module, and a cross-modal point cloud alignment module. The LiDAR point cloud registration module segments the input multi-frame LiDAR point cloud sequence into ground and non-ground point clouds, and registers them separately to generate a global LiDAR point cloud. The SfM point cloud generation module recovers the camera pose from UAV video sequences collected by the UAV and generates an SfM dense color point cloud. The cross-modal point cloud alignment module uses the cross-source point cloud registration network MMAlignNet to perform cross-modal feature alignment and rigid registration between the global LiDAR point cloud and the SfM dense color point cloud, thereby achieving accurate alignment and fusion reconstruction of the two heterogeneous point clouds.

[0108] The TGSC-Attn feature generation module performs multi-scale feature encoding on the global LiDAR point cloud and SfM dense color point cloud through one-dimensional convolutional layers, self-attention layers, and cross-attention layers to extract deep features from the global LiDAR point cloud. Deep features of SfM dense colored point clouds , among which, the The output of the self-attention layer is a feature representation of the LiDAR point cloud. SfM point cloud feature representation And through cross-attention layers in lidar point cloud feature representation The system integrates intramodal structural information and intermodal correlation information.

[0109] The GAN-FeaAlign feature alignment module uses a domain discriminator to analyze the deep features of the global LiDAR point cloud. Deep features of dense colored point clouds with SfM Adversarial learning is employed to reduce the feature distribution differences between lidar point clouds and SfM dense color point clouds, thereby achieving alignment of the two modal features in a shared latent feature space.

[0110] The RCF-RobustMatch robust matching module is used to perform coarse matching screening based on the similarity matrix and bidirectional mutual consistency constraints, and to perform fine matching filtering based on local neighborhood geometric consistency, so as to obtain a set of fine matching point pairs; the weighted singular value decomposition (WSVD) solving module calculates the rotation matrix and translation vector between the lidar point cloud and the SfM dense color point cloud based on the set of fine matching point pairs and the consistency score corresponding to each fine matching point pair, and completes the cross-modal rigid transformation estimation.

[0111] like Figure 1 As shown, this application presents a cross-source point cloud registration method based on adversarial neural networks, which specifically includes the following steps:

[0112] Step 1: Input a multi-frame LiDAR point cloud sequence collected by an Unmanned Ground Vehicle (UGV) and a UAV video sequence collected by an Unmanned Aerial Vehicle (UAV). Segment the multi-frame LiDAR point cloud sequence into ground point clouds and non-ground point clouds. In other words, the UGV continuously collects multi-frame LiDAR point cloud sequences along its path to the outdoor scene to be reconstructed, while the UAV performs aerial video acquisition of the same scene, resulting in a video sequence. The multi-frame LiDAR point cloud sequence is used for scene geometry reconstruction, while the video sequence serves as input data for subsequent SfM point cloud generation.

[0113] To improve the stability and accuracy of subsequent lidar point cloud registration, each lidar point cloud frame is divided into ground point cloud and non-ground point cloud. Specifically: for the first frame... Frame point cloud Adopting along The adaptive ground height threshold for the current frame is calculated using histogram statistics along the axis. and the first Frame LiDAR point cloud collection Divided into ground point cloud sets Non-terrestrial point cloud collection :

[0114]

[0115] in, Indicates the first Frame-by-frame lidar point cloud aggregation Indicates by the first The set of ground point clouds obtained by dividing the point cloud from a frame of lidar. Indicates by the first The set of non-ground point clouds obtained by dividing the frame lidar point cloud. Indicates the point to be judged. The height coordinates, This represents the lowest altitude value in the current frame of the lidar point cloud. Indicates the first The adaptive ground height threshold corresponding to the frame lidar point cloud. Using the above method, point clouds with continuous planar features, such as roads and ground, are separated from non-ground structures such as building facades, vegetation, and road appurtenances, providing a foundation for subsequent accurate registration using the different structural characteristics of ground and non-ground areas.

[0116] Step 2: Employ a dual-branch ICP registration strategy (Dual-ICP) to register the ground point cloud set and the non-ground point cloud set separately, and perform frame-by-frame registration and accumulation of multi-frame LiDAR point clouds to generate a global LiDAR point cloud. This specifically includes the following steps:

[0117] Step 2.1: Establish an optimization objective for the ground point cloud that includes both a point-to-plane distance term and a normal vector consistency term, in order to solve the... Frame ground point cloud collection With the Frame ground point cloud collection Rotation matrix between Translation vector The ground point cloud registration results are obtained, and the optimization objective is:

[0118]

[0119] in, and They represent the first Frame ground point cloud collection With the Frame ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and Let them represent the rotation matrix and translation vector to be solved during the optimization process, respectively. Indicates the first Frame ground point cloud collection The coordinates of the ground points currently involved in the registration calculation. For the first Frame ground point cloud collection coordinates of the center and ground point The corresponding matching point coordinates, For the first Frame ground point cloud collection Central ground point coordinates The normal vector, The weight coefficients for the consistency term of the normal vector. For the first Frame ground point cloud collection Matching point coordinates The normal vector, Indicates the first Frame ground point cloud collection The number of ground points involved in the optimization calculation Represents the variable to be summed. For ground point clouds; by introducing a normal vector consistency term, the structural stability of the ground plane region can be fully utilized to suppress low-frequency drift errors caused by carrier motion.

[0120] Step 2.2: For non-ground point clouds, construct optimization objectives using point-to-plane constraints to solve the first... non-ground point cloud collection With the non-ground point cloud collection Rotation matrix between Translation vector The non-ground point cloud registration results are obtained, and the optimization objective is:

[0121]

[0122] in, and They represent the first non-ground point cloud collection and the non-ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and This represents the rotation matrix and translation vector to be solved during the optimization process. Indicates the first non-ground point cloud collection The coordinates of the non-ground points currently involved in the registration calculation. Indicates the first non-ground point cloud collection Zhongyu The corresponding matching point coordinates, Represents the coordinates of non-ground points The normal vector at that point, Indicates the first non-ground point cloud collection The number of non-ground points involved in the optimization calculation These are non-ground point clouds;

[0123] Step 2.3, according to the first Frame ground point cloud collection , No. Frame ground point cloud collection Registration results and the first non-ground point cloud collection , No. non-ground point cloud collection The registration results are then fused to obtain the first... Frame and the The final registration transformation between frames is performed, and based on the final registration transformation, the multi-frame lidar point cloud is registered and accumulated frame by frame to generate a global lidar point cloud. The final registration formula is:

[0124]

[0125] in, This represents a transformation solver that fuses the registration results of ground point clouds with those of non-ground point clouds.

[0126] By utilizing the Dual-ICP algorithm described above, the complementary structural features of the ground and non-ground regions are fully leveraged. This allows the ground region to provide large-scale planar constraints, while the non-ground region provides rich local geometric constraints. This reduces the cumulative drift error caused by carrier motion and improves the overall registration accuracy of multi-frame lidar point clouds.

[0127] Step 3: Perform Structure of Motion (SfM) reconstruction on the UAV video sequence acquired by the UAV to restore the camera attitude and generate an SfM dense color point cloud. Step 3 specifically includes the following steps:

[0128] Step 3.1: Extract keyframes from the drone video sequence captured by the drone, and use feature detection and feature matching methods to obtain two-dimensional keypoint pairs, i.e., the first keyframes. Frame Image and the Frame Image Specifically, acquiring video sequences of the target scene captured by the drone. Keyframes are extracted from the video sequence, and feature detection and feature matching are performed on adjacent keyframes or keyframes with overlapping fields of view to obtain two-dimensional keypoint pairs between different images. These pairs are used to characterize the pixel position correspondence of the same spatial point in different keyframes, providing constraints for subsequent camera pose recovery and three-dimensional point triangulation.

[0129] Step 3.2: Based on the relative motion relationships between images, calculate the first [image] in the UAV video sequence. Frame image and the first The essential matrix between frames :

[0130]

[0131] in, Indicates the first Frame Image and the Frame Image The relative rotation matrix between them Indicates the first Frame Image With the Frame Image The relative translation vector between them Represents the relative translation vector The constructed antisymmetric matrix; according to the essential matrix Decompose and recover the relative camera pose between adjacent keyframes ;

[0132] Step 3.3, based on the essential matrix The camera pose is recovered and triangulated to obtain sparse 3D point cloud coordinates. The 3D point projection relationship is as follows:

[0133]

[0134] in, The coordinates of the sparse 3D point cloud are represented at the 1st position. Frame Image The depth scale factor in and Indicates the first Frame Image Center and three-dimensional points The corresponding two-dimensional pixel coordinates, where 1 represents the homogeneous coordinate component. Represents the camera intrinsic parameter matrix. and They represent the first Frame Image The corresponding camera rotation matrix and translation vector, The first term obtained by triangulation is represented as the third term. A sparse three-dimensional point cloud coordinate system;

[0135] Step 3.4: After obtaining the coordinates of the sparse 3D point cloud, a multi-view stereo reconstruction method is used to perform dense reconstruction on the sparse 3D point cloud, generating a dense color point cloud with RGB color information (SfM). The point cloud includes geometric structure information of the scene and relatively rich texture and color information, forming a complementary viewpoint with the LiDAR point cloud, providing an aerial viewpoint data foundation for subsequent cross-modal point cloud registration.

[0136] Step 4: Input the global LiDAR point cloud accumulated in Step 2 and the SfM dense color point cloud generated in Step 3 into the cross-source point cloud registration network MMAlignNet. The MMAlignNet network includes a TGSC-Attn feature generation module, a GAN-FeaAlign feature alignment module, an RCF-RobustMatch robust matching module, and a weighted singular value decomposition (WSVD) solving module. The TGSC-Attn feature generation module extracts deep features from the global LiDAR point cloud and the SfM dense color point cloud. Specifically, this includes the following steps:

[0137] Step 4.1, let the global lidar point cloud be... SfM dense colored point cloud is , and These represent global lidar points and SfM dense color points in Euclidean space, respectively. This represents the total number of point clouds in the global LiDAR system. To determine the number of SfM dense color point clouds, the original 3D coordinates of the global LiDAR point cloud and the SfM dense color point cloud are embedded into a one-dimensional convolutional layer. In the initial feature space, the initial feature representations of the lidar point cloud and the initial feature representation of the SfM point cloud after initial encoding are obtained:

[0138]

[0139] in, This represents the dimension of the initial feature space. This represents the initial feature representation of the global lidar point cloud after initial encoding. This is the initial feature representation of the SfM dense color point cloud after initial encoding, with superscript indicating the initial feature representation. This represents the output of layer 0 in the shared feature encoder of the TGSC-Attn feature generation module;

[0140] Step 4.2: Initial feature representation of lidar point cloud. Initial feature representation of SfM point cloud Multiple self-attention layers are applied to update the initial feature representation of the lidar point cloud layer by layer. Initial feature representation of SfM point cloud ;

[0141]

[0142] in, This represents the total number of self-attention layers. This indicates the layer index of the current self-attention layer. and These represent the lidar point cloud feature representation and SfM point cloud feature representation output from the previous layer, respectively. By using a self-attention mechanism, long-distance dependencies are established within the same modality of the network, enhancing the expressive ability of the local geometric structure and global spatial distribution of the point cloud, and obtaining the intra-modal features of the lidar point cloud and the SfM dense color point cloud, respectively.

[0143] Step 4.3: After the self-attention layer, intermodal information exchange is performed through the cross-attention layer:

[0144]

[0145]

[0146] in, This indicates the total number of cross-attention layers; the output of the last layer represents the deep features of the global LiDAR point cloud. Deep features of SfM dense colored point clouds .

[0147] Through the cross-attention mechanism, the LiDAR point cloud features not only utilize their own modal information when updating, but also simultaneously introduce SfM dense color point cloud features; similarly, the SfM dense color point cloud features also introduce LiDAR point cloud features when updating. The above method can obtain deep feature representations that simultaneously contain intramodal structural information and intermodal correlation information, providing more discriminative input features for subsequent feature alignment and matching.

[0148] Step 5: Adversarially align the deep features of the global LiDAR point cloud with the deep features of the SfM dense color point cloud using the GAN-FeaAlign feature alignment module to construct a shared latent feature space. The objective function for adversarial alignment of the deep features of the global LiDAR point cloud with the SfM dense color point cloud using the GAN-FeaAlign feature alignment module is:

[0149]

[0150] in, This refers to the shared feature encoder in the TGSC-Attn feature generation module. This represents the domain discriminator in the GAN-FeaAlign feature alignment module. This represents the input sample from the global LiDAR point cloud. This represents the input sample from a dense color point cloud (SfM). Fixed feature encoder. Domain discriminator Training is performed to enable the domain discriminator to distinguish the source of input features, thereby learning the distribution differences of the two modalities in the feature space. Then, the domain discriminator is fixed and the feature encoder is updated in reverse, making it difficult for the domain discriminator to distinguish the modal origin of the point cloud features extracted by the feature encoder, thereby reducing the domain bias between different modalities, until the preset number of training rounds or the loss function converges.

[0151] Step 6: Calculate the similarity matrix between cross-modal point clouds using the RCF-RobustMatch robust matching module. Based on the similarity matrix and bidirectional mutual consistency constraints, obtain the set of coarse-matched corresponding point pairs. Then, combine this with local neighborhood geometric consistency filtering to obtain the set of fine-matched point pairs. Specifically, this includes the following steps:

[0152] Step 6.1: Based on the shared latent feature space obtained in Step 5, let the global LiDAR point cloud aligned feature set obtained after adversarial alignment processing by the TGSC-Attn feature generation module and the GAN-FeaAlign feature alignment module be (the set of features to be defined in the original text). The SfM dense color point cloud alignment feature set is Calculate the similarity score between the global lidar point cloud alignment feature set and the SfM dense color point cloud alignment feature set. And form a similarity matrix :

[0153]

[0154]

[0155] in, Represents the alignment feature vector of LiDAR point cloud Align feature vectors with SfM dense color point cloud The inner product, The norm of the aligned feature vectors of the lidar point cloud is represented. Let be the norm of the aligned feature vectors of the SfM dense color point cloud. This indicates the total number of point clouds in the global LiDAR system. Indicates the number of dense colored point clouds in SfM. , Through the similarity matrix It can measure the similarity between any two points in the feature space of a lidar point cloud and an SfM dense color point cloud.

[0156] Step 6.2: Based on the similarity matrix For each global lidar point Selecting from SfM dense color point cloud The candidate matching points with the highest similarity are the Top-k candidate points, and a coarse matching point pair set is obtained by applying bidirectional mutual consistency constraints. :

[0157]

[0158] in, Represented as the first in the global lidar point cloud One point, Represents the first in the SfM dense colored point cloud One point, Indicates the connection with the global lidar point The most similar A set of SfM candidate points This indicates the dense colored points of the SfM. The most similar A set of candidate points for lidar;

[0159] Step 6.3: For the coarse matching point pair set Candidate point pairs By comparing the first point in the global lidar point cloud Points With SfM dense colored dots Calculate the consistency score based on the local neighborhood geometry consistency in each point cloud, and then apply the consistency score threshold. Filter to obtain a set of fine matching point pairs The formula for calculating the consistency score is:

[0160]

[0161] in, and These represent the first and second points in the global lidar point cloud, respectively. Points With SfM dense colored dots The local neighborhood, Represents the first point in the global lidar point cloud. Points Its neighboring points The Euclidean distance between them Represents SfM dense colored dots Its neighboring points The Euclidean distance between them The scale parameter is used to represent the consistency of the geometric structure of candidate matching point pairs in their respective neighborhoods. Candidate matching point pairs with consistency scores below a preset threshold are eliminated, ultimately resulting in a set of fine-grained matching point pairs. This serves as a reliable input for subsequent rigid transformation solutions.

[0162] Step 7: Input the set of fine-matched point pairs and their consistency scores obtained in Step 6 into the Weighted Singular Value Decomposition (WSVD) solution module to solve for the rigid transformation matrix from the lidar point cloud to the SfM dense color point cloud. Specifically, this includes the following steps:

[0163] Step 7.1: Filter the set of fine matching point pairs obtained in Step 6.3. and corresponding consistency score As weights, construct the rigid transformation optimization objective:

[0164]

[0165] in, Represents the rotation matrix. The translation vector is represented by the above objective function. The matching point pairs with higher confidence are given greater weight in the solution of rigid transformation, thereby enhancing the robustness of cross-modal point cloud registration in complex outdoor scenes.

[0166] Step 7.2: Based on the set of fine-matching points The consistency score of each point pair in the set will be used to refine the set of matching point pairs. The endpoints of the lidar in the set constitute the lidar matching point set. The endpoints of the fine matching point pair set are then combined to form the SfM matching point set. The weighted centroids of the lidar matching point set and the SfM matching point set are calculated respectively.

[0167]

[0168] in, This represents the weighted centroid of the LiDAR matching point set in the fine matching point pair. This represents the weighted centroid of the SfM matching point set in the fine matching point pair; the weighted centroid is used to reduce the interference of low-confidence matching point pairs on the overall transformation estimation result.

[0169] Step 7.3: Set the fine-matching point pairs Each matching point is centered relative to the weighted centroid of its respective LiDAR matching point set and the weighted centroid of the SfM matching point set, and a weighted cross-covariance matrix is ​​constructed. :

[0170]

[0171] Step 7.4: Calculate the weighted cross-covariance matrix. Perform singular value decomposition And calculate the optimal rotation matrix. Translation vector :

[0172]

[0173] in, This represents the orthogonal matrix of left singular vectors obtained by performing singular value decomposition on the weighted cross-covariance matrix. This represents the right singular vector orthogonal matrix obtained by performing singular value decomposition on the weighted cross-covariance matrix. Represents a singular value diagonal matrix. For matrix transpose, and Together they are used to calculate the optimal rotation matrix. And based on the rotation matrix Translation vector Constructing a rigid transformation matrix :

[0174]

[0175] Step 7.5, when When, the orthogonal matrix of the right singular vectors obtained by singular value decomposition is... The last column is then corrected by reversing the sign, resulting in the corrected matrix. ,in , and Represent matrices respectively The vectors in columns 1, 2, and 3; then the rotation matrix is ​​recalculated. ,make sure ,make It is a legal rigid rotation matrix.

[0176] Step 8: Apply the rigid transformation matrix obtained in Step 7 to the global LiDAR point cloud, and register the transformed global LiDAR point cloud with the SfM dense color point cloud generated in Step 3 to a unified coordinate system to obtain the multimodal 3D reconstruction result of the outdoor scene. Specifically, this includes the following steps:

[0177] Step 8.1: Apply the rigid transformation matrix obtained in Step 7. Application to global lidar point cloud The transformed lidar point cloud is obtained from each point in the center. :

[0178]

[0179] in, This represents the original global lidar point cloud. This represents the lidar point cloud after rigid transformation. Represented by rotation matrix Translation vector The rigid transformation matrix formed;

[0180] Step 8.2: Transform the lidar point cloud SfM Dense Colored Point Clouds By registering to a unified coordinate system, precise alignment and fusion of the lidar point cloud and the SfM dense color point cloud are achieved, resulting in a fused point cloud:

[0181]

[0182] Step 8.3: Based on the fused point cloud Generate multimodal 3D reconstruction results for outdoor scenes.

[0183] In this implementation, the MMAlignNet network corresponding to steps 4 to 7 is optimized using a joint loss function during the training phase. This joint loss function is used to jointly constrain the cross-modal feature generation, adversarial feature alignment, robust matching, and rigid transformation estimation processes. Specifically, feature consistency loss and adversarial loss are mainly used to constrain the cross-modal feature generation and feature alignment processes in steps 4 and 5, while geometric consistency loss is mainly used to constrain the corresponding point selection and rigid transformation estimation processes in steps 6 and 7. The total loss function is used for end-to-end training optimization of the entire MMAlignNet network.

[0184] The joint loss function includes geometric consistency loss, feature consistency loss, adversarial loss, and total loss function, the expressions of which are as follows:

[0185]

[0186]

[0187]

[0188]

[0189] in, For geometric consistency loss, For feature consistency loss, To combat the losses, For the total loss, , and These represent the weighting coefficients of each loss term.

[0190] This application quantitatively analyzes the experimental results of fusing mobile robot laser point clouds and drone video reconstructed dense color point clouds by measuring the accuracy of multi-source point cloud registration. This application constructs the MMPC multimodal point cloud dataset, which includes six different outdoor scenes from the Nanjing University of Posts and Telecommunications campus, and includes both MMPC-LiDAR and MMPC-Color point cloud datasets. The data acquisition platform consists of a mobile robot equipped with a 32-line rotating LiDAR and a DJI Flip drone equipped with a 4K high-definition camera. The acquisition platform and representative scenes are shown below. Figure 3 As shown. Figure 3 (a) in the image represents an unmanned ground vehicle data collection platform equipped with lidar. Figure 3 (b) in the image represents a drone data collection platform equipped with a camera. Figure 3 (c) in the figure represents six representative outdoor scene samples from the MMPC dataset. For LiDAR data, this application performs voxel downsampling on the raw point clouds continuously acquired for each scene before inputting them into the registration network. The average number of points in the raw point clouds for the six scenes is approximately 1,196,915, and the average number of points after compression is approximately 9,878, with an average frame count of approximately 1,220. After frame-by-frame registration and accumulation of the multi-frame LiDAR point clouds based on the Dual-ICP algorithm, as shown... Figure 4 Reconstruction results for the corresponding scenes can be obtained. For the color point cloud obtained from UAV video reconstruction, the average video acquisition time for the six scenes is approximately 39 seconds. The average number of points in the original color point cloud is approximately 792,926, and the average number of points after compression is approximately 9,686. The dense color point cloud generated using the SfM algorithm is shown below. Figure 5 As shown, the upper structure of the scene is relatively complete, while the lower region has some missing parts. Therefore, further precise registration and fusion with the LiDAR point cloud are required. The point cloud attributes of the MMPC-LiDAR point cloud dataset are shown in Table 1, where Parameter represents the parameters.

[0191] Table 1

[0192]

[0193] The point cloud attributes of the MMPC-Color color point cloud dataset are shown in Table 2.

[0194] Table 2

[0195]

[0196] To verify the effectiveness of the proposed cross-source point cloud registration network MMAlignNet, this application selected several classic point cloud registration algorithms for comparative experiments, including point-to-point ICP, point-to-line ICP, point-to-area ICP, RolTr, GeoTransformer, CoFiNet, ROP, and FMR. Rotation error (R) and translation error (t) were used as evaluation metrics. The experimental results are shown in Table 3, where Scene represents the scene.

[0197] Table 3

[0198]

[0199] The experimental results show that the average rotation error of this application in the six scenes is 0.879, and the average translation error is 0.140, both significantly better than the comparison methods. Especially in environments with more complex geometry, such as scenes 3, 4, and 5, this application still maintains smaller rotation and translation deviations. Visual comparison results of different registration methods are shown below. Figure 6 Subsequently, this application performs surface meshing processing on the reconstructed point cloud after registration and fusion, and the resulting 3D surface mesh is as follows. Figure 7 As shown, this result demonstrates that the present application can maintain good structural continuity and geometric consistency in environmental elements such as buildings, ground, and vegetation.

[0200] This application further conducted ablation experiments to investigate the effects of TGSC-Attn, GAN-FeaAlign, RCF-RobustMatch, and WSVD on the overall performance. The experimental results are shown in Table 4.

[0201] Table 4

[0202]

[0203] When using only the TGSC-Attn encoder for feature extraction, the average rotation and translation errors are 7.333 and 0.751, respectively. After introducing GAN-FeaAlign or WSVD, the registration error further decreases. When RCF-RobustMatch, GAN-FeaAlign, and WSVD are used in conjunction with TGSC-Attn, the average rotation and translation errors can be reduced to 1.125 and 0.195, and 1.113 and 0.192, respectively. When all four modules are used together, the average rotation and translation errors are further reduced to 0.879 and 0.140, respectively, indicating that the modules have a good synergistic effect and can jointly improve the accuracy and robustness of multi-source point cloud feature alignment, corresponding point selection, and rigid body transformation estimation.

[0204] This application enables high-precision registration and fusion of LiDAR point clouds collected by mobile robots and color point clouds reconstructed from UAV videos. By introducing a joint modeling mechanism of self-attention and cross-attention, an adversarial feature alignment strategy, a reliable corresponding point selection mechanism, and a weighted SVD pose solution method, it can effectively reduce the impact of modal differences, viewpoint differences, and point cloud density differences, improve the accuracy and stability of multi-source point cloud registration in complex outdoor scenes, and provide a more complete and consistent 3D scene representation for subsequent 3D reconstruction, scene understanding, and remote sensing mapping applications.

[0205] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A cross-source point cloud registration method based on adversarial neural networks, characterized in that: The cross-source point cloud registration method specifically includes the following steps: Step 1: Input a multi-frame lidar point cloud sequence collected by an unmanned ground vehicle (UGV) and a drone video sequence collected by an unmanned aerial vehicle (UAV). Divide the multi-frame lidar point cloud sequence into a ground point cloud set and a non-ground point cloud set. Step 2: Use the ground-non-ground dual-branch ICP registration strategy to register the ground point cloud set and the non-ground point cloud set respectively, and perform frame-by-frame registration and accumulation of each frame of the LiDAR point cloud in the multi-frame LiDAR point cloud sequence to generate the global LiDAR point cloud. Step 3: Perform motion recovery structure reconstruction on the UAV video sequence acquired by the UAV to restore the camera attitude and generate SfM dense color point cloud; Step 4: Input the global LiDAR point cloud generated in Step 2 and the SfM dense color point cloud generated in Step 3 into the cross-source point cloud registration network MMAlignNet. The cross-source point cloud registration network MMAlignNet includes a TGSC-Attn feature generation module, a GAN-FeaAlign feature alignment module, an RCF-RobustMatch robust matching module, and a weighted singular value decomposition (WSVD) solving module. The deep features of the global LiDAR point cloud and the SfM dense color point cloud are extracted through the TGSC-Attn feature generation module. Step 5: Use the GAN-FeaAlign feature alignment module to perform adversarial alignment between the deep features of the global lidar point cloud and the deep features of the SfM dense color point cloud to construct a shared latent feature space. Step 6: Calculate the similarity matrix between cross-modal point clouds using the RCF-RobustMatch robust matching module, and obtain the set of coarse matching corresponding point pairs based on the similarity matrix and bidirectional mutual consistency constraints. Then, combine the local neighborhood geometric consistency filtering to obtain the set of fine matching point pairs. Step 7: Input the set of fine matching point pairs and its consistency score obtained in Step 6 into the weighted singular value decomposition (WSVD) solution module to solve the rigid transformation matrix from the lidar point cloud to the SfM dense color point cloud. Step 8: Apply the rigid transformation matrix obtained in Step 7 to the global LiDAR point cloud, and register the transformed global LiDAR point cloud with the SfM dense color point cloud generated in Step 3 to a unified coordinate system to obtain the multimodal 3D reconstruction result of the outdoor scene.

2. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: In step 1, the process of segmenting each frame of the LiDAR point cloud into ground point cloud and non-ground point cloud is as follows: Adopt along The adaptive ground height threshold for the current frame is calculated using histogram statistics along the axis. and the first Frame LiDAR point cloud collection Divided into ground point cloud sets Non-terrestrial point cloud collection : in, Indicates the first Frame-by-frame lidar point cloud aggregation Indicates by the first The set of ground point clouds obtained by dividing the point cloud from a frame lidar radar. Indicates by the first The set of non-ground point clouds obtained by dividing the frame lidar point cloud. Indicates the point to be judged. The height coordinates, This represents the lowest altitude value in the current frame of the LiDAR point cloud set. Indicates the first The ground adaptive height threshold corresponding to the frame lidar point cloud.

3. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Based on the local planar structure characteristics of the ground point cloud, introduce a point-to-plane distance term and a normal vector consistency term, and apply them to the... Frame ground point cloud collection With the Frame ground point cloud collection The rotation matrix and displacement vector between the points are solved to obtain the ground point cloud registration result, and the optimization objective is: in, and They represent the first Frame ground point cloud collection With the Frame ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and Let them represent the rotation matrix and translation vector to be solved during the optimization process, respectively. Indicates the first Frame ground point cloud collection The coordinates of the ground points currently involved in the registration calculation. For the first Frame ground point cloud collection coordinates of the center and ground point The corresponding matching point coordinates, For the first Frame ground point cloud collection Ground point coordinates The normal vector, The weight coefficients for the consistency term of the normal vector. For the first Frame ground point cloud collection Matching point coordinates The normal vector, Indicates the first Frame ground point cloud collection The number of ground points involved in the optimization calculation Represents the variable to be summed. Point clouds on the ground; Step 2.2: For non-ground point clouds, solve the first step based on point-to-plane constraints. non-ground point cloud collection With the non-ground point cloud collection The rotation matrix and translation vector between them are used to obtain the non-ground point cloud registration result, and its optimization objective is: in, and They represent the first non-ground point cloud collection and the non-ground point cloud collection The optimal rotation matrix and optimal translation vector obtained from the registration and This represents the rotation matrix and translation vector to be solved during the optimization process. Indicates the first non-ground point cloud collection The coordinates of the non-ground points currently involved in the registration calculation. Indicates the first non-ground point cloud collection Coordinates of points in the middle and non-ground The corresponding matching point coordinates, Represents the coordinates of non-ground points The normal vector at that point, Indicates the first non-ground point cloud collection The number of non-ground points involved in the optimization calculation These are non-ground point clouds; Step 2.3, according to the first Frame ground point cloud collection , No. Frame ground point cloud collection Registration results and the first non-ground point cloud collection , No. non-ground point cloud collection The registration results are then fused to obtain the first... Frame and the The final registration transformation between frames is performed, and based on the final registration transformation, the multi-frame lidar point cloud is registered and accumulated frame by frame to generate a global lidar point cloud. The final registration formula is: in, This represents a transformation solver that fuses the registration results of ground point clouds with those of non-ground point clouds.

4. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1: Extract keyframes from the UAV video sequence and obtain two-dimensional keypoint pairs (i.e., the first keyframes) using feature detection and feature matching methods. Frame Image and the Frame Image ; Step 3.2: Calculate the first [item] in the UAV video sequence. Frame image and the first The essential matrix between frames : in, Indicates the first Frame Image and the Frame Image The relative rotation matrix between them Indicates the first Frame Image With the Frame Image The relative translation vector between them Represents the relative translation vector The antisymmetric matrix formed; Step 3.3, based on the essential matrix The camera pose is recovered and triangulated to obtain sparse 3D point cloud coordinates. The 3D point projection relationship is as follows: in, The coordinates of the sparse 3D point cloud are represented at the 1st position. Frame Image The depth scale factor in and Indicates the first Frame Image Center and three-dimensional points The corresponding two-dimensional pixel coordinates, where 1 represents the homogeneous coordinate component. Represents the camera intrinsic parameter matrix. and They represent the first Frame Image The corresponding camera rotation matrix and translation vector, The first term obtained by triangulation is represented as the third term. A sparse three-dimensional point cloud coordinate system; Step 3.4: After obtaining the coordinates of the sparse 3D point cloud, a multi-view stereo reconstruction method is used to perform dense reconstruction of the sparse 3D point cloud, generating an SfM dense color point cloud with RGB color information. .

5. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4.1, let the global lidar point cloud be... SfM dense colored point cloud is , and These represent global lidar points and SfM dense color points in Euclidean space, respectively. This represents the total number of point clouds in the global LiDAR system. To determine the number of SfM dense color point clouds, the original 3D coordinates of the global LiDAR point cloud and the SfM dense color point cloud are embedded into the TGSC-Attn feature generation module through a one-dimensional convolutional layer. In the initial feature space, the initial feature representations of the lidar point cloud and the initial feature representation of the SfM point cloud after initial encoding are obtained: in, This represents the dimension of the initial feature space. This represents the initial feature representation of the global lidar point cloud after initial encoding. This is the initial feature representation of the SfM dense color point cloud after initial encoding, with superscript indicating the initial feature representation. This represents the output of layer 0 in the shared feature encoder of the TGSC-Attn feature generation module; Step 4.2: Initial feature representation of lidar point cloud. Initial feature representation of SfM point cloud Multiple self-attention layers are applied to update the initial feature representation of the lidar point cloud layer by layer. Initial feature representation of SfM point cloud ; in, This represents the total number of self-attention layers. This indicates the layer index of the current self-attention layer. and These represent the lidar point cloud feature representation and the SfM point cloud feature representation output from the previous layer, respectively. Step 4.3: After the self-attention layer, intermodal information exchange is performed through the cross-attention layer: in, This indicates the total number of cross-attention layers; the output of the last layer represents the deep features of the global LiDAR point cloud. Deep features of dense colored point clouds (SfM) .

6. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: In step 5, the deep features of the global LiDAR point cloud and the deep features of the SfM dense color point cloud are adversarially aligned using the GAN-FeaAlign feature alignment module. The objective function is: in, This refers to the shared feature encoder in the TGSC-Attn feature generation module. This represents the domain discriminator in the GAN-FeaAlign feature alignment module. This represents the input samples from the global LiDAR point cloud. This represents an input sample from a dense colored point cloud (SfM).

7. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: Step 6 specifically includes the following steps: Step 6.1: Based on the shared latent feature space obtained in Step 5, let the global LiDAR point cloud aligned feature set obtained after adversarial alignment processing by the TGSC-Attn feature generation module and the GAN-FeaAlign feature alignment module be (the set of features to be defined in the original text). The SfM dense color point cloud alignment feature set is Calculate the similarity score between the global lidar point cloud alignment feature set and the SfM dense color point cloud alignment feature set. And form a similarity matrix : in, Represents the alignment feature vector of LiDAR point cloud Align feature vectors with SfM dense color point cloud The inner product, The norm of the aligned feature vectors of the lidar point cloud is represented. Let be the norm of the aligned feature vectors of the SfM dense color point cloud. This indicates the total number of point clouds in the global LiDAR system. Indicates the number of dense colored point clouds in SfM. , ; Step 6.2: Based on the similarity matrix For each global lidar point Selecting from SfM dense color point cloud The candidate matching points with the highest similarity are the Top-k candidate points, and a coarse matching point pair set is obtained by applying bidirectional mutual consistency constraints. : in, Represented as the first in the global lidar point cloud One point, Represents the first in the SfM dense colored point cloud One point, Indicates the connection with the global lidar point The most similar A set of SfM candidate points This indicates the dense colored points of the SfM. The most similar A set of candidate points for lidar; Step 6.3: For the coarse matching point pair set Candidate point pairs By comparing the first point in the global lidar point cloud Points With SfM dense colored dots Calculate the consistency score based on the local neighborhood geometry consistency in each point cloud, and then apply the consistency score threshold. Filter to obtain a set of fine matching point pairs The formula for calculating the consistency score is: in, and These represent the first and second points in the global lidar point cloud, respectively. Points With SfM dense colored dots The local neighborhood, Represents the first point in the global lidar point cloud. Points Its neighboring points The Euclidean distance between them Represents SfM dense colored dots Its neighboring points The Euclidean distance between them Indicates the scale parameter.

8. The cross-source point cloud registration method based on adversarial neural networks according to claim 7, characterized in that: Step 7 specifically includes the following steps: Step 7.1: Filter the set of fine matching points obtained in Step 6.

3. and corresponding consistency score As weights, construct the rigid transformation optimization objective: in, Represents the rotation matrix. Represents the translation vector; Step 7.2: Based on the set of fine-matching points The consistency score of each point pair in the set will be used to refine the set of matching point pairs. The endpoints of the lidar in the set constitute the lidar matching point set. The endpoints of the fine matching point pair set are then combined to form the SfM matching point set. The weighted centroids of the lidar matching point set and the SfM matching point set are calculated respectively. in, This represents the weighted centroid of the LiDAR matching point set in the fine matching point pair. This represents the weighted centroid of the SfM matching point set in a fine-matching point pair; Step 7.3: Set the fine-matching point pairs Each matching point is centered relative to the weighted centroid of its respective LiDAR matching point set and the weighted centroid of the SfM matching point set, and a weighted cross-covariance matrix is ​​constructed. : Step 7.4: Calculate the weighted cross-covariance matrix. Perform singular value decomposition And calculate the optimal rotation matrix. Translation vector : in, This represents the orthogonal matrix of left singular vectors obtained by performing singular value decomposition on the weighted cross-covariance matrix. This represents the right singular vector orthogonal matrix obtained by performing singular value decomposition on the weighted cross-covariance matrix. Represents a singular value diagonal matrix. For matrix transpose, and Together they are used to calculate the optimal rotation matrix. And based on the rotation matrix Translation vector Constructing a rigid transformation matrix ; Step 7.5, when When, the orthogonal matrix of the right singular vectors obtained by singular value decomposition is... The last column is corrected by sign flipping to ensure that it is used to construct the rigid transformation matrix. rotation matrix For a valid rotation matrix, where Represents a determinant.

9. The cross-source point cloud registration method based on adversarial neural networks according to claim 8, characterized in that: Step 8 specifically includes the following steps: Step 8.1: Apply the rigid transformation matrix obtained in Step 7. Application to global lidar point cloud The transformed lidar point cloud is obtained from each point in the center. : in, This represents the original global lidar point cloud. This represents the lidar point cloud after rigid transformation. Represented by rotation matrix Translation vector The rigid transformation matrix formed; Step 8.2: Transform the lidar point cloud after rigid transformation. SfM Dense Colored Point Clouds By registering to a unified coordinate system, the lidar point cloud and the SfM dense color point cloud are aligned and fused to obtain a fused point cloud: Step 8.3: Based on the fused point cloud Generate multimodal 3D reconstruction results for outdoor scenes.

10. The cross-source point cloud registration method based on adversarial neural networks according to claim 1, characterized in that: The cross-source point cloud registration method is implemented through a cross-source point cloud registration system, which includes a lidar point cloud registration module, an SfM point cloud generation module, and a cross-modal point cloud alignment module. The lidar point cloud registration module is used to segment the input multi-frame lidar point cloud sequence into ground point cloud and non-ground point cloud, and register them separately to generate a global lidar point cloud. The SfM point cloud generation module is used to recover the camera pose and generate SfM dense color point clouds based on the UAV video sequence collected by the UAV. The cross-modal point cloud alignment module uses the cross-source point cloud registration network MMAlignNet to perform cross-modal feature alignment and rigid registration between the global lidar point cloud and the SfM dense color point cloud; The TGSC-Attn feature generation module performs multi-scale feature encoding on the global LiDAR point cloud and SfM dense color point cloud through one-dimensional convolutional layers, self-attention layers, and cross-attention layers to extract deep features from the global LiDAR point cloud. Deep features of dense colored point clouds (SfM) , among which, the The output of the self-attention layer is a feature representation of the LiDAR point cloud. SfM point cloud feature representation And through cross-attention layers in lidar point cloud feature representation The system integrates intramodal structural information and intermodal correlation information. The GAN-FeaAlign feature alignment module uses a domain discriminator to analyze the deep features of the global LiDAR point cloud. Deep features of dense colored point clouds with SfM Adversarial learning is employed to reduce the feature distribution differences between lidar point clouds and SfM dense color point clouds, thereby achieving alignment of the two modal features in a shared latent feature space. The RCF-RobustMatch robust matching module is used to perform coarse matching filtering based on the similarity matrix and bidirectional mutual consistency constraints, and to perform fine matching filtering by combining local neighborhood geometric consistency, so as to obtain a set of fine matching point pairs. The weighted singular value decomposition (WSVD) solution module calculates the rotation matrix and translation vector between the lidar point cloud and the SfM dense color point cloud based on the set of fine matching point pairs and the consistency score corresponding to each fine matching point pair, thus completing the cross-modal rigid transformation estimation.

Images