An aircraft natural fiducial feature constraint point cloud registration method based on a pre-trained large visual model

Abstract

The application discloses an airplane natural reference feature constraint point cloud registration method based on a pre-trained large visual model, which comprises the following steps: acquiring point cloud data and corresponding image data, and establishing a corresponding relationship; inputting the image data into the pre-trained large visual model to extract the two-dimensional expression of the airplane surface natural reference feature; mapping the two-dimensional expression to the three-dimensional point cloud to obtain the three-dimensional natural reference feature of each station through spatial clustering and geometric fitting; calculating the candidate matching comprehensive confidence to obtain the natural reference feature corresponding relationship through screening; constructing the geometric constraint between stations, solving the rigid body transformation relationship between stations, and performing the point cloud coarse registration and local precise registration between stations; constructing a multi-view point cloud global optimization graph, performing weighted global optimization on the poses of multiple stations, and obtaining the large airplane whole machine point cloud registration result. The application can reduce the dependence on artificial targets and artificial homonymic points, and improve the automation degree, stability and engineering applicability of the large airplane whole machine point cloud registration.

Description

Technical Field

[0001] This invention belongs to the field of aircraft digital measurement and 3D point cloud processing technology, specifically involving a point cloud registration method based on pre-trained large visual models constrained by aircraft natural reference features. Background Technology

[0002] The manufacturing, assembly, and maintenance inspection processes of large aircraft place high demands on overall shape accuracy, assembly consistency, and surface quality. Acquiring 3D point cloud data of the aircraft surface using laser scanners, structured light scanners, and photogrammetry equipment provides a data foundation for digital modeling, shape inspection, and assembly error analysis of the entire aircraft. Due to the large size and complex structure of large aircraft, a single measuring station typically cannot acquire a complete point cloud of the aircraft surface at once. Therefore, multiple measuring stations need to be set up around the aircraft to acquire local point clouds from different perspectives, and then the point clouds from multiple measuring stations are unified into the same coordinate system to complete the stitching of the overall aircraft point cloud. Thus, multi-measuring station point cloud registration is a crucial step in the digital measurement of large aircraft.

[0003] Existing point cloud registration methods for large aircraft typically employ manual target placement, manual selection of corresponding points, or registration based on local geometric features of the point cloud. Manual target placement requires deploying multiple targets on the surface of the object being measured or in the measurement environment, establishing spatial relationships between different stations by identifying the target centers. While this method provides initial constraints for point cloud registration, the deployment and maintenance of targets in large aircraft measurement scenarios is labor-intensive, and some aircraft surfaces are unsuitable for attaching or installing targets. Manual selection of corresponding points requires operators to select the same location points in overlapping areas of different point cloud locations. This method relies on manual experience, has low automation, and is prone to selection errors when there are repetitive structures on the aircraft surface or when the overlapping area is small.

[0004] Registration methods based on local geometric features of point clouds typically use normals, curvature, FPFH, SHOT, spherical harmonic features, or voxel features to describe the shape of the local point cloud. They then establish correspondences based on feature similarity and solve rigid body transformations using random sampling consistency, singular value decomposition, iterative nearest point, or graph optimization methods. While these methods can reduce human intervention, large aircraft surfaces have extensive smooth skin, low-curvature wing surfaces, and tail surfaces, resulting in insufficient discriminative power for local geometric features. Furthermore, structures such as windows, fasteners, and skin seams on aircraft surfaces exhibit strong repetitiveness, making it easy to produce incorrect matches by relying solely on local geometric descriptors. This negatively impacts the stability of coarse registration and the overall accuracy of point cloud stitching.

[0005] In recent years, pre-trained large vision models have demonstrated strong generalization capabilities in image feature extraction, object detection, semantic segmentation, and open-vocabulary recognition, enabling them to extract rich boundary, texture, and semantic information from two-dimensional images. While large aircraft surfaces possess extensive smooth areas, they also exhibit naturally identifiable and repeatable baseline features such as window boundaries, door outlines, skin seams, wing-body junctions, wing leading edges, engine nacelle outlines, tail edges, radome boundaries, and fastener arrays. These natural baseline features are relatively easy to identify in two-dimensional images, and when mapped to three-dimensional point clouds, they can form stable three-dimensional points, lines, curves, and region constraints.

[0006] However, existing image and point cloud fusion registration methods primarily focus on fusing image texture information with point cloud geometric information in general scenarios, lacking specific constraints for the natural reference features of large aircraft. Existing aircraft point cloud registration methods mainly rely on artificial targets, artificial corresponding points, or local geometric descriptors, making it difficult to fully utilize the existing stable structural features of the aircraft surface. Especially in smooth skin regions, low-overlapping station regions, and regions with repetitive structures on large aircraft, relying solely on local point cloud geometric descriptions easily leads to problems such as unstable candidate correspondences, difficulty in eliminating incorrect matches, and accumulation of errors across multiple stations. Therefore, a multi-station point cloud registration method is needed that can utilize the natural reference features of the aircraft surface and transform 2D visual recognition results into 3D geometric constraints. Summary of the Invention

[0007] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a point cloud registration method based on pre-trained large visual models that constrains aircraft natural reference features. This method involves acquiring aircraft point cloud and image data from multiple stations, extracting aircraft natural reference features, mapping two-dimensional natural reference features to three-dimensional structures, clustering and geometric fitting of three-dimensional natural reference point sets, screening candidate matching with comprehensive confidence, coarse registration between stations, and weighted global optimization of multi-viewpoint clouds. This allows existing natural reference features on the aircraft surface to participate in multi-station point cloud registration, reducing reliance on manual targets and manual homonyms, and improving the automation, stability, and engineering applicability of large aircraft point cloud registration.

[0008] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0009] A point cloud registration method based on pre-trained large visual models and constrained by natural reference features of aircraft includes the following steps:

[0010] Step S1: Obtain aircraft point cloud data from multiple stations at corresponding viewpoints. and the point cloud data Corresponding image data For the point cloud data After preprocessing, the cloud data of the test site is obtained. Establish image data Cloud data of the testing site The correspondence between them;

[0011] Step S2: Transfer the image data from each station. Input the pre-trained large vision model to obtain the visual feature map of the aircraft surface of the corresponding station, so as to detect or segment the natural reference features of the aircraft surface of the corresponding station and obtain the two-dimensional representation of the natural reference features of the corresponding station.

[0012] Step S3: Based on image data Cloud data of the testing site The correspondence between them maps the two-dimensional representation of the natural reference features to the three-dimensional point cloud corresponding to each station. Through projection analysis and spatial clustering, the three-dimensional natural reference point set in each station is obtained. The three-dimensional natural reference point set is geometrically fitted to obtain the three-dimensional natural reference features of each station, and the geometric attributes of the three-dimensional natural reference features are calculated.

[0013] Step S4: Based on the semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence and adjacency relationship of the three-dimensional natural reference features, calculate the comprehensive confidence of candidate matching of two three-dimensional natural reference features of different stations, and filter to obtain the correspondence of natural reference features.

[0014] Step S5: Construct geometric constraints between stations based on the correspondence of the natural reference features, solve the rigid body transformation relationship between stations, and perform coarse registration and local fine registration of point clouds between stations;

[0015] Step S6: Using multiple station cloud points as graph nodes and the rigid body transformation relationship between stations as graph edges, construct a multi-viewpoint cloud global optimization graph. Combine the natural reference feature matching residual and the point cloud geometric residual to perform weighted global optimization of the poses of multiple stations, and obtain the point cloud registration result of the large aircraft.

[0016] To optimize the above technical solution, the specific measures also include:

[0017] Step S1 above includes the following process:

[0018] Step S101: Deploy around the large aircraft Each station collects aircraft point cloud data from different perspectives using 3D measurement equipment. And to ensure that there is a preset overlap area between adjacent measuring station clouds, wherein For the first Aircraft point cloud data from the perspective of each station;

[0019] Step S102: Synchronously acquire the point cloud data using a measuring device. Corresponding image data ;

[0020] Step S103: Process the point cloud data Outlier removal, noise reduction, downsampling, and normal estimation are performed to obtain the station cloud data. ;

[0021] Step S104: Establish image data based on the camera intrinsic parameters of the measuring equipment, the extrinsic parameters between the camera and the 3D measuring equipment, or the pixel-to-point cloud correspondence provided by the measuring equipment software development kit. Cloud data of the testing site The correspondence between them.

[0022] In step S102 above, when the measuring device cannot directly acquire two-dimensional image data, it uses the point cloud data... Generate a point cloud projection map as the corresponding image data. The point cloud projection map is at least one of a depth map, a normal map, a curvature map, and a reflection intensity map.

[0023] The natural reference features mentioned in step S2 above include at least one of the following: window boundary, door outline, skin seam, wing leading edge, wing trailing edge, wing-body junction, engine nacelle outline, tail edge, radome boundary, and fastener array; the two-dimensional representation of the natural reference features includes at least one of the following: two-dimensional mask, two-dimensional boundary, two-dimensional centerline, and two-dimensional key point, as well as the type label and confidence level corresponding to the two-dimensional representation.

[0024] Step S3 above includes the following sub-steps:

[0025] Based on image data Cloud data of the testing site The correspondence between them will be the first Mapping the two-dimensional mask, two-dimensional boundary, two-dimensional centerline, or two-dimensional key points of the natural benchmark features of each station to the first station. In the 3D point cloud corresponding to each station;

[0026] For cloud data at testing sites 3D points According to the camera intrinsic parameter matrix and extrinsic parameter matrix Calculate its projected pixel coordinates The relationship is as follows:

[0027]

[0028] in, As a scale factor, For the first The intrinsic parameter matrix of each station corresponding to the camera. and These are the rotation matrix and translation vector between the camera and the 3D measurement device, respectively. For three-dimensional points The pixel coordinates after projection onto the image plane;

[0029] Determine the projected pixel coordinates Whether the point falls within the neighborhood of a 2D mask, 2D boundary, 2D centerline, or 2D keypoint of the natural reference feature; if so, assign the corresponding 3D point to the next 3D keypoint. Each station has a corresponding set of candidate 3D natural reference points, and the type label and confidence level of the natural reference features are assigned to the 3D point; the type label includes a semantic type label and a geometric type label.

[0030] Spatial clustering is performed on the 3D points in the candidate 3D natural reference point set. 3D points with the same semantic type label and spatial distance satisfying the preset neighborhood condition are grouped into the same 3D natural reference point set, and isolated point sets with fewer than a preset threshold are removed to obtain the first set. A set of three-dimensional natural reference points for each station;

[0031] Geometric fitting is performed based on the geometric morphology type corresponding to the three-dimensional natural reference point set to obtain the first... Three-dimensional natural benchmark features of each station;

[0032] Calculate the geometric properties of the three-dimensional natural reference feature, including at least one of length, curvature, principal direction, contour area, aspect ratio, array spacing, and endpoint spacing.

[0033] The aforementioned geometric morphology types include at least one of closed contour type, linear type, and array type;

[0034] For closed contour-type natural datum features, a closed three-dimensional boundary curve is fitted and used as a three-dimensional natural datum feature;

[0035] For linear natural reference features, a three-dimensional spatial curve is fitted and used as the three-dimensional natural reference feature.

[0036] For array-type natural reference features, the array centerline or array direction vector is fitted and used as a three-dimensional natural reference feature.

[0037] Step S4 above includes the following sub-steps:

[0038] Step S401: Based on the semantic type labels of the three-dimensional natural reference features, perform same-type candidate matching on the three-dimensional natural reference features in different stations, and take the three-dimensional natural reference features with the same semantic type labels as candidate matching objects to obtain candidate matching natural reference features.

[0039] Step S402: For any candidate matching pair obtained in step S401 ,in and These are the three-dimensional natural benchmark features from different stations, and the candidate matching pairs are used to... The semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence, and intra-station adjacency relationship are used to calculate the type consistency score, geometric consistency score, local point cloud consistency score, visual confidence score, and adjacency relationship consistency score, respectively.

[0040] Step S403: Based on the scores of each sub-item obtained in step S402, calculate the candidate matching pair. Overall confidence of candidate matches:

[0041]

[0042] in, For type consistency score, For geometric consistency score, For local point cloud consistency score, Visual confidence score. Scoring is given based on the consistency of adjacency relationships; , , , , The weight coefficients are non-negative and satisfy the following conditions: ;

[0043] Step S404: Based on the comprehensive confidence level of the candidate matches The natural baseline feature correspondence is obtained through screening: when the comprehensive confidence of candidate matches... If the number of candidates is not less than a preset threshold, retain the corresponding candidate matching pairs. The obtained natural baseline feature correspondence; when the candidate matching comprehensive confidence level When the value is less than a preset threshold, the corresponding candidate matching pair is removed. ;

[0044] Step S405: Perform global consistency verification on the natural benchmark feature correspondence obtained by screening: estimate the rigid body transformation between candidate stations based on multiple natural benchmark feature correspondences, and determine whether the matching residual of each natural benchmark feature correspondence under the rigid body transformation between the candidate stations meets the preset residual threshold. If it does not meet the preset residual threshold, the corresponding natural benchmark feature correspondence is determined to be an outlier correspondence and is removed.

[0045] The type consistency score mentioned above is:

[0046]

[0047] in, and These are three-dimensional natural benchmark features. and semantic type tags;

[0048] Geometric consistency score:

[0049]

[0050] in, The scale parameter for the difference in geometric properties. and These are three-dimensional natural benchmark features. and Normalized geometric attribute vector;

[0051] The local point cloud consistency score is:

[0052]

[0053] in, The scale parameter for the differences in local point cloud descriptors; and Three-dimensional natural benchmark features and Local neighborhood point cloud descriptor;

[0054] The visual confidence score is:

[0055]

[0056] in, and These are three-dimensional natural benchmark features. and Visual confidence;

[0057] The adjacency consistency score is:

[0058]

[0059] in, This is the scale parameter for the difference in adjacency relationships. and These are three-dimensional natural benchmark features. and The adjacency vector.

[0060] Step S5 above includes the following sub-steps:

[0061] Step S501: Construct inter-station geometric constraints based on the correspondence of the natural reference features. The geometric constraints include at least one of point-to-point constraints, point-to-line constraints, line-to-line constraints, and region-to-region constraints.

[0062] Step S502: Based on the geometric constraints between the stations, solve for the rigid body transformation relationships between the stations, including the rotation matrix. Translation vector ;

[0063] Step S503: Apply the rigid body transformation relationship to the corresponding station cloud to obtain the coarse registration result between stations;

[0064] For the 3D points in the cloud of individual test sites The transformed points Represented as:

[0065]

[0066] in, For the station To the station The rotation matrix, For the station to the station Translation vector;

[0067] Step S504: Based on the coarse registration results between stations, establish a local point cloud correspondence within the overlapping area of ​​the two station clouds according to the distance threshold and the normal angle threshold, and construct a local fine registration objective function using the point-to-surface distance and normal deviation, iteratively optimizing the rigid body transformation between stations; the local fine registration objective function is:

[0068]

[0069] in, For the distance from a point to a surface, For the normal deviation term, This is the weighting coefficient for the normal deviation term; It is a set of local point cloud correspondences; As the source The normal vector, For target point The normal vector;

[0070] Step S505: Determine whether the residual after local fine registration meets the preset accuracy requirements. If it meets the preset accuracy requirements, retain the inter-station rigid body transformation relationship obtained after coarse registration and local fine registration as a valid inter-station registration relationship. If it does not meet the preset accuracy requirements, reduce the weight of the inter-station registration relationship in the subsequent global optimization, or return to step S4 to re-select the correspondence of natural datum features and re-solve the inter-station rigid body transformation relationship.

[0071] Step S6 above includes the following sub-steps:

[0072] Step S601: Use each preprocessed test site cloud data obtained in step S103 as a node in the global optimization graph;

[0073] Step S602: Based on the rigid body transformation relationship between stations, establish graph edges between two stations that have a natural benchmark feature correspondence or point cloud overlapping area to obtain a multi-view point cloud global optimization map.

[0074] Step S603: Match residuals based on natural benchmark features Point cloud geometric residuals Confidence of natural baseline features Matching interior point ratio and overlapping area Determine the edge weights of the graph ; and These are the scale parameters for the natural baseline feature matching residual and the point cloud geometric residual, respectively;

[0075] Step S604: Based on the global optimization graph and graph edge weights So that the following objective function Weighted global optimization of the poses of multiple stations is performed with the goal of minimizing the position of each station, resulting in the rigid body transformation of each station's geocloud to a unified coordinate system.

[0076]

[0077] in, For the set of edges of the globally optimized graph, For the station With the station The edge weights between them Match residuals to natural baseline features. For point cloud geometric residuals, This is the residual balance coefficient. and They are the stations and monitoring station Rigid body transformation to a unified coordinate system;

[0078] Step S605: Based on the rigid body transformation of each measuring station cloud to a unified coordinate system, stitch together the multiple measuring station clouds to obtain the registration result of the large aircraft's whole point cloud.

[0079] Step S606: Calculate the registration residuals of different station clouds in the overlapping area and output the registration quality evaluation results, including at least one of the following: average residual, maximum residual, root mean square error, error heatmap, and low confidence region indication.

[0080] The present invention has the following beneficial effects:

[0081] This invention utilizes a pre-trained large vision model to extract natural reference features of the aircraft surface from images or point cloud projections. It also transforms the aircraft's own structure, such as window boundaries, door outlines, skin seams, wing-body junctions, wing edges, engine nacelle outlines, and fastener arrays, into three-dimensional registration references. This reduces the need for manual target placement and manual selection of corresponding points, and improves the automation level of point cloud measurement and registration for large aircraft.

[0082] This invention extracts natural reference features from two-dimensional images or point cloud projection maps based on a pre-trained large vision model, and then maps them to a three-dimensional point cloud for spatial clustering and geometric fitting. This transforms the two-dimensional visual recognition results into three-dimensional geometric constraints that can participate in point cloud registration. It not only leverages the advantages of the pre-trained large vision model in boundary, contour, and structural region recognition, but also ensures the engineering feasibility of the method through three-dimensional geometric fitting and point cloud optimization.

[0083] This invention uses semantic type consistency, geometric consistency, local point cloud consistency, visual confidence, and adjacency consistency of three-dimensional natural reference features to jointly screen candidate correspondences. This can alleviate the problems of weak geometric features in the smooth skin area of ​​large aircraft, numerous repetitive structures such as windows and fastener arrays, and mismatches in low-overlap areas, thereby improving the stability of coarse registration between stations.

[0084] Based on the coarse registration between stations, this invention establishes a local point cloud correspondence in the overlapping area according to the distance threshold and the normal angle threshold, and constructs a local fine registration objective function with the point-to-surface distance and normal deviation, which can further improve the local alignment accuracy between adjacent station clouds.

[0085] In the process of multi-viewpoint cloud registration, this invention incorporates the natural reference feature matching residual and the point cloud geometric residual into the global optimization objective, and determines the edge weights based on the confidence level of the natural reference feature, the proportion of matched internal points, and the area of ​​the overlapping region. This can reduce the impact of local mismatches on the overall point cloud stitching result and reduce the accumulation of errors in the multi-station registration process.

[0086] This invention can output registration quality evaluation results after the whole aircraft point cloud registration is completed, providing a reliable data foundation for subsequent aircraft shape inspection, assembly error analysis, maintenance inspection and quality assessment. Attached Figure Description

[0087] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0088] Figure 1 This is a flowchart of a point cloud registration method based on pre-trained large vision model for aircraft natural reference feature constraints according to the present invention.

[0089] Figure 2 This is a schematic diagram illustrating the two-dimensional extraction, three-dimensional mapping, and geometric fitting of natural reference features in this invention;

[0090] Figure 3 This is a schematic diagram showing the comparison before and after multi-viewpoint cloud registration constrained by the natural reference feature of the present invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0092] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0093] This invention discloses a point cloud registration method for aircraft based on pre-trained large visual models and constrained by natural reference features. It utilizes existing natural reference features on the surface of large aircraft, such as window boundaries, door outlines, skin seams, wing-body junctions, wing edges, engine nacelle outlines, and fastener arrays, mapping them from two-dimensional images or point cloud projections to a three-dimensional point cloud. This forms a three-dimensional geometric constraint for multi-measurement point cloud registration, solving the problems of high workload for manual target placement, low efficiency of manual selection of corresponding points, easy mismatching of smooth skin and repetitive structural regions, and easy accumulation of multi-measurement point cloud registration errors in existing large aircraft whole-aircraft point cloud registration methods. The method flow is as follows: Figure 1 As shown, the specific steps include the following:

[0094] Step S1: Obtain aircraft point cloud data from multiple stations at corresponding viewpoints. and the point cloud data Corresponding image data For the point cloud data After preprocessing, the cloud data of the test site is obtained. Establish image data Cloud data of the testing site The correspondence between them;

[0095] Step S2: Transfer the image data from each station. Input the pre-trained large vision model to obtain the visual feature map of the aircraft surface of the corresponding station, so as to detect or segment the natural reference features of the aircraft surface of the corresponding station and obtain the two-dimensional representation of the natural reference features of the corresponding station.

[0096] Step S3: Based on image data Cloud data of the testing site The correspondence between them is mapped to the two-dimensional representation of the natural reference features to the three-dimensional point cloud corresponding to each station. The three-dimensional natural reference point set within each station is obtained through projection analysis and spatial clustering. The three-dimensional natural reference point set is geometrically fitted to obtain the three-dimensional natural reference features of each station, and the geometric attributes of the three-dimensional natural reference features are calculated.

[0097] Step S4: Based on the semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence and adjacency relationship of the three-dimensional natural reference features, calculate the comprehensive confidence of candidate matching of two three-dimensional natural reference features of different stations, and filter to obtain the correspondence of natural reference features.

[0098] Step S5: Construct geometric constraints between stations based on the correspondence of the natural reference features, solve the rigid body transformation relationship between stations, and perform coarse registration and local fine registration of point clouds between stations;

[0099] Step S6: Using multiple station cloud points as graph nodes and the rigid body transformation relationship between stations as graph edges, construct a multi-viewpoint cloud global optimization graph. Combine the natural reference feature matching residual and the point cloud geometric residual to perform weighted global optimization of the poses of multiple stations, and obtain the point cloud registration result of the large aircraft.

[0100] In this embodiment, step S1 is used to acquire point cloud data and image data of the large aircraft at multiple measurement stations, and to establish the correspondence between the image data and the point cloud data. Assume that a total of [number missing] measurement stations are deployed around the large aircraft. The first monitoring station, the first The raw aircraft point cloud data collected by each station is denoted as With the point cloud data The corresponding image data is denoted as ,in .

[0101] Based on the dimensions of the large aircraft, the field of view of the scanning equipment, and the required measurement accuracy, multiple measurement stations are deployed around the aircraft. Each station collects point cloud data of the aircraft from a local perspective, ensuring a pre-defined overlap between point cloud data from adjacent stations. Before subsequent processing, the collected point cloud data is... Preprocessing is performed to obtain the cloud data of the test site. .in, This represents the original collected point cloud. Indicates to The preprocessed station cloud data is obtained after outlier removal, denoising, downsampling, and normal estimation. Subsequent natural baseline feature mapping, spatial clustering, geometric fitting, inter-station registration, and global optimization are all based on this station cloud data. conduct.

[0102] The cloud data of the testing site Include The preprocessed 3D points and their additional geometric properties, where the first... The three-dimensional points are denoted as Its corresponding normal vector is denoted as , ,and , , , They represent the first The first of the 10 stations The three-dimensional coordinates of a point in the station coordinate system;

[0103] Preprocessing includes outlier removal, denoising, downsampling, and normal estimation. Specifically, outliers that are significantly deviated from the main body of the aircraft surface point cloud are removed; noise points caused by scanning noise, environmental interference, or surface reflection are filtered; and voxel downsampling is performed on regions with excessively high point density to reduce subsequent computation.

[0104] For any three-dimensional point retained after preprocessing Find its neighboring points within a preset radius, or find its... Find the nearest neighbor points to obtain the local neighborhood point set. Let the centroid of the local neighborhood point set be... Construct the covariance matrix based on the local neighborhood point set:

[0105]

[0106] For covariance matrix Perform eigenvalue decomposition and use the eigenvectors corresponding to the smallest eigenvalues ​​as three-dimensional points. Local normal vector In one implementation, the local normal vector can also be adjusted based on the viewpoint direction of the measuring device or the consistency of the normal vectors of adjacent points. To unify the direction.

[0107] The normal vector As a three-dimensional point The additional geometric attributes and their corresponding storage, the preprocessed station cloud data This includes the preprocessed 3D point coordinates and corresponding normal vector information. The normal vector information is used for subsequent local neighborhood point cloud feature calculation, local point cloud consistency judgment in natural reference feature matching, and normal deviation constraints in local fine registration.

[0108] When the measuring device can simultaneously acquire two-dimensional images, the simultaneously acquired two-dimensional images are used as image data. When the measuring device cannot directly acquire a two-dimensional image, it uses the point cloud data... Generate a point cloud projection map as the corresponding image data. The point cloud projection map can be a two-dimensional image, such as a depth map or a reflection intensity map, that can be directly generated from the raw point cloud data. After completing the point cloud data... Preprocessing and obtaining cloud data from the test site Subsequently, cloud data from the aforementioned testing sites can also be used. The point cloud projection map is updated, and the updated point cloud projection map can include at least one of a depth map, a normal map, a curvature map, and a reflection intensity map. Through the above method, the image data... It can be a two-dimensional image synchronously acquired by measuring equipment, or a point cloud projection map generated from point cloud data.

[0109] Image data Cloud data of the testing site The correspondence between them can be established in two ways: The first way is to establish the relationship between image pixels and station cloud data based on the camera's intrinsic parameters and the extrinsic parameters between the camera and the 3D measurement equipment. The first method involves the projection relationship between two-dimensional points. The second method involves directly establishing the mapping relationship between 2D image pixels and 3D point cloud points based on the pixel-to-point cloud correspondence provided by the measurement equipment software development kit. When the number of point cloud points or the point index changes during preprocessing, the pixel-to-point cloud correspondence can be inherited based on the original index of the retained points, or the preprocessed measurement point cloud data can be modified based on the camera's intrinsic and extrinsic parameters. Recalculate the projection to obtain the image data. Cloud data of the testing site The correspondence between them. This correspondence is used to map natural reference features in the image to a 3D point cloud.

[0110] Specifically, step S1 includes the following process:

[0111] Step S101: Based on the external dimensions of the large aircraft, the field of view of the scanning equipment, and the measurement accuracy requirements, multiple measuring stations are set up around the large aircraft to collect aircraft point cloud data from different perspectives. And to ensure that there is a preset overlap area between adjacent measuring stations;

[0112] Step S102: Synchronously collect the aircraft point cloud data. Corresponding two-dimensional image data When the measuring equipment cannot directly acquire two-dimensional image data, it uses the aircraft point cloud data... Generate a point cloud projection map as the corresponding image data. ;

[0113] Step S103: Process the aircraft point cloud data Outlier removal, noise reduction, downsampling, and normal estimation are performed to obtain preprocessed station cloud data. For any three-dimensional point retained after preprocessing... The local neighborhood point set is obtained through neighborhood query, and local plane fitting or principal component analysis is performed based on the local neighborhood point set to obtain the local normal vector corresponding to the 3D point. The normal vector As a three-dimensional point The additional geometric attributes and their corresponding storage are then completed. This is done after the point cloud data is fully processed. Preprocessing and obtaining cloud data from the test site Subsequently, cloud data from the aforementioned testing sites can also be used. The point cloud projection map obtained in step S102 is updated. The updated point cloud projection map includes at least one of the depth map, normal map, curvature map and reflection intensity map.

[0114] Step S104: Establish image data based on the camera intrinsic parameters of the measuring equipment, the extrinsic parameters between the camera and the 3D measuring equipment, or the pixel-to-point cloud correspondence provided by the measuring equipment software development kit. Cloud data of the testing site The correspondence between them. This correspondence is used to map natural reference features in the image to a 3D point cloud.

[0115] In this embodiment, step S2 converts the image data Or from the point cloud data The generated point cloud projection map is input into a pre-trained large visual model to extract natural reference features from the aircraft surface, resulting in a two-dimensional representation of these natural reference features. The image data... Two-dimensional images synchronously acquired by the measuring device, or point cloud data obtained according to step S102. The generated point cloud projection map, or the point cloud data generated after step S103 based on the preprocessed measurement point cloud data. Update the obtained point cloud projection map.

[0116] Step S2 includes the following sub-steps:

[0117] Step S201: Transfer the image data corresponding to each station. Inputting a pre-trained large-scale visual model yields a visual feature map of the aircraft surface. This pre-trained large-scale visual model can be a basic visual model capable of image feature extraction, object detection, semantic segmentation, boundary extraction, or open-vocabulary recognition. Instead of directly outputting point cloud registration results, this model is used to extract natural reference features with stable structural significance from the aircraft surface. In practice, the pre-trained large-scale visual model can be guided to detect or segment the natural reference features of the aircraft surface based on preset aircraft natural reference feature categories, text prompts, image prompts, or manually selected candidate regions.

[0118] Step S202: Detect or segment the natural reference features of the aircraft surface based on the visual feature map to obtain a two-dimensional representation of the natural reference features.

[0119] Specifically, pre-trained large vision models for image data After feature extraction, based on preset natural reference feature categories, text prompts, image prompts, or manually selected candidate regions, at least one of the following natural reference features on the aircraft surface is detected or segmented: window boundaries, door outlines, skin seams, wing leading edges, wing trailing edges, wing-body junctions, engine nacelle outlines, tail fin edges, radome boundaries, and fastener arrays. For the detected target regions, their 2D bounding boxes, 2D masks, or 2D keypoints can be extracted; for the segmented target regions, 2D mask boundaries, 2D centerlines, or 2D contour point sets can be further extracted as 2D representations of the natural reference features.

[0120] The two-dimensional representation of the natural reference feature includes at least one of a two-dimensional mask, a two-dimensional boundary, a two-dimensional centerline, and two-dimensional keypoints, as well as a type label and confidence level corresponding to the two-dimensional representation. For closed contour structures such as window boundaries, door outlines, and engine nacelle outlines, a two-dimensional mask or a two-dimensional boundary can be output; for linear structures such as skin seams, wing-body junctions, wing leading edges, and wing trailing edges, a two-dimensional centerline or boundary line can be output; for fastener arrays, two-dimensional keypoints or array regions can be output. The type label is used to indicate the semantic category to which the natural reference feature belongs, and the confidence level is used to indicate the reliability of the detection or segmentation results of the pre-trained large vision model for the natural reference feature.

[0121] In one implementation, when the confidence level of a natural reference feature is lower than a preset threshold, the natural reference feature is treated as a low-weight constraint or removed from the candidate matching set; when the confidence level of a natural reference feature is not lower than the preset threshold, the natural reference feature is retained and used as a candidate reference feature for subsequent 3D mapping, geometric fitting, and inter-station matching. In another implementation, a minimum retention threshold and a high confidence threshold can be set; when the confidence level of a natural reference feature is lower than the minimum retention threshold, it is removed; when its confidence level is higher than the minimum retention threshold but lower than the high confidence threshold, it is retained as a low-weight candidate reference feature, and its influence is reduced in the candidate matching comprehensive confidence calculation and graph edge weight calculation. Through these methods, the impact of low-confidence visual recognition results on the registration results can be reduced.

[0122] In this embodiment, step S3 is based on image data. Cloud data of testing sites The correspondence between the two-dimensional representations of the natural reference features is mapped to the three-dimensional point cloud to obtain a three-dimensional natural reference point set. The three-dimensional natural reference point set is then geometrically fitted to obtain the three-dimensional natural reference features. The correspondence is either a projection relationship determined by the camera intrinsic parameter matrix and the extrinsic parameters between the camera and the three-dimensional measurement device, or a pixel-to-point cloud index mapping relationship provided by the measurement device software development kit.

[0123] like Figure 2 As shown, after obtaining the two-dimensional representation of the natural baseline features, based on the image data... Cloud data of the testing site The correspondence between them maps the natural reference features in the two-dimensional image to the three-dimensional point cloud. Figure 2 The left side shows natural reference features such as window boundaries and door outlines in an aircraft surface image. The middle side shows the visualization results of the two-dimensional representation of these natural reference features obtained from detection or segmentation based on a pre-trained large vision model. The right side shows the three-dimensional natural reference point set and its geometric fitting results obtained by mapping the two-dimensional natural reference features to a three-dimensional point cloud. Through the above two-dimensional to three-dimensional mapping and fitting process, natural reference features with obvious semantic and contour information in the image can be transformed into three-dimensional geometric constraints that can participate in point cloud registration.

[0124] For cloud data at testing sites The first in Three-dimensional points According to the camera intrinsic parameter matrix and extrinsic parameter matrix Calculate its projected pixel coordinates The relationship is as follows:

[0125]

[0126] in, As a scale factor, For the first The intrinsic parameter matrix of each station corresponding to the camera. and These are the rotation matrix and translation vector between the camera and the 3D measurement device, respectively. For three-dimensional points The pixel coordinates after projection onto the image plane;

[0127] Determine the projected pixel coordinates Whether the projected pixel coordinates fall within the corresponding 2D representation region of the natural reference feature. The corresponding 2D representation region can be a 2D mask region, a 2D boundary neighborhood, a 2D centerline neighborhood, or a 2D keypoint neighborhood. If the projected pixel coordinates fall within the above regions, the corresponding 3D point is assigned to the corresponding candidate 3D natural reference point set, and the type label and confidence score of the natural reference feature are assigned to the 3D point. For a 3D point set obtained by mapping multiple 2D pixels, the confidence score of the 2D natural reference feature can be assigned to each 3D point in the point set, or a statistical value can be calculated based on the 2D confidence score inherited by the 3D points within the point set, which can then be used as the visual confidence score of the 3D natural reference feature.

[0128] After completing the 2D-to-3D mapping, the mapped 3D points are spatially clustered to obtain a 3D natural reference point set. Specifically, 3D points with the same semantic type label and spatial distance satisfying a preset neighborhood condition are grouped into the same 3D natural reference point set. For point sets with fewer than a preset threshold or discrete spatial distribution, they are judged as isolated or unstable point sets and are removed. The type label includes a semantic type label and a geometric morphology type. The semantic type label is used for matching similar feature candidates between different stations, and the geometric morphology type is used to determine the geometric fitting method of the 3D natural reference point set.

[0129] Subsequently, geometric fitting is performed based on the geometric morphology type corresponding to the three-dimensional natural reference point set to obtain three-dimensional natural reference features. The geometric morphology type includes at least one of closed contour type, linear type, and array type; for closed contour type natural reference features, a closed three-dimensional boundary curve is fitted; for linear type natural reference features, a three-dimensional space curve is fitted; for array type natural reference features, an array centerline or array direction vector is fitted. Through geometric fitting, the two-dimensional recognition results output by the pre-trained large vision model can be transformed into stable three-dimensional geometric constraints, reducing the impact of two-dimensional recognition noise and point cloud sampling noise on registration.

[0130] After geometric fitting is completed, the geometric properties of the three-dimensional natural reference features are calculated. These geometric properties include at least one of length, curvature, principal direction, contour area, aspect ratio, array spacing, and endpoint spacing. These geometric properties are used for subsequent candidate matching and erroneous correspondence elimination of natural reference features between different stations.

[0131] In this embodiment, step S4 establishes candidate correspondences of natural reference features between different stations based on semantic type consistency, geometric consistency, local point cloud consistency, visual confidence and adjacency relationship consistency of the three-dimensional natural reference features, and eliminates erroneous correspondences through comprehensive confidence screening and global consistency verification.

[0132] Step S401: For 3D natural reference features from different stations, firstly, based on the semantic type labels of the 3D natural reference features, perform candidate matching of the same type for the natural reference features from different stations. Specifically, 3D natural reference features with the same semantic type labels are used as candidate matching objects to obtain candidate matching pairs. For example, window boundary features are matched with window boundary features, hatch outline features are matched with hatch outline features, and skin seam features are matched with skin seam features. Through semantic type constraints, erroneous matching between different types of natural reference features can be reduced.

[0133] Step S402: For any candidate matching pair obtained through matching of the same type of candidates ,in and These are the three-dimensional natural benchmark features from different stations, and the candidate matching pairs are used to... The semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence, and intra-station adjacency relationship are used to calculate the type consistency score, geometric consistency score, local point cloud consistency score, visual confidence score, and adjacency relationship consistency score, respectively.

[0134] The geometric attributes include at least one of length, curvature, principal direction, contour area, aspect ratio, array spacing, and endpoint spacing; the local neighborhood point cloud features include at least one of local normal statistics, neighborhood point cloud geometric descriptors, and local surface morphology; the visual confidence level is the confidence level of the two-dimensional natural reference feature corresponding to the three-dimensional natural reference feature, or the statistical value of the confidence level of the two-dimensional natural reference feature inherited by each three-dimensional point in the three-dimensional natural reference point set. The intra-station adjacency relationship is the spatial relationship between a three-dimensional natural reference feature and its neighboring three-dimensional natural reference features within the same station, including at least one of the relative distance, relative direction, arrangement order, and semantic type of the neighboring features.

[0135] Step S403: Based on the scores of each sub-item obtained in step S402, calculate the candidate matching pair. Comprehensive confidence of candidate matches :

[0136]

[0137] in, For type consistency score, For geometric consistency score, For local point cloud consistency score, Visual confidence score. The adjacency consistency score is used; the visual confidence score is based on candidate matching pairs. The visual confidence scores of two three-dimensional natural benchmark features are calculated, and the adjacency consistency score is based on the candidate matching pairs. The similarity of the adjacency relationship between the two three-dimensional natural reference features within the same station was calculated. , , , , The weight coefficients are non-negative and satisfy the following conditions: ;

[0138] In one implementation, the type consistency score Based on candidate matching pairs Whether the semantic type labels of two three-dimensional natural baseline features are consistent is determined as follows:

[0139]

[0140] in, and These are three-dimensional natural benchmark features. and Semantic type tags.

[0141] In one implementation, the geometric consistency score It is calculated based on the difference in normalized geometric attribute vectors of two three-dimensional natural benchmark features. Let the three-dimensional natural benchmark features be... and The normalized geometric attribute vectors are respectively and The normalized geometric attribute vector is composed of at least one of length, curvature, principal direction, contour area, aspect ratio, array spacing, and endpoint spacing.

[0142]

[0143] in, This is a scale parameter for the differences in geometric attributes. The closer the geometric attributes are, the higher the geometric consistency score.

[0144] In one implementation, the local point cloud consistency score It is calculated based on the difference between the point cloud descriptors in the neighborhood of two 3D natural benchmark features. For 3D natural benchmark features and The local neighborhood point cloud descriptors are obtained by statistically analyzing the local normal distribution, local curvature distribution, or point cloud geometric descriptors in the neighborhood point cloud. and ,but:

[0145]

[0146] in, This is a scale parameter for the differences in local point cloud descriptors. In this way, the more similar the morphology of the point clouds surrounding two 3D natural baseline features, the higher the local point cloud consistency score.

[0147] In one implementation, the visual confidence score It is calculated based on the visual confidence scores corresponding to two three-dimensional natural benchmark features. Let the three-dimensional natural benchmark features be... and The visual confidence levels are respectively and ,but:

[0148]

[0149] in, and It can be the detection or segmentation confidence of the corresponding two-dimensional natural reference feature, or it can be the statistical value of the two-dimensional confidence inherited by each three-dimensional point in the three-dimensional natural reference point set.

[0150] In one implementation, the adjacency consistency score The similarity is calculated based on the adjacency relationship between two three-dimensional natural benchmark features within their respective stations. Let the three-dimensional natural benchmark features be... and The adjacency vectors are respectively and The adjacency vector includes at least one of the following: relative distance, relative direction, arrangement order, and semantic type of the neighboring features between the three-dimensional natural reference feature and its neighboring natural reference features:

[0151]

[0152] in, This represents the scale parameter for differences in adjacency relationships. For aircraft surface areas with many repetitive structures, such as portholes, fastener arrays, and skin seams, consistency in adjacency relationships can further reduce the probability of mismatches between similar repetitive structures.

[0153] In one implementation, , , , , All values ​​are within the range [0,1]. For measurement scenarios with many repetitive structures on the aircraft surface, the weight corresponding to the adjacency consistency score can be increased; for measurement scenarios with high image quality and high confidence in natural reference feature recognition, the weight corresponding to the visual confidence score can be increased; for regions with obvious local geometric features in the point cloud, the weight corresponding to the geometric consistency score or the local point cloud consistency score can be increased. As an example, the weight can be set to... , , , , The value is merely an example and does not constitute a limitation on the scope of protection of this invention.

[0154] Step S404: Based on the comprehensive confidence level of the candidate matches The natural baseline feature correspondences are obtained through screening. Specifically, when the comprehensive confidence of candidate matches... If the number of candidates is not less than a preset threshold, retain the corresponding candidate matching pairs. The correspondence between natural baseline features is obtained; when the comprehensive confidence of candidate matching is... When the value is less than a preset threshold, the corresponding candidate matching pair is removed. ;

[0155] Step S405: Perform a global consistency check on the selected natural benchmark feature correspondences. Specifically, estimate the rigid body transformation between candidate stations based on multiple natural benchmark feature correspondences, and determine whether the matching residuals of each natural benchmark feature correspondence under the rigid body transformation between the candidate stations meet a preset residual threshold. If the matching residual of a certain natural benchmark feature correspondence does not meet the preset residual threshold, then the natural benchmark feature correspondence is determined to be an outlier and removed. The natural benchmark feature correspondences obtained after the above screening and consistency check are used for subsequent construction of geometric constraints between stations.

[0156] In this embodiment, step S5 constructs geometric constraints between stations based on the correspondence of the selected natural reference features, solves the rigid body transformation between different stations, completes the coarse registration of point clouds between stations, and performs local fine registration based on the coarse registration.

[0157] Step S501: Construct inter-station geometric constraints based on the selected correspondences of natural datum features. The geometric constraints include at least one of point-to-point constraints, point-to-line constraints, line-to-line constraints, and region-to-region constraints; for correspondences formed by key points of natural datums, construct point-to-point constraints; for correspondences formed by boundary lines, seam lines, or center lines, construct point-to-line constraints or line-to-line constraints; for correspondences formed by hatch areas, window areas, or engine nacelle outlines, construct region-to-region constraints.

[0158] Step S502: Based on the above-mentioned inter-station geometric constraints, solve the inter-station rigid body transformation using random sampling consistency, maximum consistency set, singular value decomposition, or robust least squares methods. The rigid body transformation includes rotation matrices. Translation vector ;

[0159] Step S503: Apply the rigid body transformation to the corresponding station cloud to obtain the coarse registration result between stations. For the first... 3D points in the cloud of individual test sites The transformed points Represented as:

[0160]

[0161] in, For the station to the station The rotation matrix, For the station to the station Translation vector;

[0162] Step S504: Based on the coarse registration results between stations, establish a local point cloud correspondence within the overlapping area of ​​the two station clouds according to the distance threshold and the normal angle threshold. Then, perform local fine registration of the station clouds using point-to-surface distance, normal consistency, and local surface residuals. Local fine registration is used to further reduce the point cloud residuals in the overlapping area between stations and improve the local alignment accuracy between adjacent station clouds. Specifically, establish a local point cloud correspondence within the overlapping area of ​​the two station clouds, and further optimize the rigid body transformation between stations by minimizing the point-to-surface distance and normal deviation.

[0163] Specifically, the source testing site cloud Inter-station rigid body transformations obtained from coarse registration Transform to the target station coordinate system. For any point in the transformed source station cloud... At the target test site cloud Find its nearest neighbor. .like and The distance between them is less than a preset distance threshold And the source point normal direction The angle between the rotated point and the target point's normal is less than a preset normal angle threshold. Then As a local point cloud correspondence within the overlapping region, it is added to the local point cloud correspondence set. .

[0164] In one embodiment, the distance condition and the normal condition are respectively expressed as:

[0165]

[0166] in, As the source The normal vector, For target point The normal vector.

[0167] Based on the local point cloud correspondence set Construct the local fine registration objective function:

[0168]

[0169] The first term is the distance from the point to the surface, and the second term is the normal deviation. This represents the weighting coefficient for the normal deviation term. The local point cloud correspondence set is updated iteratively. The local fine registration objective function is minimized using the Gauss-Newton method, the Levenberg-Marquardt method, or the robust least squares method, and then updated. and This process continues until the residual change between two adjacent iterations is less than a preset threshold or the maximum number of iterations is reached. Through this local fine registration, the point cloud residual in the overlapping area between stations can be further reduced, improving the local alignment accuracy between adjacent station clouds.

[0170] Step S505: After local fine registration is completed, determine whether the residual after local fine registration meets the preset accuracy requirements. If the preset accuracy requirements are met, the inter-station rigid body transformation relationship obtained after coarse registration and local fine registration is retained as a valid inter-station registration relationship; if the preset accuracy requirements are not met, the weight of the inter-station registration relationship in the subsequent global optimization is reduced, or the process returns to step S4 to re-select the correspondence of natural datum features and re-solve the inter-station rigid body transformation relationship.

[0171] In this embodiment, step S6 uses multiple station cloud data as graph nodes and the rigid body transformation relationships between stations as graph edges to construct a multi-viewpoint cloud global optimization graph. Weighted global optimization is then performed based on the natural datum feature matching residuals and point cloud geometric residuals to obtain the overall point cloud registration result for the large aircraft. For example... Figure 3 As shown, Figure 3 The upper part shows the state before multi-station cloud registration. Different station clouds have misalignment and discontinuous outlines in local areas of the aircraft. Figure 3 The lower part shows the state after registration with natural reference features, where the overlap of adjacent station clouds at the fuselage, wing connection area and cross-sectional profile is improved. Figure 3 The magnified area in the image is used to demonstrate the alignment changes of local point clouds on the aircraft surface before and after registration, while the cross-sectional profile on the right is used to demonstrate the consistency changes in the local cross-sectional shape before and after registration. Through the correspondence of natural reference features, the geometric relationship of locally overlapping point clouds, and global optimization of multi-view point clouds, multiple local measurement point clouds can be unified into the same coordinate system to form a large aircraft whole-aircraft point cloud registration result.

[0172] Step S601: Process the preprocessed cloud data of each test site obtained in step S103. As a node in the global optimization graph.

[0173] Step S602: Establish a graph edge between two stations that have a correspondence of natural reference features or overlapping point cloud regions. The graph edge represents the relative rigid body transformation relationship between the point clouds of the two stations.

[0174] Step S603: Determine the graph edge weights based on the natural datum feature matching residuals, point cloud geometric residuals, natural datum feature confidence, matching inlier ratio, and overlapping area. .

[0175] Specifically, for the measuring station and monitoring station The natural benchmark feature matching residuals are calculated based on the correspondences of the preserved natural benchmark features. For point-like natural benchmark features, the Euclidean distance of the corresponding point under the current inter-station transformation can be calculated; for linear natural benchmark features, the point-to-line distance or line-to-line distance can be calculated; for regional natural benchmark features, the residuals of the regional center, regional boundary, or regional fitting surface can be calculated. The above residuals are averaged or weighted averaged to obtain the natural benchmark feature matching residuals. .

[0176] The point cloud geometric residual is calculated based on the local point cloud correspondence within the overlapping area of ​​two survey station clouds after coarse registration or local fine registration. In one embodiment, the point cloud geometric residual can be obtained by averaging the point-to-surface distance or point-to-point distance of corresponding points within the overlapping area. .

[0177] The confidence score of natural benchmark features is obtained by statistically analyzing the comprehensive confidence score of candidate matches based on the correspondence between natural benchmark features preserved between corresponding stations. Let the stations be... and monitoring station The set of natural baseline feature correspondences preserved between them is Then the confidence level of the natural baseline feature can be expressed as:

[0178]

[0179] in, Candidate matching pairs The overall confidence level of candidate matches.

[0180] The matching inlier ratio is the ratio of the number of retained natural baseline feature correspondences to the number of candidate matching pairs obtained in step S401, expressed as:

[0181]

[0182] in, This represents the number of natural baseline feature correspondences retained after candidate matching comprehensive confidence screening and global consistency verification. This represents the number of candidate matching pairs obtained in step S401. When... At that time, the natural benchmark feature matching graph edges between corresponding stations are not established, or the corresponding graph edge weights are reset to zero.

[0183] The area of ​​the overlapping region is estimated based on the set of overlapping points whose distance is less than a preset distance threshold after coarse registration or local fine registration. Specifically, a nearest neighbor search is performed between the transformed source and target station clouds. If the distance between the point pairs is less than a preset distance threshold... If the corresponding points are found to be true, they are assigned to the overlapping point set. Subsequently, the overlapping point set can be projected onto a local tangent plane and the number of grid cells occupied can be counted, or the area of ​​the overlapping region can be estimated based on the local triangulation area of ​​the overlapping point set. To facilitate comparisons between sides of different stations, the area of ​​the overlapping region can be normalized as follows:

[0184]

[0185] in, and They are the stations and monitoring station The estimated effective coverage area of ​​the corresponding point cloud.

[0186] In one implementation, graph edge weights It can be represented as:

[0187]

[0188] in, Confidence level of natural baseline features To match the interior point ratio, To normalize the area of ​​the overlapping region, Match residuals to natural baseline features. For point cloud geometric residuals, and These are the scale parameters for the natural baseline feature matching residual and the point cloud geometric residual, respectively. Therefore, the higher the confidence level of the natural baseline feature, the higher the proportion of matched inliers, the larger the overlapping area, the smaller the natural baseline feature matching residual, and the smaller the point cloud geometric residual, the higher the corresponding graph edge weight. The larger.

[0189] Step S604: Perform weighted global optimization on the poses of multiple stations based on the graph edge weights to obtain the rigid body transformation of each station cloud to a unified coordinate system; the weighted global optimization aims to achieve the following objective function. Minimum as the target:

[0190]

[0191] in, For the set of edges of the globally optimized graph, For the station With the station The edge weights between them Match residuals to natural baseline features. For point cloud geometric residuals, This is the residual balance coefficient. and They are the stations and monitoring station Rigid body transformation to a unified coordinate system;

[0192] The natural benchmark feature matching residual Based on the correspondence of natural reference features in global rigid body transformation and The point cloud geometric residual is calculated from at least one of the following: point-to-point residual, point-to-line residual, line-to-line residual, or region-to-region residual; It is calculated based on at least one of the following: point-to-point distance, point-to-surface distance, or normal deviation between corresponding points within the overlapping area of ​​the two measuring stations.

[0193] Among them, in step S503 and Indicates the station to the station The relative rigid body transformation, in step S604 and This represents the global rigid body transformation from each station to a unified coordinate system. When and When representing the rigid body transformation from the cloud at each measuring station to a unified coordinate system, it is... and The station can be derived to the station The relative transformation. Global optimization reduces error accumulation during the continuous splicing process of multiple stations by constraining this relative transformation to be consistent with the relative rigid body transformation between stations obtained in step S503.

[0194] Step S605: Through the above weighted global optimization, a rigid body transformation from each station cloud to a unified coordinate system is obtained. Based on the rigid body transformation, multiple station clouds are stitched together to obtain the registration result of the entire large aircraft point cloud.

[0195] Step S606: After completing the registration of the point cloud of the large aircraft, calculate the registration residuals of different measurement point clouds in the overlapping area, and output the registration quality evaluation results. The registration quality evaluation results include at least one of the following: average residual, maximum residual, root mean square error, error heatmap, and low confidence region indication. For regions where the registration residual exceeds a preset threshold, they can be marked as low confidence regions for operators to review or perform supplementary scanning.

[0196] The present invention provides a point cloud registration method based on pre-trained large visual models for aircraft natural reference features. By transforming the natural reference features of the aircraft surface into three-dimensional geometric constraints, the dependence on manual targets and manual homonyms is reduced. Candidate matches are screened by semantic type consistency, geometric consistency, local point cloud consistency, visual confidence, and adjacency consistency of the three-dimensional natural reference features, which improves the registration stability in low-overlap and repetitive structure scenarios. Weighted global optimization of natural reference feature matching residuals and point cloud geometric residuals reduces the error accumulation in multi-measurement point cloud stitching. It is applicable to scenarios such as large aircraft manufacturing, assembly inspection, maintenance inspection, and overall aircraft shape quality assessment.

[0197] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0198] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for registration of point clouds constrained by natural reference features of aircraft based on a pre-trained large visual model, characterized in that, Includes the following steps: Step S1: Obtain aircraft point cloud data from multiple stations at corresponding viewpoints. and the point cloud data Corresponding image data For the point cloud data After preprocessing, the cloud data of the test site is obtained. Establish image data Cloud data of the testing site The correspondence between them; Step S2: Transfer the image data from each station. Input the pre-trained large vision model to obtain the visual feature map of the aircraft surface of the corresponding station, so as to detect or segment the natural reference features of the aircraft surface of the corresponding station and obtain the two-dimensional representation of the natural reference features of the corresponding station. Step S3: Based on image data Cloud data of the testing site The correspondence between them maps the two-dimensional representation of the natural reference features to the three-dimensional point cloud corresponding to each station. Through projection analysis and spatial clustering, the three-dimensional natural reference point set in each station is obtained. The three-dimensional natural reference point set is geometrically fitted to obtain the three-dimensional natural reference features of each station, and the geometric attributes of the three-dimensional natural reference features are calculated. Step S4: Based on the semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence and adjacency relationship of the three-dimensional natural reference features, calculate the comprehensive confidence of candidate matching of two three-dimensional natural reference features of different stations, and filter to obtain the correspondence of natural reference features. Step S5: Construct geometric constraints between stations based on the correspondence of the natural reference features, solve the rigid body transformation relationship between stations, and perform coarse registration and local fine registration of point clouds between stations; Step S6: Using multiple station cloud points as graph nodes and the rigid body transformation relationship between stations as graph edges, construct a multi-viewpoint cloud global optimization graph. Combine the natural reference feature matching residual and the point cloud geometric residual to perform weighted global optimization of the poses of multiple stations, and obtain the point cloud registration result of the large aircraft.

2. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, Step S1 includes the following process: Step S101: Deploy around the large aircraft Each station collects aircraft point cloud data from different perspectives using 3D measurement equipment. And to ensure that there is a preset overlap area between adjacent measuring station clouds, wherein For the first Aircraft point cloud data from the perspective of each station; Step S102: Synchronously acquire the point cloud data using a measuring device. Corresponding image data ; Step S103: Process the point cloud data Outlier removal, noise reduction, downsampling, and normal estimation are performed to obtain the station cloud data. ; Step S104: Establish image data based on the camera intrinsic parameters of the measuring equipment, the extrinsic parameters between the camera and the 3D measuring equipment, or the pixel-to-point cloud correspondence provided by the measuring equipment software development kit. Cloud data of the testing site The correspondence between them.

3. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 2, characterized in that, In step S102, when the measuring device cannot directly acquire two-dimensional image data, it uses the point cloud data... Generate a point cloud projection map as the corresponding image data. The point cloud projection map is at least one of a depth map, a normal map, a curvature map, and a reflection intensity map.

4. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, The natural reference features mentioned in step S2 include at least one of the following: window boundary, door outline, skin seam, wing leading edge, wing trailing edge, wing-body junction, engine nacelle outline, tail edge, radome boundary, and fastener array; the two-dimensional representation of the natural reference features includes at least one of the following: two-dimensional mask, two-dimensional boundary, two-dimensional centerline, and two-dimensional key point, as well as the type label and confidence level corresponding to the two-dimensional representation.

5. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, Step S3 includes the following sub-steps: Based on image data Cloud data of the testing site The correspondence between them will be the first Mapping the two-dimensional mask, two-dimensional boundary, two-dimensional centerline, or two-dimensional key points of the natural benchmark features of each station to the first station. In the 3D point cloud corresponding to each station; For cloud data at testing sites 3D points According to the camera intrinsic parameter matrix and extrinsic parameter matrix Calculate its projected pixel coordinates The relationship is as follows: ； in, As a scale factor, For the first The intrinsic parameter matrix of each station corresponding to the camera. and These are the rotation matrix and translation vector between the camera and the 3D measurement device, respectively. For three-dimensional points The pixel coordinates after projection onto the image plane; Determine the projected pixel coordinates Whether the point falls within the neighborhood of a 2D mask, 2D boundary, 2D centerline, or 2D keypoint of the natural reference feature; if so, assign the corresponding 3D point to the next 3D keypoint. Each station has a corresponding set of candidate 3D natural reference points, and the type label and confidence level of the natural reference features are assigned to the 3D point; the type label includes a semantic type label and a geometric type label. Spatial clustering is performed on the 3D points in the candidate 3D natural reference point set. 3D points with the same semantic type label and spatial distance satisfying the preset neighborhood condition are grouped into the same 3D natural reference point set, and isolated point sets with fewer than a preset threshold are removed to obtain the first set. A set of three-dimensional natural reference points for each station; Geometric fitting is performed based on the geometric morphology type corresponding to the three-dimensional natural reference point set to obtain the first... Three-dimensional natural benchmark features of each station; Calculate the geometric properties of the three-dimensional natural reference feature, including at least one of length, curvature, principal direction, contour area, aspect ratio, array spacing, and endpoint spacing.

6. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 5, characterized in that, The geometric shape type includes at least one of closed contour type, linear type and array type; For closed contour-type natural datum features, a closed three-dimensional boundary curve is fitted and used as a three-dimensional natural datum feature; For linear natural reference features, a three-dimensional spatial curve is fitted and used as the three-dimensional natural reference feature. For array-type natural reference features, the array centerline or array direction vector is fitted and used as a three-dimensional natural reference feature.

7. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, Step S4 includes the following sub-steps: Step S401: Based on the semantic type labels of the three-dimensional natural reference features, perform same-type candidate matching on the three-dimensional natural reference features in different stations, and take the three-dimensional natural reference features with the same semantic type labels as candidate matching objects to obtain candidate matching natural reference features. Step S402: For any candidate matching pair obtained in step S401 ,in and These are the three-dimensional natural benchmark features from different stations, and the candidate matching pairs are used to... The semantic type label, geometric attributes, local neighborhood point cloud features, visual confidence, and intra-station adjacency relationship are used to calculate the type consistency score, geometric consistency score, local point cloud consistency score, visual confidence score, and adjacency relationship consistency score, respectively. Step S403: Based on the scores of each sub-item obtained in step S402, calculate the candidate matching pair. Overall confidence of candidate matches: ； in, For type consistency score, For geometric consistency score, For local point cloud consistency score, Visual confidence score. Scoring is given based on the consistency of adjacency relationships; , , , , The weight coefficients are non-negative and satisfy the following conditions: ; Step S404: Based on the comprehensive confidence level of the candidate matches The natural baseline feature correspondence is obtained through screening: when the comprehensive confidence of candidate matches... If the number of candidates is not less than a preset threshold, retain the corresponding candidate matching pairs. The correspondence between natural reference features is obtained; When the overall confidence level of candidate matching When the value is less than a preset threshold, the corresponding candidate matching pair is removed. ; Step S405: Perform global consistency verification on the natural benchmark feature correspondence obtained by screening: estimate the rigid body transformation between candidate stations based on multiple natural benchmark feature correspondences, and determine whether the matching residual of each natural benchmark feature correspondence under the rigid body transformation between the candidate stations meets the preset residual threshold. If it does not meet the preset residual threshold, the corresponding natural benchmark feature correspondence is determined to be an outlier correspondence and is removed.

8. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 7, characterized in that, The type consistency score is: ； in, and These are three-dimensional natural baseline features. and semantic type tags; Geometric consistency score: ； in, The scale parameter for the difference in geometric properties. and These are three-dimensional natural baseline features. and Normalized geometric attribute vector; The local point cloud consistency score is: ； in, The scale parameter for the differences in local point cloud descriptors; and Three-dimensional natural reference features and Local neighborhood point cloud descriptor; The visual confidence score is: ； in, and These are three-dimensional natural baseline features. and Visual confidence; The adjacency consistency score is: ； in, This is the scale parameter for the difference in adjacency relationships. and These are three-dimensional natural baseline features. and The adjacency vector.

9. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, Step S5 includes the following sub-steps: Step S501: Construct inter-station geometric constraints based on the correspondence of the natural reference features. The geometric constraints include at least one of point-to-point constraints, point-to-line constraints, line-to-line constraints, and region-to-region constraints. Step S502: Based on the inter-station geometric constraints, solve the rigid body transformation relationship between the stations, including the rotation matrix. Translation vector ; Step S503: Apply the rigid body transformation relationship to the corresponding station cloud to obtain the coarse registration result between stations; For the 3D points in the cloud of individual test sites The transformed points Represented as: ； in, For the station To the station The rotation matrix, For the station To the station Translation vector; Step S504: Based on the coarse registration results between stations, establish a local point cloud correspondence within the overlapping area of ​​the two station clouds according to the distance threshold and the normal angle threshold, and construct a local fine registration objective function using the point-to-surface distance and normal deviation, iteratively optimizing the rigid body transformation between stations; the local fine registration objective function is: ； in, For the distance from a point to a surface, For the normal deviation term, This is the weighting coefficient for the normal deviation term; It is a set of local point cloud correspondences; As the source The normal vector, For target point The normal vector; Step S505: Determine whether the residual after local fine registration meets the preset accuracy requirements. If it meets the preset accuracy requirements, retain the inter-station rigid body transformation relationship obtained after coarse registration and local fine registration as a valid inter-station registration relationship. If it does not meet the preset accuracy requirements, reduce the weight of the inter-station registration relationship in the subsequent global optimization, or return to step S4 to re-select the correspondence of natural datum features and re-solve the inter-station rigid body transformation relationship.

10. The method for registration of point clouds constrained by natural reference features of an aircraft based on a pre-trained large visual model according to claim 1, characterized in that, Step S6 includes the following sub-steps: Step S601: Use the preprocessed cloud data of each test site as a node in the global optimization graph; Step S602: Based on the rigid body transformation relationship between stations, establish graph edges between two stations that have a natural benchmark feature correspondence or point cloud overlapping area to obtain a multi-view point cloud global optimization map. Step S603: Match residuals based on natural benchmark features Point cloud geometric residuals Confidence of natural baseline features Matching interior point ratio and overlapping area Determine the edge weights of the graph ; and These are the scale parameters for the natural baseline feature matching residual and the point cloud geometric residual, respectively; Step S604: Based on the global optimization graph and graph edge weights So that the following objective function Weighted global optimization of the poses of multiple stations is performed with the goal of minimizing the position of each station, resulting in the rigid body transformation of each station's geocloud to a unified coordinate system. ； in, For the set of edges of the globally optimized graph, For the station With the station The edge weights between them Match residuals to natural baseline features. For point cloud geometric residuals, This is the residual balance coefficient. and They are the stations and monitoring station Rigid body transformation to a unified coordinate system; Step S605: Based on the rigid body transformation of each measuring station cloud to a unified coordinate system, stitch together the multiple measuring station clouds to obtain the registration result of the large aircraft's whole point cloud. Step S606: Calculate the registration residuals of different station clouds in the overlapping area and output the registration quality evaluation results, including at least one of the following: average residual, maximum residual, root mean square error, error heatmap, and low confidence region indication.

Images