A flexible combined three-dimensional measurement method and device for the overall appearance of a large aircraft

Abstract

The present invention relates to a flexible combined three-dimensional measurement method and device for the overall appearance of a large aircraft, including establishing a base station to obtain the spatial position and posture of a three-dimensional laser scanner; using a three-dimensional measurement device to enable a three-dimensional laser scanner to scan and collect local three-dimensional point cloud data of the large aircraft; using a rough registration method based on base station positioning to roughly register the collected local three-dimensional point cloud data; performing voxel-based sampling on each local three-dimensional point cloud data after the rough registration, and calculating the overlap between any two point clouds, and extracting the maximum spanning tree according to the overlap; proposing a deep learning algorithm "predator" to accurately register the obtained maximum spanning tree; calculating the overlap between any two point clouds again, constructing a maximum spanning tree according to the overlap, and using a traditional algorithm ICP for refined registration. The present invention realizes flexible and intelligent measurement of aircraft overall data, efficient and accurate measurement of the overall appearance of large aircraft, and meets the precision requirements of the overall appearance measurement of large aircraft.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional data measurement of the overall shape of a large aircraft, and in particular to a flexible combined three-dimensional measurement method and device for the overall shape of a large aircraft. Background Art

[0002] The most essential feature of large aircraft is high quality, including high performance, high stability and high reliability. Achieving these high quality standards requires a lot of precise means, among which appearance inspection can help improve the manufacturing efficiency and flight safety of large aircraft, and is an indispensable part of large aircraft processing, assembly and daily maintenance.

[0003] In digital engineering technology, advanced 3D scanning equipment can be used to acquire geometric data of the aircraft shape, and then the geometric data analysis can be completed based on the definition of the aircraft digital model. However, the shape features of large aircraft are complex, and the scanning and calculation accuracy requirements are strict. At present, both at home and abroad, there are still problems such as the difficulty in completely scanning the entire aircraft surface (47m×50m×15m) and the inability to guarantee high precision requirements (0.001-0.5mm). The existing large aircraft shape 3D measurement system often lacks sufficient flexibility and is difficult to adapt to the ever-changing measurement needs in the large aircraft manufacturing process. It is difficult to balance the flexibility and accuracy of the measurement.

[0004] In response to this challenge, a new method and device is urgently needed to ensure measurement accuracy while improving system adaptability and measurement efficiency, so as to meet the needs of high-precision and high-efficiency measurement in large aircraft overall shape measurement. A high-precision flexible global automated measurement method for large aircraft is proposed, which solves the problem of low automation and low efficiency in large-scale component measurement caused by the solidification of existing measurement field patterns and low visibility of reference points. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a flexible combined three-dimensional measurement method and device for the overall appearance of a large aircraft, which solves the problem of low automation and low efficiency in the measurement of large-size components caused by the rigidification of the existing measurement field pattern and low visibility of the reference points. The method obtains local three-dimensional point cloud data of the large aircraft through a flexible combination of measurement systems; uses a coarse alignment method based on base station positioning to align the collected point cloud data; processes the point cloud data after coarse alignment, simplifies the point cloud and calculates the overlap, and constructs a maximum spanning tree; proposes a deep learning algorithm "Predator" to accurately align the processed point cloud data with the overlap, and finally, based on the better initial pose obtained by the accurate alignment, uses the iterative closest point method (Iterative Closest Points, ICP) to complete the refined alignment, achieves high-precision alignment of each local point cloud data, and completes the overall appearance measurement of the aircraft.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a flexible combined three-dimensional measurement method for the overall appearance of a large aircraft, comprising the following steps:

[0007] S1. Establish a base station to track and locate the 3D laser scanner in the measurement space in real time and obtain the spatial position of the 3D laser scanner;

[0008] S2, using a three-dimensional measuring device to enable a three-dimensional laser scanner to scan and collect data of the large aircraft from multiple angles, and obtain local three-dimensional point cloud data of the upper and lower surfaces of the large aircraft;

[0009] S3, using a coarse registration method based on base station positioning to coarsely register the collected local three-dimensional point cloud data;

[0010] S4, performing voxel-based sampling on each local three-dimensional point cloud data after rough registration, calculating the overlap between any two point clouds, and extracting the maximum spanning tree according to the overlap;

[0011] S5. A deep learning algorithm “Predator” is proposed to precisely align the obtained maximum spanning tree.

[0012] S6. Calculate the overlap between any two point clouds again, construct a maximum spanning tree based on the overlap, and use the traditional ICP algorithm for refined registration.

[0013] Furthermore, in step S1, a base station is established, specifically including: arranging a plurality of binocular cameras in the entire measurement space to form a base station, covering the envelope space of the large aircraft being measured.

[0014] Further, in step S2, the three-dimensional measuring device specifically includes:

[0015] Binocular cameras, used to locate the position of the 3D laser scanner in real time, are distributed throughout the measurement space to form base stations;

[0016] Industrial robotic arms, used to carry short-range terminal measurement devices to collect point cloud data of large aircraft using a 3D laser scanner;

[0017] The lifting mechanism is used for data collection on the upper surface of large aircraft. An industrial robotic arm is installed at the lower end of the lifting mechanism, and the height of the industrial robotic arm is adjusted by adjusting the extension degree;

[0018] AGV trolley is used for collecting data from the lower surface of large aircraft. In this case, the industrial robot arm is carried by the AGV trolley, driving the 3D laser scanner to move on the ground;

[0019] Intelligent terminal platform, used to send automation equipment control instructions, control base stations and 3D laser scanners for data collection.

[0020] Furthermore, the lifting mechanism specifically includes: a lifting mechanism active motor, a lifting mechanism frame body, a lifting mechanism line frame and a crossbeam assembly for ensuring the stability and anti-winding of the line, wherein:

[0021] The crossbeam assembly includes a line, a rotating shaft, a line frame, a driven wheel of the line frame, a limit key of the line frame, a lifting mechanism trolley, a driving wheel of the lifting mechanism trolley, a crossbeam, a driving trolley and a turntable. One end of the crossbeam is connected to the rotating shaft of the gantry, and the other end is connected to the driving trolley. The driving trolley is installed on the turntable, and the driving trolley carries the crossbeam to rotate around the rotating shaft;

[0022] The upper end of the lifting mechanism is connected to the lifting mechanism trolley of the crossbeam, and the lifting mechanism trolley can move along the inner rail of the crossbeam, making the scanning more comprehensive and flexible;

[0023] The line is suspended on several line racks, and the lifting mechanism trolley moves driven by the driving wheel of the lifting mechanism trolley while stretching the line rack to make parallel movement.

[0024] Furthermore, S3 specifically includes: knowing the internal and external parameters of the binocular camera deployed by the base station, roughly unifying the local three-dimensional point cloud data obtained by the three-dimensional laser scanner into the global coordinate system according to the camera parameters and the spatial posture of the three-dimensional laser scanner, and completing the rough alignment.

[0025] Furthermore, in step S4, the specific process includes the following steps:

[0026] S41, in the process of voxel-based sampling, traverse the entire 3D point cloud data set, for each point that has not been marked as deleted, first convert the 3D point cloud data into a 3D voxel space, and select the voxel center in each voxel as a sampling point;

[0027] S42, in the process of calculating the overlap between point clouds, for any two point clouds, find the nearest neighbor point pair between the two, calculate the distance between the matching point pairs, set a distance threshold, and count the number of point pairs between the matching point pairs whose distance is less than the threshold. The overlap is defined as the ratio of the number of point pairs whose distance is less than the threshold to the total number of points in the source point cloud;

[0028] S43. A weighted fully connected graph is constructed according to the overlap between each pair of point clouds. Each point cloud constitutes a node of the graph. The overlap between two point clouds represents the weight of the edge between the nodes. Then, a maximum spanning tree is constructed according to the weighted fully connected graph.

[0029] Furthermore, in step S5, the specific process includes the following steps:

[0030] S51. For any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P, and the other is used as the target point cloud Q;

[0031] S52, the source point cloud P and the target point cloud Q are respectively input into the encoder, which contains multiple residual blocks and cross-layer convolution blocks. The encoder downsamples the two point clouds into coarse-scale super point sets P' and Q', and extracts the potential feature code X related thereto. P′ and X Q′ ;

[0032] S53, coarse-scale superpoint sets P' and Q' and latent feature encoding X P′ and X Q′ The global context information aggregation module with geometry perception and position perception is input, the global information between the same frames is learned in a rotation-invariant way through the geometry-aware self-attention module, and the cross-frame global interaction is realized through the position-aware cross-attention module, and the deep conditional feature F of the super point is output. P′ and F Q′ , overlap fraction o P′ and Q′ and the cross-overlap score and

[0033] S54, the decoder decodes the two sets of super-point information into a detailed point-by-point description. For the source point cloud P, first convert F P′ With o P′ , The concatenation is then performed into a decoder structure that combines upsampling and linear layers and includes cross-layer connections from the encoder of the first module. The upsampling is done by querying the relevant features of the nearest point from the previous layer. Finally, the decoder outputs the refined point-by-point features F P and F Q , overlap fraction o P and Q And the matching score m P and m Q ;

[0034] S55, probabilistic sampling RANSAC solves the rigid transformation, uses the product of the overlap score o and the matching score m to bias the sampling of the interest points, then estimates the correspondence based on the feature matching from the two point cloud sampling points, and finally uses singular value decomposition to solve the translation matrix and rotation matrix between the source point cloud P and the target point cloud Q;

[0035] S56, "Predator" involves three losses and ground-truth correspondences to supervise the training process, including three loss functions: 1) Circle loss, for the source point cloud P, first extract the point p that has at least one corresponding relationship in Q i , corresponding point positive example set ε p (p i ) is defined as i Nearby radius r pPoints inside, negative example set ε n (p i ) is defined as the s Point sets other than circle loss Where Δ n ,Δ p Indicates the positive and negative margins of hyperparameters, n p represents the number of sampling points, represents the distance in the feature space, and is the weight obtained for each positive and negative example, and the loss of point cloud Q in the opposite direction The calculation is similar; 2) Overlap loss, for point cloud P, use overlap loss Supervision, the ground-truth label in Eq. Defined as Where r o represents the overlap threshold, represents the basis rigid transformation, The calculation method is similar; 3) Matching loss, for point cloud P, the loss function is The base tag for Where r m represents the matching threshold, NN F represents the nearest neighbor search in the feature space, The calculation is similar; the total round loss is based on The total overlap loss is obtained according to The total matching loss is obtained according to We get the total loss by adding the three together;

[0036] S57. After the proposed "predator" obtains the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree, any point cloud is selected as a reference from all local three-dimensional point cloud data, and the remaining local three-dimensional point cloud data is converted to the reference system of the reference three-dimensional point cloud data according to the rotation and translation relationship contained in the edge established in the spanning tree, thereby completing the precise alignment of the three-dimensional point cloud data of the entire machine.

[0037] Furthermore, in step S53, the specific process includes the following steps:

[0038] S531, a self-attention module with geometric perception, which includes a geometric branch and a context branch; the geometric branch embeds G' from the geometric structure information P Mining geometric clues in the process can be expressed as:

[0039] G′ P=CAT[ρ i,j ,η i,j ,α i,j ];

[0040] Among them, CAT represents the splicing operation. For the super point p i and p j , described as the distance ρ between two points in Euclidean space i,j =||p i -p j ||2; for p i The two nearest neighbors and p j The two nearest neighbors Described as the local triangle perimeter difference In the formula is the Euclidean distance between two points; for p j and p i The three nearest neighbors and Calculate the vector p j -p i and and Three points form the normal vector of the plane and the local plane normal related angle Calculate angles;

[0041] In the geometry branch, we first have two learnable matrices W' G and W' E From G' P Linear projection obtains the geometric context G'=W' G G' P and its position code E'=W' E G' P ;

[0042] In the context branch, the latent feature encoding X of the input P' And the corresponding learnable matrix W' Q , W' K and W' V Linear mapping gives Q'=W' Q X P' , K'=W' K X P' , V'=W' V X P' , mixed attention weight parameter In order to prevent the inner product from being too large, the final self-attention output super-point conditional feature FP′ =Feed Forward(A'G')+Feed Forward(A'V') is composed of the sum of two branches, softmax is a normalized exponential function, and Feed Forward is a feedforward neural network;

[0043] S532, according to the input super point condition feature F P′ and F Q′ , first according to the learnable matrix and Project to and Finally output the updated features

[0044] S533, after three times of self-attention and cross-attention are completed alternately, the obtained features are linearly projected into the overlap score o P′ and Q′ , and get the cross-overlap score. For the coarse-scale superpoint set P', the cross-overlap score <·,·> represents the inner product operation, t represents the temperature hyperparameter, The calculation is similar.

[0045] Furthermore, in step S6, the specific process includes the following steps:

[0046] S61, repeat steps S42 and S43;

[0047] S62. For any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P, and the other is used as the target point cloud Q;

[0048] S63. Obtain point q from target point cloud Q i ∈Q;

[0049] S64. Search for point p in the source point cloud P i , so that the distance ||p i -q i ||min;

[0050] S65, by calculating the translation matrix T and the rotation matrix R, the error function Reach the minimum value, where n is the number of corresponding point pairs involved in the calculation;

[0051] S66: According to the translation matrix T and rotation matrix R obtained in step S64, the point q in the target point cloud is i Transform and update the corresponding superpoint set Q'{q i '=Rq i +T,q i ∈Q};

[0052] S67, calculate the midpoint q of the updated superpoint set Q' i 'with p i The average distance

[0053] S68, if d is less than a preset threshold or the number of iterations reaches an upper limit, stop the loop iteration, otherwise return to step S64;

[0054] S69. After obtaining the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree by the ICP algorithm, arbitrarily select a point cloud as a reference, and according to the rotation and translation relationship contained in the edges established in the spanning tree, convert the remaining local three-dimensional point cloud data into the reference system where the reference three-dimensional point cloud data is located, so as to complete the refined alignment of the whole machine three-dimensional point cloud data.

[0055] By means of the above technical solution, the present invention provides a flexible combined three-dimensional measurement method and device for the overall appearance of a large aircraft, which has at least the following beneficial effects:

[0056] The flexible combined three-dimensional measurement method for the overall appearance of a large aircraft of the present invention adopts a flexible combined method to flexibly and accurately measure the overall data of a large aircraft, thereby solving the problem of inefficient data collection caused by the huge size of components of a large aircraft, overcoming the limitations of a single device in terms of measurement range, measurement accuracy, and sampling density, and realizing information complementarity and optimizing measurement results; in terms of device, the crossbeam active component drives the trolley, compared to the current gantry-type scanning device, and there is no need to solve the synchronization of the movement of the trolleys on both sides or the design of the transmission mechanism in terms of movement; the rotational movement of the crossbeam with the crossbeam shaft as the axis does not require a line telescopic mechanism to be set on the turntable on the line, compared to the current gantry-type scanning device. The line only needs to be connected to the steel frame on the upper part of the crossbeam shaft. When scanning the upper surface, the lifting mechanism, beam, industrial robot arm and other devices are integrated. The beam rotates around the axis under the drive of the trolley, and the lifting mechanism can move along the inner rail of the beam; when scanning the lower surface, the AGV trolley is used to carry the industrial robot arm and move freely on the ground, so that the short-range terminal measurement device-the 3D laser scanner can flexibly collect data from multiple angles; during the registration process, due to the complex shape and structure of large aircraft, in order to achieve high-precision docking and integration of multiple local point clouds, the registration is carried out from coarse to fine and then refined, combining the coarse registration method based on base station positioning, the proposed deep learning fine registration algorithm "Predator" and the refined registration algorithm based on ICP. After the coarse registration, the voxel downsampling of each local 3D point cloud data is performed, and the overlap between the point clouds is calculated; the graph structure is constructed according to the overlap, the maximum spanning tree is extracted, and then the "Predator" is proposed for the registration. The global information aggregation module in the "predator" aggregates global information in a rotationally invariant manner, and has the same-frame and cross-frame spatial perception characteristics, and pays great attention to the overlapping area. The strategy of using "self-attention-cross-attention" to execute three times can improve the saliency of the features. The geometric structure information embedding proposed in the self-attention module enables it to have geometric information perception ability, which ensures the accuracy of the precise registration. The overlap is combined during registration. This method gives priority to point clouds with large overlap, which greatly reduces the error generation. After precise registration, the maximum spanning tree is extracted again, and refined registration based on ICP is performed. If the initial position is unreasonable, the ICP algorithm will fall into the local optimum, and the "predator" provides a better initial posture for ICP, which ensures the accuracy of ICP refined registration, and finally rotates and translates each local point cloud to the coordinate system of the reference point cloud. The present invention realizes the flexible, accurate and intelligent measurement of the whole machine data of large aircraft, assists in improving the whole machine detection efficiency of large aircraft, and thus ensures the operation performance and safety performance of large aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0058] Figure 1 This is a flow chart of the flexible combined three-dimensional measurement method for the overall appearance of a large aircraft of the present invention;

[0059] Figure 2 This is a front view of the flexible combined three-dimensional measuring device for the overall appearance of a large aircraft according to the present invention;

[0060] Figure 3 A top view of the flexible combined three-dimensional measuring device for the overall appearance of a large aircraft according to the present invention;

[0061] Figure 4 It is the crossbeam and lifting mechanism structure of the flexible combined three-dimensional measuring device for the overall appearance of a large aircraft of the present invention;

[0062] Figure 5 It is the "predator" deep neural network structure in the flexible combined three-dimensional measurement method of the large aircraft overall appearance of the present invention;

[0063] Figure 6 This is the self-attention structure of the "predator" deep neural network in the flexible combined three-dimensional measurement method for the entire large aircraft shape of the present invention.

[0064] In the attached drawings: 1. binocular camera; 2. industrial robot arm; 3. 3D laser scanner; 4. lifting mechanism; 41. lifting mechanism driving motor; 42. lifting mechanism frame body; 43. lifting mechanism circuit frame; 44. beam assembly; 441. circuit; 442. rotating shaft; 443. circuit frame; 444. circuit frame driven wheel; 445. circuit frame limit key; 446. lifting mechanism trolley; 447. lifting mechanism trolley driving wheel; 448. beam; 449. driving trolley; 450. turntable; 5. AGV trolley; 6. gantry. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0066] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0067] Please refer to Figure 1-Figure 6, showing a specific implementation of this embodiment. In this embodiment, a three-dimensional measurement device is used to enable a three-dimensional laser scanner to scan multiple angles to comprehensively and flexibly collect large aircraft data. During the registration process, due to the complex external structure of the large aircraft, in order to achieve high-precision docking and integration of multiple local point clouds, a registration method from coarse to fine and then to fine is adopted, thereby achieving flexible, accurate and intelligent measurement of the entire aircraft data, assisting in improving the overall inspection efficiency of the large aircraft, and thus ensuring the operating performance and safety performance of the large aircraft.

[0068] Please refer to Figure 1 This embodiment proposes a flexible combined three-dimensional measurement method for the overall appearance of a large aircraft, the method comprising the following steps:

[0069] S1. Establish a base station to track and locate the 3D laser scanner in the measurement space in real time and obtain the spatial position of the 3D laser scanner;

[0070] As a preferred implementation of step S1, in step S1, a base station is established, specifically including: arranging a plurality of binocular cameras in the entire measurement space to form a base station, covering the envelope space of the large aircraft being measured.

[0071] In this embodiment, the binocular camera tracks and locates the three-dimensional laser scanner in the measurement space in real time, obtains the spatial position and posture of the three-dimensional laser scanner, and lays the foundation for subsequent rough alignment.

[0072] S2, using a three-dimensional measuring device to enable a three-dimensional laser scanner to scan and collect data of the large aircraft from multiple angles, and obtain local three-dimensional point cloud data of the upper and lower surfaces of the large aircraft;

[0073] As a preferred implementation of step S2, Figure 2 As shown, in step S2, the three-dimensional measuring device specifically includes:

[0074] Binocular cameras 1 are used to locate the position of the 3D laser scanner in real time and are distributed throughout the measurement space to form a base station; each binocular camera 1 can track multiple 3D laser scanners 3 in the measurement space in real time, and the layout of each camera covers the envelope space of the measured target, ensuring the accuracy of data collection;

[0075] Industrial robot arm 2, a 3D laser scanner 3 for carrying a short-range terminal measuring device to collect point cloud data of large aircraft; the 3D laser scanner 3 measures by laser triangulation method, with the laser light source as the indicator light source for measurement, projecting the modulated mesh line laser onto the measured surface, which is captured by the CCD camera at the same time, and the point cloud data of the measured surface can be solved by the geometric optical calculation method;

[0076] The lifting mechanism 4 is used for collecting data on the upper surface of the large aircraft. The industrial robot arm 2 is installed at the lower end of the lifting mechanism 4. The height of the industrial robot arm 2 is adjusted by adjusting the telescopic degree;

[0077] The AGV trolley 5 is used for collecting data on the lower surface of the large aircraft. At this time, the industrial robot arm 2 is carried by the AGV trolley 5, driving the three-dimensional laser scanner 3 to move freely on the ground, and flexibly and comprehensively collects data on the lower surface of the large aircraft; the AGV trolley 5 can autonomously navigate and perform measurement tasks without human intervention, and move along a preset path, thereby reducing labor costs and improving measurement flexibility;

[0078] The intelligent terminal platform is used to send automation equipment control instructions to make the equipment move along a predetermined path. Every time a measurement site is reached, the software will send a measurement trigger signal to control the base station and the three-dimensional laser scanner 3 to collect data, and then fuse the collected data.

[0079] More specifically, if Figure 3 and Figure 4 As shown, the lifting mechanism 4 specifically includes: a lifting mechanism active motor 41, a lifting mechanism frame body 42, a lifting mechanism line frame 43 for ensuring the stability and anti-winding of the line, and a crossbeam assembly 44; wherein:

[0080] The crossbeam assembly 44 includes a line 441, a rotating shaft 442, a line frame 443, a line frame driven wheel 444, a line frame limit key 445, a lifting mechanism trolley 446, a lifting mechanism trolley driving wheel 447, a crossbeam 448, a driving trolley 449 and a turntable 450. One end of the crossbeam 448 is connected to the rotating shaft 442 of the gantry 6, and the other end is connected to the driving trolley 449. The driving trolley 449 is installed on the turntable 450, and the driving trolley 449 carries the crossbeam 448 to rotate around the rotating shaft 442; the line frame limit key 445 ensures the one-way line accuracy of the line frame 443, and the line is connected from the upper steel frame of the rotating shaft 442;

[0081] The upper end of the lifting mechanism 4 is connected to the lifting mechanism trolley 446 of the cross beam 448, and can move along the inner rail of the cross beam 448 when carried by the lifting mechanism trolley 446, so that the scanning is more comprehensive and flexible;

[0082] Accordingly, in order to ensure the signal transmission and energy input of the lifting mechanism trolley driving wheel 447, the lifting mechanism driving motor 41, the industrial robot arm 4 and the three-dimensional laser scanner 3, the line 441 is suspended on several line racks 443, and the lifting mechanism trolley 446 moves driven by the lifting mechanism trolley driving wheel 447 while stretching the line rack 443 to make parallel movement.

[0083] In this embodiment, for data collection on the upper surface of a large aircraft, a short-range terminal measuring device, a three-dimensional laser scanner 3, is carried by an industrial robot arm 2, which is mounted on a lifting mechanism 4. The lifting mechanism 4 moves left and right along the inner rail of a crossbeam 448. The end of the crossbeam 448 is connected to a driving trolley 449, which can rotate around an axis along a turntable 450 under the drive of the driving trolley 449, thereby driving the free movement of the entire structure, and realizing the multi-angle flexible scanning of the upper surface of the fuselage, wings, and tail wing by the three-dimensional laser scanner 3;

[0084] For data collection of the lower surface of a large aircraft, the three-dimensional laser scanner 3 is carried by an industrial robot arm 2, and they move on the ground under the support of an AGV trolley 5 to complete the work of collecting the lower surface data.

[0085] S3, using a coarse registration method based on base station positioning to coarsely register the collected local three-dimensional point cloud data;

[0086] As a preferred implementation of step S3, S3 specifically includes: knowing the internal and external parameters of the binocular camera deployed by the base station, and roughly unifying the local three-dimensional point cloud data obtained by the three-dimensional laser scanner 3 into the global coordinate system according to the camera parameters and the spatial posture of the three-dimensional laser scanner 3 to complete the rough alignment.

[0087] S4, performing voxel-based sampling on each local three-dimensional point cloud data after rough registration, calculating the overlap between any two point clouds, and extracting the maximum spanning tree according to the overlap;

[0088] As a preferred implementation of step S4, in step S4, the specific process includes the following steps:

[0089] S41, in the process of voxel-based sampling, traverse the entire 3D point cloud data set, for each point that has not been marked as deleted, first convert the 3D point cloud data into a 3D voxel space, and select the voxel center in each voxel as a sampling point;

[0090] S42, in the process of calculating the overlap between point clouds, for any two point clouds, find the nearest neighbor point pair between the two, calculate the distance between the matching point pairs, set a distance threshold, and count the number of point pairs between the matching point pairs whose distance is less than the threshold. The overlap is defined as the ratio of the number of point pairs whose distance is less than the threshold to the total number of points in the source point cloud;

[0091] S43. A weighted fully connected graph is constructed according to the overlap between each pair of point clouds. Each point cloud constitutes a node of the graph. The overlap between two point clouds represents the weight of the edge between the nodes. Then, a maximum spanning tree is constructed according to the weighted fully connected graph.

[0092] In this embodiment, for the local point cloud data of large aircraft with messy and uneven distribution, a voxel-based sampling algorithm is used to effectively achieve uniform sampling of point clouds, improve the efficiency of subsequent steps, and ensure the accuracy of analysis and processing. The overlap between any two point clouds is calculated and the maximum spanning tree is constructed, which provides the necessary conditions for the subsequent precise registration process. In actual measurement, the voxel-based sampling method is used to process the local point cloud data of large aircraft to perform point cloud registration operations.

[0093] S5. A deep learning algorithm “Predator” is proposed to precisely align the obtained maximum spanning tree.

[0094] As a preferred implementation of step S5, Figure 5 As shown, in step S5, the specific process includes the following steps:

[0095] S51, such as Figure 6 As shown, for any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P and the other is used as the target point cloud Q;

[0096] S52, the source point cloud P and the target point cloud Q are respectively input into the encoder, which contains multiple residual blocks and cross-layer convolution blocks. The cross-layer connection between the decoders can assist in the subsequent accurate dense feature description. The encoder downsamples the two point clouds into coarse-scale super point sets P' and Q', and extracts the potential feature code X related to them P′ and X Q′ ;

[0097] S53, coarse-scale superpoint sets P' and Q' and latent feature encoding X P′ and X Q′ The global context information aggregation module with geometry perception and position perception is input, the global information between the same frames is learned in a rotation-invariant way through the geometry-aware self-attention module, and the cross-frame global interaction is realized through the position-aware cross-attention module, and the deep conditional feature F of the super point is output. P′ and F Q′ , overlap fraction o P′ and Q′ and the cross-overlap score and

[0098] More specifically, in step S53, the specific process includes the following steps:

[0099] S531, a self-attention module with geometry perception, wherein the self-attention can be abstracted as a fully connected graph, formed by connecting each super-point in the point cloud with other super-points, and the module includes a geometry branch and a context branch;

[0100] The geometry branch embeds G' from the geometric structure information P Mining geometric clues in the process can be expressed as:

[0101] G′ P =CAT[ρ i,j ,η i,j ,α i,j ];

[0102] Among them, CAT represents the splicing operation. For the super point p i and p j , described as the distance ρ between two points in Euclidean space i,j =||p i -p j ||2, then perform sinusoidal encoding and use a hyperparameter to adjust the sensitivity of its distance change; for p i The two nearest neighbors and p j The two nearest neighbors Described as the local triangle perimeter difference In the formula is the Euclidean distance between two points, and then sinusoidal encoding and adjustment are performed in the same way; for p j and p i The three nearest neighbors and Calculate the vector p j -p i and and Three points form the normal vector of the plane and the local plane normal related angle Calculate the angle and then encode it using a sine function, with hyperparameters controlling the sensitivity of the angle changes;

[0103] In the geometry branch, we first have two learnable matrices W' G and W' E From G' P Linear projection obtains the geometric context G'=W' G G' P and its position code E'=W' E G' P ;

[0104] In the context branch, the latent feature encoding X of the input P' And the corresponding learnable matrix W' Q , W' K and W' V Linear mapping gives Q'=W' Q X P' , K'=W' K XP' , V'=W' V X P' , mixed attention weight parameter In order to prevent the inner product from being too large, the final self-attention output super-point conditional feature F P′ =Feed Forward(A'G')+Feed Forward(A′V′) is composed of the sum of two branches, softmax is a normalized exponential function, and Feed Forward is a feedforward neural network;

[0105] S532, cross attention with position awareness. Taking the information flow direction of P'→Q' as an example, the operation in the opposite direction Q'→P' is similar; according to the input super-point conditional feature F P′ and F Q′ , first according to the learnable matrix and Project to and Finally output the updated features

[0106] S533, after three times of self-attention and cross-attention are completed alternately, the obtained features are linearly projected into the overlap score o P′ and Q′ , and get the cross-overlap score. For the coarse-scale superpoint set P', the cross-overlap score <·,·> represents the inner product operation, t represents the temperature hyperparameter, The calculation is similar.

[0107] S54, the decoder decodes the two sets of super-point information into a detailed point-by-point description. Taking the source point cloud P as an example, first convert F P′ With o P′ , The concatenation is then performed into a decoder structure that combines upsampling and linear layers and includes cross-layer connections from the encoder of the first module. The upsampling is done by querying the relevant features of the nearest point from the previous layer. Finally, the decoder outputs the refined point-by-point features F P and F Q , overlap fraction o P and Q And the matching score m P and m Q ;

[0108] S55, Probabilistic sampling RANSAC solves rigid transformation. Different from the commonly used random sampling, this algorithm uses the mechanism of probabilistic sampling. Specifically, the product of the overlap score o and the matching score m is used to bias the sampling of interest points, and then the correspondence is estimated based on the feature matching from the two point cloud sampling points. Finally, the singular value decomposition is used to solve the translation matrix and rotation matrix between the source point cloud P and the target point cloud Q.

[0109] S56, "Predator" involves three losses and ground-truth correspondences to supervise the training process, including three loss functions: 1) Circle loss, for the source point cloud P, first extract the point p that has at least one corresponding relationship in Q i , corresponding point positive example set ε p (p i ) is defined as i Nearby radius r p Points inside, negative example set ε n (p i ) is defined as the s Point sets other than circle loss Where Δ n ,Δ p Indicates the positive and negative margins of hyperparameters, n p represents the number of sampling points, represents the distance in the feature space, and is the weight obtained for each positive and negative example, and the loss of point cloud Q in the opposite direction The calculation is similar; 2) Overlap loss, estimating the overlap score can be transformed into a two-classification problem. For the point cloud P, the overlap loss is used Supervision, the ground-truth label in Eq. Defined as Where r o represents the overlap threshold, represents the basis rigid transformation, The calculation method is similar; 3) Matching loss, this loss is used to supervise the matching score, which is also converted into a binary classification problem. For the point cloud P, the loss function is The base tag for Where r m represents the matching threshold, NN F represents the nearest neighbor search in the feature space, The calculation is similar; the total round loss is based on The total overlap loss is obtained according to The total matching loss is obtained according to We get the total loss by adding the three together;

[0110] S57. After the proposed "predator" obtains the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree, any point cloud is selected as a reference from all local three-dimensional point cloud data, and the remaining local three-dimensional point cloud data is converted to the reference system of the reference three-dimensional point cloud data according to the rotation and translation relationship contained in the edge established in the spanning tree, thereby completing the precise alignment of the three-dimensional point cloud data of the entire machine.

[0111] In this embodiment, a deep learning algorithm "Predator" is proposed for precise registration. The global information aggregation module in the "Predator" aggregates global information in a rotationally invariant manner, and has the characteristics of same-frame and cross-frame spatial perception, and pays great attention to overlapping areas. The strategy of using "self-attention-cross-attention" three times can improve the significance of features. The geometric structure information embedding proposed in the self-attention module enables it to have geometric information perception ability, which ensures the accuracy of precise registration. The overlap is combined during registration. This method gives priority to point clouds with large overlap, which greatly reduces the generation of errors. The better initial posture obtained in this link provides a guarantee for the registration effect of ICP in the refined registration.

[0112] S6. Calculate the overlap between any two point clouds again, construct a maximum spanning tree based on the overlap, and use the traditional ICP algorithm for refined registration.

[0113] As a preferred implementation of step S6, in step S6, the specific process includes the following steps:

[0114] S61, repeat steps S42 and S43;

[0115] S62. For any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P, and the other is used as the target point cloud Q;

[0116] S63. Obtain point q from target point cloud Q i ∈Q;

[0117] S64. Search for point p in the source point cloud P i , so that the distance ||p i -q i ||min;

[0118] S65, by calculating the translation matrix T and the rotation matrix R, the error function Reach the minimum value, where n is the number of corresponding point pairs involved in the calculation;

[0119] S66: According to the translation matrix T and rotation matrix R obtained in step S64, the point q in the target point cloud is i Transform and update the corresponding superpoint set Q'{qi '=Rq i +T,q i ∈Q};

[0120] S67, calculate the midpoint q of the updated superpoint set Q' i 'with p i The average distance

[0121] S68, if d is less than a preset threshold or the number of iterations reaches an upper limit, stop the loop iteration, otherwise return to step S64;

[0122] S69. After obtaining the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree by the ICP algorithm, arbitrarily select a point cloud as a reference, and according to the rotation and translation relationship contained in the edges established in the spanning tree, convert the remaining local three-dimensional point cloud data into the reference system where the reference three-dimensional point cloud data is located, so as to complete the refined alignment of the whole machine three-dimensional point cloud data.

[0123] In this embodiment, the overlap is calculated again and the maximum spanning tree is extracted based on the high-precision results obtained by the "predator", and the point cloud data is refined and aligned in combination with the overlap, which further reduces the error. ICP-based refined alignment is performed. If the initial position is unreasonable, the ICP algorithm will fall into the local optimum and affect the measurement quality. However, the "predator" provides a better initial pose for ICP, which ensures the accuracy of ICP refinement and alignment, and finally each local point cloud is rotated and translated to the coordinate system of the reference point cloud.

[0124] The method of sampling from coarse to fine and then refining in the present invention uses the preceding step as the basis for the succeeding step, fully utilizes the hardware attributes, and combines deep learning with traditional algorithms; the present invention realizes flexible and intelligent measurement of the whole aircraft data, efficient and accurate measurement of the whole aircraft shape of large aircraft, meets the precision requirements of the whole aircraft shape measurement of large aircraft, thereby ensuring its operating performance and safety performance, and assists in improving the whole aircraft detection efficiency of large aircraft, thereby ensuring the operating performance and safety performance of large aircraft.

[0125] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0127] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A flexible combined three-dimensional measurement method for the overall appearance of a large aircraft, characterized in that: The following steps are involved: S1. Establish a base station to track and locate the 3D laser scanner in the measurement space in real time and obtain the spatial position of the 3D laser scanner; S2, using a three-dimensional measuring device to enable a three-dimensional laser scanner to scan and collect data of the large aircraft from multiple angles, and obtain local three-dimensional point cloud data of the upper and lower surfaces of the large aircraft; S3, using a coarse registration method based on base station positioning to coarsely register the collected local three-dimensional point cloud data; S4, performing voxel-based sampling on each local three-dimensional point cloud data after rough registration, calculating the overlap between any two point clouds, and extracting the maximum spanning tree according to the overlap; S5. A deep learning algorithm "Predator" is proposed to precisely align the obtained maximum spanning tree; S51. For any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P, and the other is used as the target point cloud Q; S52, the source point cloud P and the target point cloud Q are respectively input into the encoder, which contains multiple residual blocks and cross-layer convolution blocks. The encoder downsamples the two point clouds into coarse-scale super point sets P' and Q', and extracts the potential feature code X related thereto. P′ and X Q′ ; S53, coarse-scale superpoint sets P' and Q' and latent feature encoding X P′ and X Q′ The global context information aggregation module with geometry perception and position perception is input, the global information between the same frames is learned in a rotation-invariant way through the geometry-aware self-attention module, and the cross-frame global interaction is realized through the position-aware cross-attention module, and the deep conditional feature F of the super point is output. P′ and F Q′ , overlap fraction o P′ and Q′ and the cross-overlap score and S54, the decoder decodes the two sets of super-point information into a detailed point-by-point description. For the source point cloud P, first convert F P′ With o P′ , The concatenation is then performed into a decoder structure that combines upsampling and linear layers and includes cross-layer connections from the encoder of the first module. The upsampling is done by querying the relevant features of the nearest point from the previous layer. Finally, the decoder outputs the refined point-by-point features F P and F Q , overlap fraction o P and Q And the matching score m P and m Q ; S55, probabilistic sampling RANSAC solves the rigid transformation, uses the product of the overlap score o and the matching score m to bias the sampling of the interest points, then estimates the correspondence based on the feature matching from the two point cloud sampling points, and finally uses singular value decomposition to solve the translation matrix and rotation matrix between the source point cloud P and the target point cloud Q; S56, "Predator" involves three losses and ground-truth correspondences to supervise the training process, including three loss functions: 1) Circle loss, for the source point cloud P, first extract the point p that has at least one correspondence in Q i , corresponding point positive example set ε p (p i ) is defined as i Nearby radius r p Points inside, negative example set ε n (p i ) is defined as the s Point sets other than circle loss Where Δ n ,Δ p Indicates the positive and negative margins of hyperparameters, n p represents the number of sampling points, represents the distance in the feature space, and is the weight obtained for each positive and negative example, and the loss of point cloud Q in the opposite direction The calculation is similar; 2) Overlap loss: For point cloud P, overlap loss is used Supervision, the ground-truth label in Eq. Defined as Where r o represents the overlap threshold, represents the basis rigid transformation, The calculation method is similar; 3) Matching loss, for point cloud P, the loss function is The base tag for Where r m represents the matching threshold, NN F Represented in feature space Nearest neighbor search in The calculations are similar; Total round loss based on The total overlap loss is obtained according to The total matching loss is obtained according to We get the total loss by adding the three together; S57, after obtaining the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree by the proposed "predator", select one point cloud as the reference among all the local 3D point cloud data, and transform the remaining local 3D point cloud data into the reference system of the reference 3D point cloud data according to the rotation and translation relationship contained in the edge established in the spanning tree, so as to complete the precise registration of the 3D point cloud data of the whole machine; S6. Calculate the overlap between any two point clouds again, construct a maximum spanning tree based on the overlap, and use the traditional ICP algorithm for refined registration.

2. The flexible combined three-dimensional measurement method for the entire appearance of a large aircraft according to claim 1 is characterized by: In step S1, a base station is established, specifically including: multiple binocular cameras are arranged in the entire measurement space to form a base station, covering the envelope space of the large aircraft to be measured.

3. The flexible combined three-dimensional measurement method for the overall appearance of a large aircraft according to claim 1 is characterized by: The S3 specifically includes: knowing the internal and external parameters of the binocular camera deployed at the base station, roughly unifying the local three-dimensional point cloud data obtained by the three-dimensional laser scanner into the global coordinate system according to the camera parameters and the spatial posture of the three-dimensional laser scanner, and completing the rough alignment.

4. The flexible combined three-dimensional measurement method for the entire aircraft shape according to claim 1 is characterized by: In step S4, the specific process includes the following steps: S41, in the process of voxel-based sampling, traverse the entire 3D point cloud data set, for each point that has not been marked as deleted, first convert the 3D point cloud data into a 3D voxel space, and select the voxel center in each voxel as a sampling point; S42, in the process of calculating the overlap between point clouds, for any two point clouds, find the nearest neighbor point pair between the two, calculate the distance between the matching point pairs, set a distance threshold, and count the number of point pairs between the matching point pairs whose distance is less than the threshold. The overlap is defined as the ratio of the number of point pairs whose distance is less than the threshold to the total number of points in the source point cloud; S43. A weighted fully connected graph is constructed according to the overlap between each pair of point clouds. Each point cloud constitutes a node of the graph. The overlap between two point clouds represents the weight of the edge between the nodes. Then, a maximum spanning tree is constructed according to the weighted fully connected graph.

5. The flexible combined three-dimensional measurement method for the entire appearance of a large aircraft according to claim 1 is characterized by: In step S53, the specific process includes the following steps: S531, a self-attention module with geometry perception, the module comprising a geometry branch and a context branch; The geometry branch embeds G' from the geometric structure information P Mining geometric clues in the , which can be expressed as: G′ P =CAT[ρ i,j ,or i,j ,a i,j ]; Among them, CAT represents the splicing operation. For the super point p i and p j , described as the distance ρ between two points in Euclidean space i,j =||p i -p j ||2; for p i The two nearest neighbors and p j The two nearest neighbors Described as the local triangle perimeter difference In the formula is the Euclidean distance between two points; for p j and p i The three nearest neighbors and Calculate the vector p j -p i and and Three points form the normal vector of the plane and the local plane normal related angle Calculate angles; In the geometry branch, we first have two learnable matrices W' G and W' E From G' P Linear projection obtains the geometric context G'=W' G G' P and its position code E'=W' E G' P ; In the context branch, the latent feature encoding X of the input P' And the corresponding learnable matrix W' Q , W' K and W' V Linear mapping gives Q'=W' Q X P' , K'=W' K X P' , V'=W' V X P' , mixed attention weight parameter In order to prevent the inner product from being too large, the final self-attention output super-point conditional feature F P′ =Feed Forward(A'G')+Feed Forward(A'V') is composed of the sum of two branches, softmax is a normalized exponential function, and FeedForward is a feedforward neural network; S532, according to the input super point condition feature F P′ and F Q′ , first according to the learnable matrix and Project to and Finally output the updated features S533, after three times of self-attention and cross-attention are completed alternately, the obtained features are linearly projected into the overlap score o P′ and Q′ , and get the cross-overlap score. For the coarse-scale superpoint set P', the cross-overlap score <·,·> represents the inner product operation, t represents the temperature hyperparameter, The calculation is similar.

6. The flexible combined three-dimensional measurement method for the entire appearance of a large aircraft according to claim 4 is characterized by: In step S6, the specific process includes the following steps: S61, repeat steps S42 and S43; S62. For any two nodes with edges in the maximum spanning tree, one of them is used as the source point cloud P, and the other is used as the target point cloud Q; S63. Obtain point q from target point cloud Q i ∈Q; S64. Search for point p in the source point cloud P i , so that the distance ||p i -q i ||min; S65, by calculating the translation matrix T and the rotation matrix R, the error function Reach the minimum value, where n is the number of corresponding point pairs involved in the calculation; S66: transform the point qi in the target point cloud according to the translation matrix T and rotation matrix R obtained in step S64, and update the corresponding superpoint set S67, calculate the midpoint q of the updated superpoint set Q' i 'with p i The average distance S68, if d is less than a preset threshold or the number of iterations reaches an upper limit, stop the loop iteration, otherwise return to step S64; S69. After obtaining the translation matrix and rotation matrix between any two point clouds with edges in the maximum spanning tree by the ICP algorithm, arbitrarily select a point cloud as a reference, and according to the rotation and translation relationship contained in the edges established in the spanning tree, convert the remaining local three-dimensional point cloud data into the reference system where the reference three-dimensional point cloud data is located, so as to complete the refined alignment of the whole machine three-dimensional point cloud data.

7. A device for a flexible combined three-dimensional measurement method of a large aircraft overall appearance according to any one of claims 1 to 6, characterized in that: In step S2, the three-dimensional measuring device specifically includes: Binocular cameras (1), used for real-time positioning of the three-dimensional laser scanner, are distributed throughout the measurement space to form a base station; An industrial robot arm (2) is used to carry a three-dimensional laser scanner (3) for collecting point cloud data of a large aircraft using a short-range terminal measurement device; A lifting mechanism (4) is used for collecting data on the upper surface of a large aircraft. An industrial mechanical arm (2) is installed at the lower end of the lifting mechanism (4). The height of the industrial mechanical arm (2) is adjusted by adjusting the telescopic degree. The AGV trolley (5) is used for collecting data from the lower surface of a large aircraft. In this case, the industrial robot arm (2) is carried by the AGV trolley 5 and drives the three-dimensional laser scanner (3) to move on the ground; The intelligent terminal platform is used to send automation equipment control instructions and control the base station and the three-dimensional laser scanner (3) to collect data.

8. The device for the flexible combined three-dimensional measurement method of the entire large aircraft shape according to claim 7 is characterized in that: The lifting mechanism (4) specifically comprises: a lifting mechanism active motor (41), a lifting mechanism frame body (42), a lifting mechanism line frame (43) for ensuring the stability and anti-winding of the line, and a crossbeam assembly (44); wherein: The crossbeam assembly (44) comprises a line (441), a rotating shaft (442), a line frame (443), a line frame driven wheel (444), a line frame limit key (445), a lifting mechanism trolley (446), a lifting mechanism trolley driving wheel (447), a crossbeam (448), a driving trolley (449) and a turntable (450). One end of the crossbeam (448) is connected to the rotating shaft (442) of the gantry (6), and the other end is connected to the driving trolley (449). The driving trolley (449) is installed on the turntable (450). The driving trolley (449) carries the crossbeam (448) and rotates around the rotating shaft (442). The upper end of the lifting mechanism (4) is connected to a lifting mechanism trolley (446) of the cross beam (448), and can move along the inner rail of the cross beam (448) when carried by the lifting mechanism trolley (446), so that scanning is more comprehensive and flexible; The line (441) is suspended on a plurality of line racks (443), and the lifting mechanism trolley (446) moves under the drive of the lifting mechanism trolley driving wheel (447) while stretching the line rack (443) to make parallel movement.

Images