An Automated 3D Inspection Method Using Surface Structured Light for Complex Aerospace Parts

Abstract

This invention discloses an automated 3D inspection method using structured light for complex aerospace parts. First, the theoretical model of the aerospace part is analyzed and processed to generate shooting viewpoints, and the optimal viewpoint set is selected. Then, the optimal path is calculated using parameters such as the motion cost of the robotic arm. The scanner captures images along the viewpoint path, employing state-of-the-art binocular structured light multi-frequency heterodyne method and an improved phase stereo matching algorithm to acquire point clouds. Coarse registration of the point clouds is performed using an improved marker pasting and combination stitching method; fine registration is achieved using optimized ICP (Integrated Circuit Propagation) to fuse the multi-view point clouds. High-quality point clouds are generated and the 3D model is reconstructed through algorithms such as point cloud simplification, point cloud denoising and smoothing, point cloud misalignment adjustment, and hole filling. Finally, the acquired model is compared with the theoretical model to obtain error analysis. Experimental verification of this invention effectively improves the accuracy and automation of digital 3D measurement of complex aerospace parts.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology and 3D accuracy inspection based on surface structured light in reverse engineering, specifically to an automated 3D inspection method for aerospace parts using surface structured light. Background Technology

[0002] In the aircraft manufacturing industry, critical aerospace components have complex shapes and their forming precision is difficult to control. The accuracy and efficiency of their production are crucial to the quality of aircraft manufacturing. Currently, quality inspection of aerospace components largely relies on manual inspection, which suffers from low efficiency and unstable accuracy. With the development of digital measurement technology, structured light scanners are widely used in the precision manufacturing and assembly of aircraft products due to their high precision and efficiency. Traditional scanners require manual experience to determine the shooting position, resulting in low efficiency. Furthermore, the reconstruction of irrelevant backgrounds during the scanning and reconstruction process affects the efficiency and quality of the reconstruction. Therefore, these technical problems need to be solved in the automated 3D inspection of complex aerospace components using structured light. Summary of the Invention

[0003] Purpose of the Invention: The purpose of this invention is to address the shortcomings and defects of existing technologies by proposing an automated 3D inspection method using structured light for complex aerospace parts. This aims to solve the technical problems of low efficiency in determining shooting positions based on manual experience and poor quality of models reconstructed by scanners.

[0004] Technical solution:

[0005] To solve the above problems, the present invention can adopt the following technical solution.

[0006] An automated 3D inspection method using structured light for complex aerospace parts includes the following steps:

[0007] (1) Obtain the theoretical data model of complex aerospace parts;

[0008] (2) The theoretical data model of the part is partitioned with the measurement space as a constraint;

[0009] (3) Generate initial viewpoints for the theoretical data models of the parts completed in each partition. Establish a spherical coordinate system with the center of each partition as the center and the shooting distance as the radius. Generate several viewpoints covering the spherical surface by deviating the radial and lateral vector angles from the center of the sphere by one angle.

[0010] (4) Evaluate each viewpoint and select the best viewpoint for each surface based on the number of features captured and the occlusion rate under the shooting conditions of that viewpoint, so as to obtain the top few best viewpoints for each partition.

[0011] (5) After the screening is completed, all the best viewpoints are counted, the optimal path is planned for the robotic arm, and the optimal path passing through each optimal viewpoint is calculated. The optimal robotic arm path point order is calculated by ant colony algorithm for all the generated optimal viewpoints. The preset start point and end point of the robotic arm are added to form the optimal path for this part scanning.

[0012] (6) The binocular camera is calibrated. The intrinsic and extrinsic parameters of the binocular camera are obtained by using the Zhang's checkerboard calibration method. After the image is corrected by combining the above intrinsic and extrinsic parameters, it is directly used to calculate the reprojection matrix Q of the image pixel coordinates to world coordinates.

[0013] (7) Provide real complex aerospace parts and tooling for fixing the complex aerospace parts, and set non-coded markers on the tooling for binocular cameras to identify the type of parts;

[0014] (8) The robotic arm starts up and traverses each viewpoint according to the obtained path sequence. It projects 24 generated three-frequency eight-step phase-shift patterns onto the surface of the complex aerospace parts through an industrial projector and then uses an industrial camera to capture the data and transmits it to the computer.

[0015] (9) Calculate three-dimensional point cloud: Use 24 three-frequency eight-step phase shift patterns captured to perform multi-frequency heterodyne phase deconvolution algorithm; calculate two continuous phase images of the left and right sides of the continuous modulated aerospace complex parts, and then perform automatic threshold segmentation on the phase images to distinguish the aerospace complex parts and the background in the phase images.

[0016] (9) Match the pixel information of the parts in the left and right camera images. After the matching is completed, subtract the corresponding points of the image coordinates to obtain the disparity map matrix D of the left and right cameras. Calculate the three-dimensional point cloud of the parts in the world coordinate system captured by the current viewpoint using the reprojection matrix Q and the disparity map matrix D obtained in the above steps, and output the point cloud data of the current viewpoint.

[0017] (10) Calculate the transformation matrix required for multi-view point cloud stitching by identifying the pasted marker points; apply the obtained transformation matrix to the point clouds obtained from the previous and next views, that is, unify the two point clouds under the same coordinate system and complete the stitching of the common area. Use the matrix obtained above to complete the preliminary coarse stitching of the point cloud.

[0018] (11) Perform fine registration of point cloud using ICP registration algorithm, traverse all calculated viewpoints, calculate point cloud and complete automatic point cloud stitching to obtain preliminary complete point cloud data;

[0019] (12) The coarse point cloud obtained is optimized by point cloud processing algorithm to obtain accurate point cloud data;

[0020] (13) Triangulate and reconstruct the obtained precise point cloud data to obtain the STL triangular mesh model and convert it into the STEP model;

[0021] (14) The obtained step model is compared with the theoretical data model of the part to obtain the error data of the part manufacturing.

[0022] Furthermore, in step (5), after the filtering is completed, all optimal viewpoint sets {p1, p2...p} are statistically analyzed. i For a robotic arm carrying a scanner, optimal path planning is performed to calculate the most efficient and interference-free path that passes through each optimal viewpoint. The path weight is the motion cost of the robotic arm. Calculating the optimal path for traversing a set of points simplifies to a Traveling Salesman Problem (TSP) with a weighted directed graph, and the path weight is set as the motion cost of the robotic arm. In Cartesian space, the motion cost is defined as the angles of each axis that need to change from the robotic arm's pose at the current viewpoint to the pose at the next viewpoint. The following formula is the parameter for calculating the motion cost:

[0023]

[0024] Where η is the cost of the robotic arm during the two viewpoint changes, α is the cost coefficient, and θ′ i With θ” i The angles of each axis are shown from the front and rear views of axes 1 to 6.

[0025] All generated optimal viewpoint sets {p1, p2... p i The optimal sequence of robotic arm path points {p} is calculated using the algorithm described above. m ,p m+1 ...p m+i}, where m~m+i are the generated optimal viewpoint set, the order of the new traversal points obtained by the path planning algorithm, plus the robot arm's preset start and end points, which constitute the optimal path {p} for this part scanning. start ,p m ,p m+1 …p m+i ,p end}, p start p is the starting point of the optimal path. end The optimal path endpoint serves as the data foundation for subsequent 3D reconstruction.

[0026] Furthermore, the scanner includes two industrial cameras and an industrial projector. The two industrial cameras are placed on a horizontal fixed frame with their optical centers on the same straight line and their shooting directions intersecting at approximately 30 degrees. The industrial projector is placed between the two cameras. The positions of the cameras and the projector satisfy a certain relationship: the rectangular area projected by the projector is completely contained within the common field of view of the images captured by the two cameras.

[0027] Furthermore, in step (6), the intrinsic parameters include the coefficients dx,dy for transforming the left and right camera image coordinate systems to the world coordinate system with the optical center of the left camera as the origin, the radial distortion coefficients k1,k2,k3 for the left and right cameras, and the tangential distortion coefficients p1,p2. The extrinsic parameters include the transformation matrix R,T for transforming the two camera image planes to the binocular parallel model.

[0028] Furthermore, in step (9), the continuous phase diagrams Φ(x,y) of the left and right sides of the modulated aerospace complex part are calculated. left ,Φ(x,y) right Subsequently, due to issues such as repetitive phase calculations and mismatches in invalid measurement areas (background and shadow areas) during 3D reconstruction, the segmentation of the effective measurement area was studied based on the fringe modulation. Automated threshold segmentation of the phase map was performed to distinguish the part from the background. The algorithm steps are as follows: The Otsu algorithm is introduced into the modulation threshold calculation. Under different lighting and environmental backgrounds, the optimal threshold is determined using the Otsu algorithm, removing discrete noise while automatically segmenting the area of ​​the part to be measured from the irrelevant background area. The modulation function is as follows:

[0029]

[0030] In the formula, i,j are image coordinates, B(i,j) is the modulation value of the pixel to be calculated, N is the number of phase shift steps used, and 8 steps are used in this invention. n (i,j) represents the pixel value at the corresponding coordinates in the 24 images collected; after calculating one modulation image from these 24 images, a threshold is set and image processing algorithms are used to segment the parts from the irrelevant background.

[0031] The Otsu algorithm is introduced to calculate the modulation threshold. This threshold, along with the modulation map B(i,j), is used to segment the image region containing the part from the background-independent region, resulting in the continuous phase matrix map Φ(x,y) of the part after removing the irrelevant regions. left ′,Φ(x,y) right Solve for the continuous phase Φ(x,y) used for stereo matching. left ′,Φ(x,y) right After that, the phase value is optimized using the step correction algorithm to obtain the optimized phase matrix Φ(x,y). left ,Φ(x,y) right ".

[0032] Further, in step (9), the phase map is distorted by using the radial distortion coefficients k1, k2, k3 and the tangential distortion coefficients p1, p2 of the left and right cameras; then stereo correction is performed by using the left and right camera transformation matrix R, T. In the corrected left and right images, the corresponding pixels are on the same horizontal line, that is, the vertical coordinates are equal. The phase values ​​are searched by row, and those that are equal are considered to be a pair of corresponding points. All corresponding points are set to complete the stereo matching of the binoculars. Then the horizontal coordinates of the corresponding point images are subtracted to obtain the disparity map matrix D. The reprojection matrix Q and the above disparity map matrix D are input to calculate the three-dimensional point cloud of the part in the world coordinate system with the optical center of the left camera as the origin. The reprojectImageTo3D() function of the OpenCV library is called to output the point cloud data of the current view. The principle formula is as follows:

[0033]

[0034] Where Xw / W, Yw / W, and Zw / W are world coordinates, u and v are image coordinates, d is disparity, and Q is the reprojection matrix.

[0035] Furthermore, in step (10), the transformation matrix calculation process is as follows: first, the threshold parameter is manually changed using the modulation function to restore the 24 patterns to grayscale images without projected stripes; the area where the attached circular black and white non-coded markers are located is separated by image algorithms; image dilation and erosion are implemented using the morphologyEx function based on the OpenCV library; and the adaptiveThreshold function can be used to adaptively perform binarization segmentation on the grayscale image.

[0036] After preprocessing the grayscale image using the GaussianBlur function in the OpenCV library, the Canny operator function in the OpenCV library is used to extract the boundary point information of the image, and the Hough transform is used to detect the elliptical contour of the marker point. The marker point is then selected based on the roundness of the ellipse, the length of the ellipse axis at the preset shooting distance, and the grayscale information parameters of the sampled circular coded point.

[0037] The center parameter of the fitted ellipse is the center of the circle identified in the image. The three-dimensional coordinates corresponding to the two-dimensional coordinates of the center are mapped from the calculated point cloud and recorded as the current viewpoint marker set A. The robotic arm moves to the next target point and repeats the above steps to obtain the marker set B. The calculated marker set A and marker set B are used to perform marker point correspondence matching and transformation matrix calculation.

[0038] Based on distance invariance, a distance description matrix is ​​established for each point cluster. Let the set of marker points be A{p1~p2}. m} and the set of marker points B{q1~q nThe distance between each point in the set and the rest of the set is given by the following formula:

[0039]

[0040] In the formula p m p m-1 q n q n-1 This represents the 3D distance between two points. Then, a row-by-row matching process is performed on these two distance matrices, setting a threshold and identifying the pair of points p with the highest number of equal distances between two given rows. r With q t , to this pair of points {p r ,q t} is considered the first pair of matching points and taken as the basic origin. Then, distance vectors a and b from the basic origin are constructed:

[0041] a = [p] r p1,p r p2…p r p m ]

[0042] b = [q t q1,q t q2…q t q n ]

[0043] In the formula, p represents p r With q t The vector is constructed by sorting the distances to the remaining points; similarity matching is performed again to find two more pairs of points {p}. r ′,q t ′},{p r ,q t "} and satisfy the constraint condition: {p r ,p r ′,p r} and {q t ,q t ′,q t "} Forming congruent triangles means that corresponding sides are equal. At this point, three pairs of matching points have been found, and the next step is to solve the transformation matrix;

[0044] After obtaining three pairs of points, the transformation matrix required for point cloud registration is obtained using SVD decomposition, as shown in the following formula:

[0045]

[0046] In the formula, J is the objective function, k is the number of matching point pairs (k=3), R,T is the rotation and translation matrix to be solved, and q i ,p i Let J be the three-dimensional coordinates of the corresponding point, so that J is minimized;

[0047] The obtained transformation matrix is ​​applied to the point clouds obtained from the front and rear views, that is, the two point clouds are unified in the same coordinate system and the common area is stitched together. The TransformPointCloud function in the PCL library is called to use the matrix obtained above to complete the initial coarse stitching.

[0048] Furthermore, in step (11), the ICP registration algorithm steps are as follows:

[0049] (11.1) First, estimate the initial transformation matrix T0, and then perform the transformation on each point P in the sampled source point cloud data A. ai Applying the transformation matrix T0, we obtain P. ai ′.

[0050] (11.2) From the set of marker points B, use a KD Tree to quickly search for each distance P. ai The nearest point P bi , with P ai ′ Form corresponding point pairs;

[0051] The optimal transformation matrix ΔT is obtained by acquiring the point pairs:

[0052]

[0053] (11.3) The convergence condition is the iteration error and the number of iterations. If convergence is achieved, calculate T = ΔT * T0 and output the final transformation matrix. Otherwise, calculate T0 = ΔT * T0 and return to step (11.1) for the next iteration.

[0054] Furthermore, in step (12), the overlapping parts of the splicing are preprocessed to remove excess noise and invalid points; the coarse point cloud obtained is optimized by point cloud processing algorithm, using built-in algorithm functions in the PCL open source library, including downsampling filtering, noise removal of outliers, normal adjustment, and hole filling, to obtain accurate point cloud data.

[0055] Furthermore, in step (13), the final accurate point cloud data is triangulated and reconstructed. The reconstruct function in PCL is used to obtain the STL triangular mesh model and convert it into a STEP model using the OCC library. The model is then imported into the Geomagic software for accuracy testing using the numerical model comparison analysis function.

[0056] The beneficial effects of this invention are as follows:

[0057] 1) This invention utilizes a photogrammetric measurement method to rapidly and non-contactly reverse engineer high-precision digital models of parts. Through automatic viewpoint planning and robotic arm-assisted movement of these viewpoints, it is suitable for automated measurement on production lines. Compared to traditional manual visual or instrument inspection, it causes no damage to parts, is non-contact, and offers high speed and accuracy. It fully automates precision inspection and feeds data back to the production line in real time to form a closed-loop control, effectively improving production quality and efficiency.

[0058] 2) This invention processes and divides the imported theoretical model, generating suitable viewpoints. This replaces the previous method of relying on manual experience to determine the shooting angle, which resulted in excessive repetitive areas, tedious manual teaching, and low efficiency. This planning method can be better integrated into flexible production line inspection of products, making inspection more diversified and efficient.

[0059] 3) The point cloud stitching technology of this invention combines coded point numbering with non-coded point matching technology. Marker points are pasted around the tooling. Coded points are used to identify the theoretical model corresponding to the number, while non-coded points are used to calculate the transformation matrix and match the point cloud under multi-view photography. Compared to conventional pasting on the part surface, multiple pastings are unnecessary; the tooling only needs to be pasted once. Simultaneously, it does not damage the part surface and reconstructs a hole-free structure.

[0060] 4) Compared with conventional micrometers and coordinate measuring machines, the model comparison function of this invention is non-contact and uses a completely virtual interface for measurement, making it more automated and digital. Furthermore, the data can be analyzed and fed back to the production line, forming a closed-loop control system. Attached Figure Description

[0061] Figure 1 This is a flowchart of the calculation process for an automated 3D scanning system based on binocular structured light.

[0062] Figure 2 A flowchart illustrating the process of automatically generating the optimal viewpoint.

[0063] Figure 3 This is a diagram showing the labeling of 8-bit CCT encoding markers.

[0064] Figure 4 This is a schematic diagram of the principle of binocular ranging.

[0065] Figure 5 This is a schematic diagram of the point cloud multi-view stitching process based on marker points.

[0066] Figure 6 This is a simplified diagram of the scanning device.

[0067] Figure 7 This is a 3D model of the theoretical model of the part to be tested.

[0068] Figure 8 This is a sine fringe pattern.

[0069] Figure 9 This is the left phase diagram of the segmented part.

[0070] Figure 10 This is a parallax diagram of the part.

[0071] Figure 11 Point cloud diagram of the part.

[0072] Figure 12 Point cloud diagram after precise alignment of parts.

[0073] Figure 13 This is a 3D comparison inspection report image of the part. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the scope of protection of the present invention:

[0075] This invention proposes an automated inspection method based on binocular structured light 3D reconstruction. First, a theoretical model of a complex aerospace part is partitioned and viewpoints are generated. After setting filtering parameters, the optimal viewpoint for each partition is obtained, and path planning is performed. After the binocular structured light scanner captures images at each viewpoint, the continuous phase used for stereo matching is automatically thresholded to segment the background and part, and 3D reconstruction is performed based on the principle of triangulation. For point cloud stitching under multiple views, tooling is used to attach marker points. Finally, a complete, high-quality 3D model is obtained and compared with the theoretical model to obtain the required error analysis report for the inspection area. For the specific process, see [link to detailed process]. Figure 1 .

[0076] The hardware components of this invention are divided into a data acquisition device, a power device, and a computing device. The software components include viewpoint generation and path planning functions, 3D reconstruction functions, and data processing functions.

[0077] Data acquisition device: A 3D scanner is used, consisting of two industrial cameras mounted on a horizontal stand with their optical centers aligned in a straight line and their shooting directions intersecting at approximately 30 degrees. A DLP industrial projector is positioned between the two cameras, ensuring the projected image is not obstructed by the camera lenses, and that the camera and projector positions are related in a specific way: the rectangular area projected by the projector must be completely contained within the common field of view of the images captured by both cameras. The camera and lens selection must ensure that the depth of field and field of view are sufficient for the sizes of commonly used complex aerospace components.

[0078] Furthermore, the scanner also includes a heat dissipation device and a laser rangefinder.

[0079] Power unit: This invention uses a robotic arm carrying a 3D scanner for automated measurement. Compared to rotary table, Cartesian coordinate, or handheld methods, the robotic arm can be widely applied to scanning paths of various complex parts, and it offers good flexibility, high automation, and high reliability. After selecting a robotic arm with a suitable load and installing the data acquisition device, hand-eye calibration is required to obtain the relative position parameters between the scanner and the robotic arm.

[0080] Computing device: namely, industrial control computer. To meet the requirements of model viewpoint generation and large-scale point cloud processing, the industrial control computer's graphics card and processor performance must meet minimum requirements and be adaptable to the complex environment of the production environment, requiring a heat sink and dustproof enclosure. The industrial control computer connects to the scanner and the robotic arm control terminal via a data cable.

[0081] The software is written in C++ within the QT framework and includes configurations for OpenCV, PCL, and OpenCascade libraries.

[0082] This embodiment of automated 3D inspection using structured light for complex aerospace parts includes the following parts:

[0083] Viewpoint generation and path planning:

[0084] S1.1 First, the part data model of the complex aerospace component, such as in STEP or IGES format, is obtained and imported into the software of this invention for display. Features such as corners, edges, planes, and surfaces can be selected using the mouse. The model processing module interface of this software is written using the QT framework and configured with the OpenCascade open-source library (OCC). This library is mainly used to develop two-dimensional or three-dimensional geometric modeling programs, capable of creating basic geometric bodies, performing Boolean operations on the geometric bodies, and calculating geometric spatial relationships. In this software, the STEPControl_Reader class can be used to quickly read the part's digital model, and the AIS_InteractiveContext class can be used to display the model and perform interactive selection operations.

[0085] S1.2 uses manual interaction by clicking on feature patches or default global selection to partition the part data model with the measurement space as a constraint. Different partitions employ viewpoint generation and filtering algorithms. The measurement space refers to the maximum effective measurement area achievable by the 3D scanner after selecting and assembling the camera and projector according to production requirements; it is typically a platform space determined by the field of view and depth of field. Aerospace parts have numerous detailed features of varying sizes. Large integral parts, such as turbine blades, cannot be reconstructed within a single measurement space. Therefore, it is necessary to partition them into cubes and use multiple perspectives and multiple shots to complete the overall part reconstruction. The Bnd_Box class from the opencascade open-source library can be used to create bounding boxes for the model, with the cube's side length being 1.2 times the depth of field.

[0086] After partitioning in S1.3, initial viewpoints are generated for the models completed in each partition. Using the center of each partition as the sphere's center and the shooting distance as the radius, a spherical coordinate system is established using the Geom_SphericalSurface class from the OCC library. A series of viewpoints are generated on this spherical surface, with radial and lateral vector angles deviating by 15 degrees from the sphere's center. Due to the extreme complexity of aerospace parts, the features within the cubic partitions are diverse in type and size. Therefore, to reduce the number of final viewpoints and capture the maximum number of features, feature classification is necessary.

[0087] S1.4 The optimal viewpoint is generated below; see the flowchart. Figure 2 As shown. The cube partition has six faces, so the normal vectors of each small feature face can be used as the classification basis, classifying them into the orientation of the six faces. The normal vectors can be obtained by calling the Normal function of BRepGProp_Face in OCC. A normal vector is considered to belong to a class if the angle between it and the normal vector of a face of the cube is within 45 degrees. Each viewpoint is evaluated by setting evaluation indicators such as occlusion rate, feature count, and verticality. The optimal viewpoint for each face is selected based on the most features captured in a certain class under that viewpoint and the least occlusion rate, resulting in the top six optimal viewpoints for each partition. Occlusion rate refers to whether the ray (established by the gp_Lin class) pointing from the center of the face to the viewpoint intersects with surrounding features, which can be achieved by calling the IntAna_IntConicQuad class. Feature count is the maximum number of unoccluded features that can be obtained under that viewpoint. Verticality is determined by calculating the angle between the ray and the normal; the smaller the angle, the higher the verticality and the better the image quality.

[0088] After S1.5 is completed, all optimal viewpoint sets {p1, p2, ..., p} are calculated. i For a robotic arm carrying a scanner, optimal path planning is performed to calculate the most efficient and interference-free path that passes through each optimal viewpoint. The path weight is the robotic arm's motion cost. Calculating the optimal path through a set of points can be simplified to a Traveling Salesman Problem (TSP) with a weighted directed graph, where the path weight is set as the robotic arm's motion cost. In Cartesian space, the cost of linear motion between two points is minimized, but this is difficult for the robotic arm to control intuitively. Therefore, the motion cost is defined as the angles along each axis that need to change from the current viewpoint's pose to the next viewpoint's pose. The following formula represents the parameters for calculating the motion cost.

[0089]

[0090] Where η is the cost of the robotic arm during the two viewpoint changes, α is the cost coefficient, and θ′ i With θ” i The angles of each axis are shown from the front and rear views of axes 1 to 6.

[0091] S1.6 The ant colony algorithm can then be used to solve the TSP problem. The ant colony algorithm is derived from learning how ants determine their next path based on pheromones when foraging in nature. This algorithm uses publicly available algorithms and can be found in the fourth paragraph of the article "Theory of Artificial Ant Colony Algorithm and Its Implementation in Classical TSP Problems" (author: Li Suoping), published in *Transportation Systems Engineering and Information*, 2002, Issue 1, pages 54-57, which explains the specific application of the ant colony algorithm in solving the TSP problem. In this invention, all generated optimal viewpoint sets {p1, p2…p i The optimal sequence of robotic arm path points {p} is calculated using the algorithm described above. m ,p m+1 …p m+i}, plus the preset start and end points of the robotic arm, constitute the optimal path {p} for this part scanning. start ,p m ,p m+1 …p m+i ,p end This provides a data foundation for subsequent 3D reconstruction.

[0092] 3D Reconstruction:

[0093] S2.1, based on the OpenCV library, first calibrates the stereo camera using the Zhang checkerboard calibration method to obtain its intrinsic and extrinsic parameters. The intrinsic parameters include the transformation coefficients dx and dy from the left and right camera image coordinate systems to the world coordinate system (with the left camera's optical center as the origin), the radial distortion coefficients k1, k2, and k3 from the left and right cameras, and the tangential distortion coefficients p1 and p2. The extrinsic parameters include the transformation matrices R and T from the two camera image planes to the stereo parallel model. These matrices can be used for stereo correction (the purpose of stereo correction is to convert the converged stereo model into a parallel model to facilitate the calculation of depth information using triangulation principles), and the reprojection matrix Q, which, after combining the above parameters and correcting the image (including distortion correction and stereo correction), can be directly used to calculate the transformation of image pixel coordinates to world coordinates. The calibration algorithm used here is a publicly available algorithm in the prior art. For a detailed explanation of the Zhang's checkerboard calibration algorithm, please refer to the article "A flexible new technique for camera calibration" (author: Zhang Dingyou) published in the December 2000 issue, Vol. 22, No. 11, pp. 1330-1334. For hand-eye calibration of the robotic arm and camera, the position transformation matrix E between the left optical center of the binocular camera and the tool coordinate system of the robotic arm (set at the 6-axis end flange) can be obtained. The algorithm used here is the Tsai-Lenz two-step calibration algorithm, which is also a publicly available algorithm in the prior art. For a detailed explanation of the hand-eye calibration algorithm, please refer to the article "A new technique for fully autonomous and efficient 3D robotics hand / eye calibration" (author: Tsai-Lenz) published in the June 1989 issue, Vol. 5, No. 3, pp. 345-358.

[0094] S2.2 Next, prepare for point cloud data stitching: attach several non-coded marker points around the part mounting fixture. In the above operation, the pasting rule is: ensure that three or more marker points are pasted in each face of each partition, and ensure that the binocular camera can capture at least three unobstructed, non-collinear circular non-coded marker points on the fixture from the current viewpoint. This is the basis for accurately calculating the multi-view point cloud stitching transformation matrix. The pasted coded points are CCT ring codes, and the pattern can be seen in the attached figure (3). The coding principle is: the inner circle is the positioning circle, the outer circle is for anti-interference, and the middle circle is the coding information. Set N-bit binary code, and then divide the middle circle into 360 / N fan-shaped areas, with black being 0 and white being 1. The binary code is checked sequentially using a cyclic method. To prevent the decimal value from being uniquely determined when the image is captured and decoded from any angle, the generated N-bit binary code is continuously shifted. After one cycle, the smallest corresponding decimal number is set as the code value represented by that binary code. If a previous code value already exists, it is skipped. This generates a series of unique code points. Its specific function is to identify part types, facilitating automated inspection by identifying part types and their correspondence with theoretical models.

[0095] S2.3 The optimal path {p} generated above start ,p m ,p m+1 …p m+i ,p end The coordinates of} are calculated by multiplying the matrix E, which is the result of hand-eye calibration in S2.1, by the matrix that transforms the coordinates from the user coordinate system (the coordinate system set by the tooling fixture where the part is located) to the robotic arm tool coordinate system. This transformation converts the coordinates of the points to the actual motion trajectory of the robotic arm, P. start ,P m ,...,P m+i ,P end The data is sent to the robotic arm control center via the communication interface. The robotic arm starts up and traverses each viewpoint in the obtained path sequence, pausing for 2.00 seconds each time. During this time, 24 generated three-frequency eight-step phase-shift patterns are projected onto the surface of the part through a DLP projector, and the data is automatically transmitted to the computer for calculation after being captured by a camera.

[0096] S2.4 Method for Calculating 3D Point Clouds: A multi-frequency heterodyne phase resolution algorithm is used based on 24 captured three-frequency, eight-step phase-shift patterns. This algorithm employs a publicly available method, which can be found in the second paragraph of the doctoral dissertation "Research on Structured Light 3D Measurement Technology and System Based on Digital Grating Projection" (author: Li Zhongwei), published in 2009 at Huazhong University of Science and Technology, which explains the specific application of the multi-frequency heterodyne phase resolution algorithm. This method calculates two consecutive phase images Φ(x,y) of the modulated aerospace complex part. left ,Φ(x,y) rightSubsequently, due to issues such as repetitive phase calculations and mismatches in invalid measurement areas (background and shadow areas) during 3D reconstruction, the segmentation of the effective measurement area was studied based on the fringe modulation degree. Automated threshold segmentation of the phase map was performed to distinguish the part from the background. The algorithm steps are as follows: The Otsu algorithm is introduced into the modulation degree threshold calculation. Under different lighting and environmental backgrounds, the optimal threshold is determined using the Otsu algorithm, removing discrete noise while automatically segmenting the area of ​​the part to be measured from the irrelevant background area. The modulation degree function is as follows:

[0097]

[0098] In the formula, i,j are image coordinates, B(i,j) is the modulation value of the pixel to be calculated, N is the number of phase shift steps used, and 8 steps are used in this invention. n (i,j) represents the pixel value at the corresponding coordinates in the 24 acquired images. After calculating the modulation index of one of these 24 images, the general method is to set a threshold based on environmental experiments and use an image processing algorithm (binarization) to segment the part from the irrelevant background. However, this invention introduces the Otsu algorithm to calculate the modulation index segmentation threshold. Using this threshold and the modulation index image B(i,j), the image region containing the part can be segmented from the background-irrelevant region, resulting in the continuous phase matrix image Φ(x,y) of the part after processing away the irrelevant region. left ′,Φ(x,y) right The Otsu algorithm uses a publicly available algorithm in the prior art. For a detailed explanation of the application of the Otsu automatic thresholding algorithm, please refer to the article "Two-dimensional Otsu automatic thresholding method for grayscale images" (author: Liu Jianzhuang) published in the Journal of Automation, Vol. 19, No. 19, 1993, pp. 101-105.

[0099] S2.5 Solve the continuous phase Φ(x,y) for stereo matching using the above steps. left ′,Φ(x,y) right After the initial phase step error, a phase step error exists, which requires the use of a step correction algorithm to optimize the phase value and obtain an optimized phase matrix Φ(x,y). left ,Φ(x,y) right The algorithm adopts a publicly available algorithm in the existing technology. For details, please refer to the third paragraph of the article "A Phase Noise Correction Method for Shadow Regions Based on the Improved Monotonic Method" (author: Tao Sijie), published in the journal "Progress in Lasers and Optoelectronics" in 2020, Vol. 2, No. 2, pp. 256-263, which explains the specific application of the phase ladder error optimization algorithm.

[0100] The S2.6 binocular 3D reconstruction principle is based on the triangulation principle. Please refer to... Figure 4As shown in the diagram: Point P is the distance to be measured, Or and Ot are the optical centers of the two cameras, and Pr and Pt are the points on the imaging planes of the left and right cameras respectively (in reality, the imaging planes are behind the optical centers due to the pinhole imaging principle, but are placed in front for ease of calculation). f is the camera focal length, B is the distance to the optical centers, and Z is the distance to be measured. Therefore, based on the principle of similar triangles and corresponding proportional relationships, we can obtain:

[0101] d = X R -X T

[0102]

[0103]

[0104] In the formula, d is called parallax, and X R ,X T The x-coordinates of the two camera imaging points are the left and right coordinates, respectively. The distance Z can be calculated based on the proportional relationship.

[0105] S2.7 Based on the principle in S2.6 above, the key to obtaining the point cloud data of the part from the current viewpoint lies in binocular stereo matching: that is, matching the corresponding pixel information of the part in the images of the left and right cameras. After matching, subtracting the corresponding points in the image coordinates yields the disparity map matrix D of the left and right cameras. Therefore, the phase matrix Φ(x,y) calculated through step S2.5 above is... left ,Φ(x,y) right "This allows for rapid stereo matching. The phase-based stereo matching algorithm works as follows: First, the radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2 of the left and right cameras, obtained from calibration in S2.1, are used to correct the phase map for distortion. Then, stereo correction is performed using the left and right camera transformation matrices R, T from S2.1. In the corrected left and right images, corresponding pixels will be on the same horizontal line (i.e., their vertical coordinates are equal). Phase values ​​are searched row by row; those with equal values ​​are considered a pair of corresponding points. All corresponding points are grouped into a set to complete stereo matching. Finally, the disparity map matrix D is obtained by subtracting the horizontal coordinates of the corresponding point images."

[0106] S2.8 Input camera calibration S2.1 The reprojection matrix Q and the disparity map matrix D obtained in the above steps can be used to calculate the 3D point cloud of the part in the world coordinate system (with the optical center of the left camera as the origin) in the current viewpoint. The current viewpoint point cloud data can be output by calling the reprojectImageTo3D() function of the OpenCV library. The principle formula is as follows:

[0107]

[0108] Where Xw / W, Yw / W, and Zw / W are world coordinates, u and v are image coordinates, d is disparity, and Q is the reprojection matrix.

[0109] In step S2.9, the transformation matrix required for multi-view point cloud stitching needs to be calculated by identifying the pasted marker points. The calculation process is shown in Figure (5). First, the threshold parameter can be manually changed using the modulation function in S2.4 to restore the 24 patterns to grayscale images without projected stripes. The regions where the pasted circular black and white non-coded marker points are located are separated by performing opening and closing operations and binarization using image processing algorithms. Based on the OpenCV library, the morphologyEx function can be used to implement image dilation and erosion, and the adaptiveThreshold function can be used to adaptively perform binarization segmentation on the grayscale image.

[0110] S2.10 preprocesses the grayscale image using a Gaussian filtering algorithm (GaussianBlur function in the OpenCV library), then uses the Canny operator function in the library to extract the boundary point information of the grayscale image, and uses Hough transform to detect the elliptical contour of the marker points (fitEllipse function). Marker points are selected based on parameters such as the roundness of the ellipse, the ellipse axis length at a preset shooting distance, and the grayscale information of the sampled circular coded points (inner circle white, outer circle black). The CCT coded point recognition process is the same as above, except for the addition of a decoding process. The specific decoding process is as follows: According to the code value generation rules in S2, the middle circle pixel values ​​of the coded points to be identified are sampled around the center of the identified circle, and then decoded based on the grayscale values, following the same rules as the generation rules in S2.

[0111] The center parameter of the fitted ellipse in S2.11 is the center of the circle identified in the image. The three-dimensional coordinates corresponding to the two-dimensional coordinates of this center are mapped from the calculated point cloud and recorded as the current viewpoint marker set A. The robotic arm moves to the next target point and repeats steps S2.3-S2.10 to obtain marker set B. In order to transform the part point cloud calculated from multiple shooting perspectives to a unified coordinate system and complete accurate registration, that is, to overlap and stitch the point clouds with common areas, the calculated marker set A and B will be used to perform marker point correspondence matching and transformation matrix calculation.

[0112] After obtaining the 3D point sets A and B of the marker points from two perspectives in S2.12, the key to calculating the transformation matrix lies in finding at least three pairs of non-collinear marker point coordinates common to both perspectives, i.e., congruent triangles. Therefore, the first step is to match the same point from different perspectives. Since the 3D coordinates from each perspective are established in a coordinate system with the optical center of the left camera at the current pose as the origin, the obtained 3D coordinates are relative coordinates and cannot be directly matched. However, the distance between the marker points is constant, so we can establish the topological relationship between the points in the point set for matching. The matching principle is introduced below.

[0113] S2.13 First, based on the distance invariance, establish the distance description matrix for each point cluster. Let the three-dimensional point set of the marker points be A{p1~p... m} and B{q1~q n The distance between each point in the set and the rest of the set is given by the following formula:

[0114]

[0115] In the formula p m p m-1 q n q n-1 This represents the 3D distance between two points. Then, a row-by-row matching process is performed on these two distance matrices, setting a threshold (points less than the threshold are considered equal distances), and identifying the pair of points p with the highest number of equal distances between two rows. r With q t , to this pair of points {p r ,q t} is recognized as the first pair of matching points and used as the basic origin.

[0116] Next, construct distance vectors a and b from the fundamental origin:

[0117] a = [p] r p1,p r p2...p r p m ]

[0118] b = [q t q1,q t q2...q t q n ]

[0119] In the formula, p represents p r With q t The vector is constructed by sorting the distances to the remaining points. Similarity matching is performed again to find two more pairs of points {p}. r ′,q t ′},{p r ,q t "} and satisfy the constraint condition: {p r,p r ′,p r} and {q t ,q t ′,q t "} Forming congruent triangles means that corresponding sides are equal. At this point, three pairs of matching points have been found, and the next step of solving the transformation matrix can be performed. This algorithm adopts a publicly available algorithm in the existing technology. For the specific application of the marker point matching algorithm based on distance invariance, please refer to the third paragraph of the master's thesis "Research on Automatic Registration Method of Point Cloud Based on Key Point Matching" (author: Zhang Bin) published in 2020 by Minnan Normal University.

[0120] After obtaining the three pairs of points in S2.14, the transformation matrix required for point cloud registration can be obtained using SVD decomposition, as shown in the following formula:

[0121]

[0122] In the formula, J is the objective function, k is the number of matching point pairs (3 here), R,T is the rotation and translation matrix to be solved, and q i ,p i Given the three-dimensional coordinates of the corresponding point, we need to minimize J.

[0123] S2.15 applies the obtained transformation matrix to the point clouds obtained from the two perspectives, that is, unifies the two point clouds in the same coordinate system and completes the stitching of the common area. The TransformPointCloud function in the PCL library is called, and the matrix obtained above can be used to complete the initial coarse stitching.

[0124] S2.16 point cloud fine registration can use the most reliable and popular ICP registration algorithm. The ICP algorithm steps are as follows:

[0125] (1) First, estimate the initial transformation matrix T0, and then perform a transformation on each point P in the sampled source point cloud data (the reference point cloud data). ai Applying the transformation matrix T0, we obtain P. ai ′.

[0126] (2) From the target point cloud data (the point cloud data that needs to be registered and transformed with the reference), use K-DTree to quickly search for each distance P. ai The nearest point P bi , with P ai ′ Form corresponding point pairs.

[0127] The optimal transformation matrix ΔT is obtained by acquiring the point pairs:

[0128]

[0129] (3) The conditions for convergence are the iteration error and the number of iterations. If convergence is achieved, calculate T = ΔT * T0 and output the final transformation matrix. If not, calculate T0 = ΔT * T0 and return to step (1) for the next iteration.

[0130] S2.17 Repeat the above steps and traverse all calculated viewpoints along the planned path to calculate the point cloud and complete the automatic point cloud stitching to obtain preliminary complete point cloud data, waiting for the next step of processing.

[0131] Data processing functions:

[0132] Because the acquired point clouds are dense, processing them is time-consuming and inefficient, necessitating point cloud simplification. The goal of point cloud simplification is to represent as much model information as possible with as few points as possible, saving space and time for subsequent point cloud processing. Currently, point cloud data simplification methods can be divided into two main categories based on whether a triangular mesh is constructed: mesh-based simplification methods and methods that directly simplify the point cloud. The principle of the simplification algorithm is to divide the overall point cloud into small cubes according to requirements and resolution, with each small cube containing a number of points. The original algorithm takes the center of the small cube to represent all the points inside, while the improved algorithm calculates the centroid of the point set inside the cube as a sample.

[0133] S3.1 First, the overlapping parts of the stitched data need to be preprocessed to remove redundant noise and invalid points. The coarse point cloud obtained is then optimized using point cloud processing algorithms. Built-in algorithm functions from the PCL open-source library are used, including downsampling filtering (RandomSample), outlier removal (setStddevMulThresh), normal adjustment (setNormalEstimationMethod), and hole filling (BoundaryEstimation), among other steps, to obtain accurate point cloud data.

[0134] S3.2 performs triangulation reconstruction on the final accurate point cloud data, uses the reconstruct function in PCL to obtain the STL triangular mesh model, and converts it into a STEP model using the OCC library. The model is then imported into the Geomagic software's numerical model comparison analysis function for accuracy testing.

[0135] The step model obtained in S3.3 is compared with the theoretical data model of the part. Through manual interaction or global comparison, the error data such as the curvature, straightness, and roughness of the part manufacturing can be compared intuitively and accurately, providing a basis for subsequent process optimization.

[0136] This invention targets complex aerospace parts. Based on surface structure 3D reconstruction technology, combined with model processing viewpoint generation and path planning technologies, it automates the scanning of parts and performs multi-view point cloud stitching. After obtaining the complete point cloud, it automatically processes and reconstructs it using triangulation. The resulting high-precision model can be compared with the theoretical model to determine the part's manufacturing errors.

[0137] The following practical example verifies the high-precision model that can be obtained by the above embodiments and the comparison report with the theoretical model.

[0138] In this actual operation, a Dahua A7A20MG9 12-megapixel industrial camera was used, and a Texas Instruments DLP4500 industrial projector was selected. A schematic diagram of the scanning device's specific placement is shown in Appendix (6). The robotic arm is a KUKA-kr20r1810 with a load capacity of 20KG and an accuracy of 0.05mm. The industrial computer is configured with an AMD Ryzen 7 1800X motherboard, an Intel i7-8700 CPU, 16GB DDR4 memory, and a GTX1650 graphics card.

[0139] The part being scanned is a component of an aero-engine, as shown in Figure (7). The scanning requirement is to measure the planar area of ​​the upper surface.

[0140] First, viewpoint generation and path planning are performed. Since the measurement area required by the model is within the measurement space, model partitioning is not required in this case. The software of this invention first selects the area to be detected in the theoretical model, then generates candidate viewpoints, setting the filtering parameters as occlusion rate, verticality, and feature number. The final optimal pathpoint order is shown in the table below.

[0141] Optimal path point:

[0142]

[0143] In the table, home and homeback are the initial points, and the rest are the selected optimal path points.

[0144] The 3D reconstruction function will be performed next. First, calibration is performed, and the calibration result matrix is ​​shown in the table below. For detailed explanations of the parameters, please refer to the above text.

[0145] Left camera intrinsic parameter matrix:

[0146]

[0147] Left camera distortion coefficient matrix:

[0148]

[0149] Right camera intrinsic parameter matrix:

[0150]

[0151] Right camera distortion coefficient matrix:

[0152]

[0153] Dual-target RT matrix:

[0154]

[0155] The robotic arm drives the scanner to start taking pictures in the order of the generated points. The projected three-frequency eight-step sine fringe pattern is shown below. Figure 8 The continuous phase map and automatic threshold background segmentation map obtained by multi-frequency heterodyne phase solution algorithm are shown below. Figure 9 .

[0156] The point cloud data after using the threshold segmentation algorithm of this invention are compared as follows:

[0157] Comparison of point cloud data after 3D reconstruction

[0158]

[0159] After obtaining the continuous phase map of the part, the disparity map can be calculated using the stereo matching algorithm. See Figure 10 .

[0160] The reprojection matrix Q obtained during system initial calibration is:

[0161]

[0162] By calling the `reprojectImageTo3D()` function in OpenCV and using the Q-matrix and disparity map, the 3D point cloud of the part from the current viewpoint can be obtained. Figure 11 .

[0163] By combining CCT coded points with circular non-coded points, and using coded markers to establish part numbers, the type and principle of parts can be quickly identified and matched.

[0164] In terms of the model, the coded points are pasted onto the parts or tooling to prepare data for the subsequent multi-view point cloud stitching.

[0165] Steps: First, identify and decode the generated coded marker points, preprocess the image, then find the circular outline using filtering and the Canny algorithm to locate the region where the coded points are located. Select the ROI region, and determine whether it is a coded point by sampling the code values ​​of the inner, middle, and outer layers. Then, reverse-engineer the unique code value according to the rules for generating the code, as described above. In this example, only one type of part is measured, so coded marker points are not needed. However, they can be used in production lines measuring multiple parts.

[0166] After the pasted circular non-coded points are identified by image algorithm and ellipse fitting, the three-dimensional coordinates of the marker points are calculated using epipolar search and binocular triangulation principles.

[0167] The following step involves point cloud stitching: After acquiring point clouds from two consecutive perspectives, the stitching process begins. After establishing the distance matrix between marker points in both perspectives, at least three pairs of marker points are matched according to certain rules. SVD decomposition yields the transformation matrix, completing the coarse stitching of the point cloud. The ICP algorithm is then used iteratively to complete the fine stitching. The rotation transformation matrix calculated by SVD is as follows:

[0168]

[0169] The coarse approval and fine ICP matching were performed using the PCL transformation matrix function, and the results are displayed. (See attached image for details.) Figure 12 .

[0170] Finally, the robotic arm completed all the shooting tasks. After the system software processed all the point cloud data, it performed triangulation reconstruction, imported it into the third-party Geomagic software, performed measurements according to the measurement tasks, and generated a report. Figure 13 .

Claims

1. A method for automated 3D inspection of complex aerospace parts using structured light, characterized in that, Includes the following steps: (1) Obtain the theoretical data model of complex aerospace parts; (2) The theoretical data model of the part is partitioned with the measurement space as a constraint; (3) Generate initial viewpoints for the theoretical data models of the parts completed in each partition. Establish a spherical coordinate system with the center of each partition as the center and the shooting distance as the radius. Generate several viewpoints covering the spherical surface by deviating the vector angles from the center of the sphere in the longitudinal and transverse directions by an angle. (4) Evaluate each viewpoint and select the best viewpoint for each surface based on the number of features captured and the occlusion rate under the shooting conditions of that viewpoint, so as to obtain the top few best viewpoints for each partition. (5) After the screening is completed, all the best viewpoints are counted, the optimal path is planned for the robotic arm, and the optimal path passing through each optimal viewpoint is calculated. The optimal robotic arm path point order is calculated by ant colony algorithm for all the generated optimal viewpoints. The preset start point and end point of the robotic arm are added to form the optimal path for this part scanning. (6) The binocular camera is calibrated. The intrinsic and extrinsic parameters of the binocular camera are obtained by using Zhang's checkerboard calibration method. After the image is corrected by combining the above intrinsic and extrinsic parameters, it is directly used to calculate the reprojection matrix Q of the image pixel coordinates to world coordinates. (7) Provide real complex aerospace parts and tooling for fixing the complex aerospace parts, and set non-coded markers on the tooling for binocular cameras to identify the type of parts; (8) The robotic arm starts up and traverses each viewpoint in the order of the obtained path. It projects 24 three-frequency eight-step phase-shift patterns generated by the industrial projector onto the surface of the complex aerospace parts, and then uses an industrial camera to take follow-up pictures and transmit the data to the computer. (9) Calculate three-dimensional point cloud: Use 24 three-frequency eight-step phase shift patterns captured to perform multi-frequency heterodyne phase decomposition algorithm; After calculating two consecutive phase images of the modulated aerospace complex parts, the phase images are automatically thresholded to distinguish the aerospace complex parts from the background in the phase images. (9) Match the pixel information of the parts in the left and right camera images. After the matching is completed, subtract the corresponding points of the image coordinates to obtain the disparity map matrix D of the left and right cameras. Calculate the three-dimensional point cloud of the parts in the world coordinate system captured by the current viewpoint using the reprojection matrix Q and the disparity map matrix D obtained in the above steps, and output the point cloud data of the current viewpoint. (10) Calculate the transformation matrix required for multi-view point cloud stitching by identifying the pasted marker points; apply the obtained transformation matrix to the point clouds obtained from the previous and next views, that is, unify the two point clouds under the same coordinate system and complete the stitching of the common area. Use the matrix obtained above to complete the preliminary coarse stitching of the point cloud. (11) Perform fine registration of point cloud by ICP registration algorithm, traverse all calculated viewpoints, calculate point cloud and complete automatic point cloud stitching to obtain preliminary complete point cloud data; (12) The coarse point cloud obtained is optimized by point cloud processing algorithm to obtain accurate point cloud data; (13) Triangulate and reconstruct the obtained precise point cloud data to obtain the STL triangular mesh model and convert it into the STEP model; (14) The obtained step model is compared with the theoretical data model of the part to obtain the error data of the part manufacturing.

2. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 1, characterized in that: In step (5), after the filtering is completed, all optimal viewpoint sets are counted. For a robotic arm carrying a scanner, optimal path planning is performed to calculate the most efficient and interference-free path that passes through each optimal viewpoint. The path weight is the cost of the robotic arm's movement. Calculating the optimal path for traversing a set of points simplifies to a Traveling Salesman Problem (TSP) in a weighted directed graph, where the path weights are set as the motion cost of the robotic arm. In Cartesian space, the motion cost is defined as the angles along each axis that the robotic arm needs to change from its current viewpoint to the next viewpoint. The following formula represents the parameters for calculating the motion cost: ， in The cost to the robotic arm when the two viewpoints change. The cost coefficient, and The angles of each axis are shown in the front and rear views of axes 1 to 6. All generated optimal viewpoint sets The optimal sequence of robotic arm path points is calculated using the algorithm described above. ,in ~ The optimal viewpoint set is generated and ordered according to the new traversal points obtained by the path planning algorithm, plus the robot arm's preset start and end points, thus forming the optimal path for this part scanning. , The starting point of the optimal path. The optimal path endpoint serves as the data foundation for subsequent 3D reconstruction.

3. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 2, characterized in that: The scanner includes two industrial cameras and one industrial projector. The two industrial cameras are placed on a horizontal fixed frame with their optical centers on the same straight line and their shooting directions intersecting at approximately 30 degrees. The industrial projector is placed between the two cameras. The positions of the cameras and the projector satisfy a certain relationship: the rectangular area projected by the projector is completely contained within the common field of view of the images captured by the two cameras.

4. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 3, characterized in that: In step (6), the intrinsic parameters include the coefficients for transforming the left and right camera image coordinate systems to the world coordinate system with the optical center of the left camera as the origin. Radial distortion coefficients of left and right cameras Tangential distortion coefficient The extrinsic parameters include the transformation matrices that convert the two camera image planes to the stereo parallel model. .

5. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 4, characterized in that: In step (9), the two consecutive phase diagrams of the modulated aerospace complex parts are calculated. Subsequently, due to the problems of repeated phase calculation for invalid measurement areas, including background and shadow areas, and mismatch in invalid measurement areas during 3D reconstruction, the segmentation of the effective measurement area was studied based on the fringe modulation degree. Automated threshold segmentation of the phase map was performed to distinguish the part from the background. Algorithm steps: The Otsu algorithm was introduced into the modulation degree threshold calculation; under different lighting and environmental backgrounds, the optimal threshold was determined using the Otsu algorithm, removing discrete noise while automatically segmenting the area of ​​the part to be measured from the irrelevant background area; the modulation degree function is as follows: In the formula, x, y are the image coordinates, and B(x, y) is the modulation value of the pixel to be calculated. The number of phase shift steps used is 8 steps. n (x,y) represents the pixel value at the corresponding coordinates in the 24 collected images; After calculating the modulation image of these 24 images, a threshold is set and image processing algorithms are used to segment the parts from the irrelevant background; The Otsu algorithm is introduced to calculate the modulation segmentation threshold. This threshold, along with the modulation map B(x,y), is used to segment the image region containing the part from the background-independent region, resulting in a continuous phase matrix image of the part after removing the irrelevant regions. Solve for the continuous phase used for stereo matching Then, the phase values ​​are optimized using a step correction algorithm to obtain the optimized phase matrix. .

6. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 5, characterized in that: In step (9), the radial distortion coefficients of the left and right cameras are used. and tangential distortion coefficient Distortion correction is performed on the phase map; then stereo correction is performed using the left and right camera transformation matrix. In the corrected left and right images, corresponding pixels are on the same horizontal line (i.e., their vertical coordinates are equal). Phase values ​​are searched row by row; those with equal values ​​are considered a pair of corresponding points. All corresponding points are grouped into a set to complete stereo matching. Then, the horizontal coordinates of the corresponding point images are subtracted to obtain the disparity map matrix D. The reprojection matrix Q and the disparity map matrix D are input to calculate the 3D point cloud of the part captured from the current viewpoint in the world coordinate system, with the optical center of the left camera as the origin. The OpenCV library's `reprojectImageTo3D()` function is called to output the current viewpoint point cloud data. The principle formula is as follows: ， in , , As a world coordinate system, For image coordinates, Let be the parallax and Q be the reprojection matrix.

7. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 6, characterized in that: In step (10), the transformation matrix calculation process is as follows: First, the threshold parameter is manually changed using the modulation function to restore the 24 patterns to grayscale images without projected stripes. The area where the attached circular black and white non-coded markers are located is separated by the image algorithm. The image dilation and erosion are implemented using the morphologyEx function based on the OpenCV library. The adaptiveThreshold function can adaptively perform binarization segmentation on the grayscale image. After preprocessing the grayscale image using the GaussianBlur function in the OpenCV library, the Canny operator function in the OpenCV library is used to extract the boundary point information of the image, and the Hough transform is used to detect the elliptical contour of the marker point. The marker point is then selected based on the roundness of the ellipse, the length of the ellipse axis at the preset shooting distance, and the grayscale information parameters of the sampled circular coded point. The center parameter of the fitted ellipse is the center of the circle identified in the image. The three-dimensional coordinates corresponding to the two-dimensional coordinates of the center are mapped from the calculated point cloud and recorded as the current viewpoint marker set A. The robotic arm moves to the next target point and repeats the above steps to obtain the marker set B. The calculated marker set A and marker set B are used to perform marker point correspondence matching and transformation matrix calculation. Based on distance invariance, a distance description matrix is ​​established for each point cluster. Let the set of marker points A{ ~ } and the set of marker points B{ ~ The distance between each point in the set and the rest of the set is given by the following formula: ， In the formula , This represents the 3D distance between two points; then, a row-by-row matching is performed on these two distance matrices, a threshold is set, and the pair of points with the most equal distances in a given two-row matrix is ​​identified. and , to these two points Recognize the first pair of matching points as the basic origin, and then construct the distance vector from the basic origin. , : ， In the formula, respectively and The vector is constructed by sorting the distances to the remaining points; similarity matching is performed again to find two more pairs of points. }, { } and satisfy the constraints: { }and{ If congruent triangles are formed, meaning corresponding sides are equal, then three pairs of matching points have been found, and the next step is to solve the transformation matrix. After obtaining three pairs of points, the transformation matrix required for point cloud registration is obtained using SVD decomposition, as shown in the following formula: ， In the formula, Let be the objective function. It is the number of matching point pairs. Take 3. The rotation and translation matrix to be solved. Let the three-dimensional coordinates of the corresponding point be such that Minimum; The obtained transformation matrix is ​​applied to the point clouds obtained from the front and rear views, that is, the two point clouds are unified in the same coordinate system and the common area is stitched together. The TransformPointCloud function in the PCL library is called to use the matrix obtained above to complete the initial coarse stitching.

8. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 7, characterized in that: In step (11), the ICP registration algorithm steps are as follows: (11.1) First, estimate the initial transformation matrix. For each point in the sampled set of marker points A Apply the transformation matrix ,get ; (11.2) From the set of markers B, use a KD Tree to quickly search for each distance. nearest point ,and Form corresponding point pairs; The optimal transformation matrix is ​​obtained by acquiring the point pairs. : ， (11.3) The condition for convergence is the iteration error and the number of iterations. If convergence is achieved, the calculation... Output the final transformation matrix; otherwise, calculate... Then return to step (11.1) for the next iteration.

9. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 8, characterized in that: In step (12), the overlapping parts of the splicing are preprocessed to remove excess noise and invalid points; the coarse point cloud obtained is optimized by point cloud processing algorithm, using built-in algorithm functions in the PCL open source library, including downsampling filtering, noise removal of outliers, normal adjustment, and hole filling, to obtain accurate point cloud data.

10. The automated three-dimensional inspection method for complex aerospace parts using surface structured light according to claim 9, characterized in that: In step (13), the final accurate point cloud data is triangulated and reconstructed. The reconstruct function in PCL is used to obtain the STL triangular mesh model and convert it into a STEP model using the OCC library. The model is then imported into the Geomagic software for accuracy testing using the numerical model comparison analysis function.

Images