A method for detecting gap and step difference of aircraft skin based on surface structured light

Abstract

The present invention relates to a method for detecting gap and step difference of aircraft skin based on surface structured light, comprising: using binocular surface structured light method to obtain skin seam image and original skin seam point cloud, segmenting seam features on skin seam image based on light compensation fine-tuning large model deep learning network, mapping seam features on skin seam image to point cloud data, and performing error verification after obtaining skin seam pixel area of ​​image; then performing centerline fitting and profile line extraction, obtaining point cloud data distribution of profile line, and finally analyzing and calculating edge chamfer seam gap and step difference of obtained profile line point cloud data, the method can quickly complete accurate detection of aircraft skin seam gap and step difference by combining multi-modal analysis of two-dimensional image by segmentation large model and processing point cloud by surface structured light. The method solves the problems of low efficiency of laser detection and difficulty in edge chamfer seam detection of traditional methods.

Description

Technical Field

[0001] The invention relates to the technical field of aircraft skin seam detection, and in particular to an aircraft skin gap and step difference detection method based on surface structured light. Background Art

[0002] During the assembly process of the aircraft, the inspection of the skin's external dimensions involves the inspection of many seam structures. The inspection of the aircraft's skin seams must meet high-precision standards, which not only ensures the aircraft's aerodynamic shape and sealing performance, but also has a key impact on the stealth performance of the new generation of fighter jets. Faced with the heavy task of seam inspection, it is crucial to efficiently and accurately obtain the gap and step difference data of the seam structure for the overall inspection of the aircraft. Traditional seam inspection methods rely on manpower and tools, and have problems such as low accuracy, poor efficiency, and poor traceability, and cannot meet the special needs of high precision and high efficiency.

[0003] With the rapid development of technologies such as non-destructive testing and digital testing, the most advanced seam detection method is to collect data with the help of non-contact digital testing equipment and build a detection system with corresponding algorithms. Structured light detection technology has been widely used in seam detection due to its high efficiency, accuracy, speed and good adaptability. Nanjing University of Aeronautics and Astronautics, Beijing University of Aeronautics and Astronautics and Zhejiang University have successively proposed point cloud seam gap and step difference detection methods based on infrared laser, single-line structured light or multi-line structured light. Specifically, the traditional construction method of its step difference and gap model is as follows: for step difference detection, a plane is fitted with the help of two laser lines on the step difference plane, and the step difference height is determined by calculating the distance between the planes; for gap detection, the image is first refined to find the breakpoints of the gap, connect the breakpoints on the same side into a straight line, and then calculate the average distance from the two points on the other side to the straight line to obtain the gap width.

[0004] However, the above methods are mainly aimed at single straight sharp edge seams. It is difficult to accurately detect gaps and step differences for chamfered edge seams on aircraft skins. In addition, due to the large number and length of aircraft skin seams, the application scope and measurement speed of detection methods based on line structured light or infrared laser are greatly limited.

[0005] To solve the above situation, the present invention adopts a binocular vision detection method based on surface structured light to obtain images and point clouds. The two-dimensional images processed by the large model segmentation algorithm provide assistance for the three-dimensional point cloud data analysis, and realize the rapid and accurate detection of the gap and step difference of the chamfered seam of the aircraft skin edge. This method can complete the rapid and accurate segmentation of the chamfered seam of the edge, greatly improving the efficiency and accuracy of the aircraft skin seam detection. Summary of the invention

[0006] In order to solve the problems of low laser detection efficiency and difficulty in detecting edge chamfer seams, the present invention provides a method for detecting gaps and step differences in aircraft skin based on surface structured light. This method uses a multi-modal combination of large model segmentation to analyze two-dimensional images and surface structured light to process point clouds, which can quickly and accurately detect gaps and step differences in aircraft skin seams.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for detecting gap and step difference of aircraft skin based on surface structured light, comprising the following steps:

[0008] S1. Use binocular structured light method to obtain skin seam image and original skin seam point cloud;

[0009] S2, Mask-SAM, a fine-tuned large model deep learning network based on light compensation, segments the seam features on the skin seam image;

[0010] S3, mapping the seam features on the skin seam image to the point cloud data, obtaining the skin seam pixel area of ​​the image and then performing error verification;

[0011] S4, performing centerline fitting and profile line extraction to obtain point cloud data distribution of the profile line;

[0012] S5. Analyze and calculate the edge chamfer gap and step difference of the obtained profile line point cloud data.

[0013] Further, in step S1, a skin seam image and an original skin seam point cloud are obtained using a binocular surface structured light method; the specific process includes the following steps:

[0014] S11, a structured light generator capable of emitting surface structured light and two industrial cameras are installed at a certain angle and position according to their optical axes, and a three-step phase-shifted fringe pattern is projected onto an aircraft skin with a seam using the structured light generator;

[0015] S12. The left and right cameras are symmetrically distributed according to the binocular vision principle, and the structured light generator is placed at the perpendicular midline of the line connecting the optical centers of the two cameras through a supporting frame;

[0016] S13, using a checkerboard calibration plate to calibrate the binocular structured light system, using the relationship between the pixel coordinates of all corner points and the world coordinates to calculate the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion parameters, completing the single-target positioning, and finally completing the binocular positioning through the relative position relationship between the left and right cameras;

[0017] S14, synchronously triggering two calibrated industrial cameras to collect skin seam images, projecting structured light on skin seams of different shapes in different aircraft skin regions and taking photos, to obtain clear and visible skin seam images;

[0018] S15. The original skin seam point cloud Q is obtained by processing the image data through the fringe projection structured light measurement algorithm and the software system.

[0019] Further, in step S15, the original skin seam point cloud Q is obtained by processing the image data through the fringe projection structured light measurement algorithm and the software system; the specific process is:

[0020] First, epipolar correction, phase matching and phase unfolding are performed on the skin seam fringe images collected by the left and right cameras to obtain the disparity map and the left unfolded phase map. Then, the left camera coordinates are used as the measurement coordinates to construct a binocular stereo structured light model and calculate the three-dimensional point cloud information of the seam features. The model is described as:

[0021] ;

[0022] Where: , Respectively represent the depth value of the 3D point in the left and right camera coordinate systems, Represents the depth value of a 3D point in the projector coordinate system. It represents the horizontal coordinate of the projector image plane in the binocular structured light system. Its core function is to solve the three-dimensional coordinates of the object surface through phase encoding. Represents the ordinate of the projector image plane in the binocular structured light system; represents the horizontal coordinate of the left camera image, Indicates the ordinate of the left camera image; represents the horizontal coordinate of the right camera image, Indicates the ordinate of the right camera image; and They are the projection matrices of the left and right cameras after image stereo rectification; is the projection matrix of the projector; for ;ω is the pattern image width; is the number of stripes, and They are the transformation matrices from the corrected left camera coordinates to the projector coordinates and the corrected right camera coordinates. The coordinates of the three-dimensional space point in the left camera coordinate system are solved by solving the eight linear equations provided by the left and right cameras. .

[0023] Further, in step S2, the fine-tuned large model deep learning network Mask-SAM based on light compensation segments the seam features on the skin seam image, and the specific process includes the following steps:

[0024] S21. The large model uses a fine-tuned Mask-SAM network. Based on the SAM pre-trained model, fine-tuning is performed on the aircraft skin seam features. The image dataset with seam pixel annotation is divided into a training set and a test set to train the large model.

[0025] S22, during the training phase, the encoder parameters are fixed to a non-trainable state, i.e., the gradient update function is disabled;

[0026] S23. Before the image is input into the Mask-SAM network, it needs to be grayscaled and adaptively exposed to reduce the influence of the strong light absorption characteristics of the composite skin;

[0027] S24, inputting the skin seam image of the area to be tested into the preprocessed and trained large model, the network can identify the skin seam in the left and right camera images and perform image segmentation.

[0028] Furthermore, step S3 specifically includes: projecting the 3D point cloud onto the right camera image for error verification, converting the points in the left camera coordinate system to the right camera coordinate system, extracting the actual feature points in the right image, and calculating the reprojection error between the projection of the 3D point on the right camera image and the actually detected feature points. .

[0029] Furthermore, in step S4, centerline fitting and profile line extraction are performed to obtain the point cloud data distribution of the profile line. The specific process includes the following steps:

[0030] S41. In the workpiece coordinate system, a section parallel to the skin surface is established, the edge points on both sides of the point cloud are paired, and the midpoint perpendicular line is taken every 0.5 mm along the seam direction of the skin, and the midpoint perpendicular point of the point cloud on both sides of the seam is calculated as the preliminary center point;

[0031] S42. A uniform cubic B-spline curve is randomly initialized according to the preliminary center point. The total equation of the B-spline curve is:

[0032] ;

[0033] in, are the characteristic points of the control curve, is the K-order B-spline basis function;

[0034] S43. The distance measurement from the perpendicular point in the slit to the spline curve is realized by using the Frenet frame combined with the square distance minimization method. That is, a coordinate system is constructed with a point on the spline curve to be fitted as the origin and the tangent vector and normal vector at the point as the coordinate axes. The square distance analysis function from the perpendicular point in the slit to the spline curve is defined as:

[0035] ;

[0036] in, The vertical point of the joint To the corresponding perpendicular point on the spline curve The distance between is the curvature at the foot point, and The foot points The unit tangent vector and normal vector at ;

[0037] S44, perform endpoint constraints and cusp constraints on the B-spline center curve, use the minimized objective function to adjust the positions of other control points, and minimize the total objective function. It is defined as the sum of the vertical point errors. The specific formula is as follows:

[0038] ;

[0039] S45, repeat step S44 until the objective function value is less than the error threshold Or the number of iterations exceeds a fixed value Finally, the iteratively optimized cubic B-spline center curve and its parameter equation are obtained. ;

[0040] S46, dividing the cubic B-spline center curve obtained in step S45 into segments according to arc length, and then calculating the normal plane and the profile line, and obtaining the two-dimensional point set data distribution of the profile line.

[0041] Further, in step S41, the midpoint of the point clouds on both sides of the seam is calculated as the preliminary center point; the specific process includes the following steps:

[0042] S411, filter outliers and gap zero points from the original point cloud Q, and smooth the point cloud to obtain the point cloud ;

[0043] S412, point cloud Cluster segmentation is performed using the regional growing clustering method based on Euclidean distance to divide the skin seam point cloud into two categories: left and right. and ;

[0044] S413, calculate point clouds separately and Get their edge points respectively , , calculate point cloud and The curvature changes of all adjacent points in the , and the points with the largest curvature changes on both sides are recorded as the edge point sets on both sides and ;

[0045] S414: Estimation of the local main direction is performed, and the local main direction of each point is calculated using PCA analysis to ensure the connection direction of the paired point pairs The direction is consistent with the extension direction of the seam. If the direction deviation exceeds the angle difference threshold of 15°, the pairing is eliminated;

[0046] S415, perform midpoint calculation, and calculate the effective matching points. , calculate the vertical point , as the preliminary centerline candidate point;

[0047] S416, DBSCAN clustering is used to remove outliers to reduce the interference of bifurcation or noise. The key definition is as follows:

[0048] ;

[0049] in, is a dataset, is the Euclidean distance, As the core point, Indicate point of Neighborhood, that is, all the points in the data set The distance does not exceed the threshold The set of points, It is the basic unit for defining core points and extended clustering in DBSCAN. The neighborhood of contains at least minPts points, including itself. minPts is the threshold for defining the core point density in DBSCAN, which directly affects the number of clusters, noise filtering, and algorithm robustness.

[0050] Further, in step S46, the cubic B-spline center curve obtained in step S45 is segmented according to arc length to obtain the normal plane and the profile line, and obtain the two-dimensional point set data distribution of the profile line; the specific process includes the following steps:

[0051] S461, centerline arc length segmentation, for parameterized curves , in the parameters Small increments nearby , the length of the displacement element of the corresponding curve in three-dimensional space is ,right from arrive Integrate to get the total arc length of the curve ,from arrive The calculation formula is:

[0052] ;

[0053] in, , , The curve is derivatives in direction;

[0054] S462, after the center line is divided into sections, there are midpoints, calculate the normal section plane of each midpoint respectively, and suppose a midpoint is ,in It is the first segment after the center line is segmented. The parameter value corresponding to the midpoint is used to locate the specific position of the point on the curve, and the equation of the normal section plane of the midpoint is:

[0055] ;

[0056] in, is the center line of the B-spline Tangent vector at the point;

[0057] S463, respectively calculate the distance between the data points on the point cloud and the normal section plane, and set a distance threshold max to extract the point set of the original skin seam point cloud Q near the normal plane. ;

[0058] S464, point set Plane projection on the normal section plane obtains the two-dimensional point set data distribution of the section line.

[0059] Further, in step S5, the obtained profile point cloud data is subjected to edge chamfer gap and step difference analysis and calculation, and the specific process includes the following steps:

[0060] S51, using the local PCA method to perform cluster segmentation on the two-dimensional point set of the normal section plane based on the curvature threshold and the Euclidean distance, and segmenting the left and right rounded corners of the skin seam and the straight line area;

[0061] S52, use Gaussian mixture model GMM to fit two straight lines in the skin straight line area of ​​the two-dimensional point set of the normal section plane and , then the distance between the two straight lines That is, the difference in the seam steps, where and is a point on the fitted line, and is the direction vector of the line, is the angle between the two straight line direction vectors, and the parameter and is a scalar, representing the direction of the vector along the line and A linear scale of , used to generate all points on the line;

[0062] S53, respectively use the RANSAC fitting method to perform circular fitting on the left and right edge fillet areas of the skin seam to obtain the left fillet equation and , and then the skin gap can be obtained , where the center coordinates and radius of the left fillet are and , the center coordinates and radius of the right fillet are and .

[0063] By means of the above technical solution, the present invention provides an aircraft skin gap and step difference detection method based on surface structured light, which has at least the following beneficial effects:

[0064] The present invention uses binocular surface structured light method to obtain skin seam images and original skin seam point clouds, and uses Mask-SAM, a deep learning large model for fine-tuning seam image segmentation based on light compensation, to detect seam features on the image, and maps the seam features on the skin seam image to point cloud data. After obtaining the skin seam pixel area of ​​the image, error verification is performed to verify the quality of the three-dimensional point cloud and the accuracy of the camera calibration; then centerline fitting and profile line extraction are performed to obtain the point cloud data distribution of the profile line, and finally the obtained profile line point cloud data is analyzed and calculated for the edge chamfer seam gap and step difference. This method uses a large segmentation model to analyze two-dimensional images and a multi-modal combination of surface structured light processing point clouds to quickly complete accurate detection of aircraft skin seam gaps and step differences. This solves the problems of low efficiency of laser detection and difficulty in edge chamfer seam detection in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0066] Figure 1 It is a schematic diagram of the process of the present invention;

[0067] Figure 2 This is a schematic diagram of the architecture of the aircraft skin seam detection system based on binocular structured light;

[0068] Figure 3 This is a schematic diagram of the framework of the aircraft skin seam image detection network;

[0069] Figure 4 It is a schematic diagram of the center line and normal section plane of the seam fitting;

[0070] Figure 5Schematic diagram of the edge chamfer gap and step difference detection calculation model. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0072] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0073] Please refer to Figure 1-Figure 5 , shows a specific implementation of this embodiment. This embodiment uses binocular surface structured light method to obtain skin seam images and original skin seam point clouds, and uses Mask-SAM, a deep learning model for fine-tuning seam image segmentation based on light compensation, to detect seam features on the image, and maps the seam features on the skin seam image to the point cloud data. After obtaining the skin seam pixel area of ​​the image, error verification is performed to verify the quality of the three-dimensional point cloud and the accuracy of the camera calibration; then centerline fitting and profile line extraction are performed to obtain the point cloud data distribution of the profile line, and finally the obtained profile line point cloud data is analyzed and calculated for the edge chamfer seam gap and step difference. This method uses the segmentation large model to analyze two-dimensional images and the surface structured light to process point clouds in a multi-modal manner, which can quickly complete the accurate detection of aircraft skin seam gaps and step differences. It solves the problems of low efficiency of laser detection and difficulty in edge chamfer seam detection in traditional methods.

[0074] Please refer to Figure 1 This embodiment proposes a method for detecting gap and step difference of aircraft skin based on surface structured light, and the method comprises the following steps:

[0075] S1. Use binocular structured light method to obtain skin seam image and original skin seam point cloud;

[0076] As a preferred implementation of step S1, in step S1, a skin seam image and an original skin seam point cloud are obtained using a binocular surface structured light method; the specific process includes the following steps:

[0077] S11. A structured light generator capable of emitting surface structured light and two industrial cameras are installed at a certain angle and position according to their optical axes, and a three-step phase-shifted fringe pattern (at least three phase-shifted images are required to solve the phase) is projected onto the aircraft skin with a seam using the structured light generator;

[0078] S12. According to the binocular vision principle, the left and right cameras are symmetrically distributed, and the structured light generator is placed at the perpendicular midline of the line connecting the optical centers of the two cameras through a supporting frame, such as Figure 2 As shown, in order to ensure the measurement range of the structured light measurement device, combined with the actual test results, the angle between the two cameras is set to 50°, and the baseline length (the distance between the optical centers of the left and right cameras) is set to 175mm; is the pixel coordinate, is the background intensity, is the modulation intensity, is the phase value, refers to the phase shift, then the intensity of the three-step phase shift fringe pattern can be described by the formula:

[0079] ;

[0080] ;

[0081] S13. Use the checkerboard calibration plate to calibrate the binocular structured light system, establish the correspondence between the camera pixel coordinate system and the world coordinate system, and determine the relative position relationship between the cameras. The principle is as follows: adjust the object distance between the checkerboard and the camera to maintain about 300mm, fix the world coordinate system on the checkerboard, that is, the depth coordinate value of each point on the checkerboard in the world coordinate system is 0, and after obtaining the image of the placed calibration plate, the pixel coordinates of each corner point can be obtained using the detection algorithm. , and then the coordinates of each corner point in the world coordinate system are , and then use the relationship between the pixel coordinates of all corner points and the world coordinates to calculate the intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters to complete single-target positioning, and finally complete dual-target positioning through the relative position relationship of the left and right cameras; , are the rotation matrix and translation vector from the world coordinate system to the left camera coordinate system, , are the rotation matrix and translation vector from the world coordinate system to the right camera coordinate system, , They are the rotation matrix and translation vector from the left camera coordinate system to the right camera coordinate system. The position relationship of the right camera relative to the left camera is as follows:

[0082] ;

[0083] Move the chessboard at least 10 times to ensure that its position and posture are evenly distributed within the camera field of view. Repeat the above step S13 to complete the binocular structured light system calibration and record various parameters of the left and right cameras, including the camera focus. , aperture center , the scaling factor of the depth map ;

[0084] S14, synchronously triggering two calibrated industrial cameras to collect skin seam images, projecting structured light on skin seams of different shapes in different aircraft skin regions and taking photos, to obtain clear and visible skin seam images;

[0085] S15. The original skin seam point cloud Q is obtained by processing the image data through the fringe projection structured light measurement algorithm and the software system.

[0086] Specifically, in step S15, the original skin seam point cloud Q is obtained by processing image data through a fringe projection structured light measurement algorithm and a software system; the specific process is:

[0087] First, epipolar correction, phase matching and phase unfolding are performed on the skin seam fringe images collected by the left and right cameras to obtain the disparity map and the left unfolded phase map. Then, the left camera coordinates are used as the measurement coordinates to construct a binocular stereo structured light model and calculate the three-dimensional point cloud information of the seam features. The model is described as:

[0088] ;

[0089] Where: , Respectively represent the depth value of the 3D point in the left and right camera coordinate systems, Represents the depth value of a 3D point in the projector coordinate system. It represents the horizontal coordinate of the projector image plane in the binocular structured light system. Its core function is to solve the three-dimensional coordinates of the object surface through phase encoding. Represents the ordinate of the projector image plane in the binocular structured light system; represents the horizontal coordinate of the left camera image, Indicates the ordinate of the left camera image; represents the horizontal coordinate of the right camera image, Indicates the ordinate of the right camera image; and They are the projection matrices of the left and right cameras after image stereo rectification; is the projection matrix of the projector; for ;ω is the pattern image width; is the number of stripes, and are the transformation matrices from the corrected left camera coordinates to the projector coordinates and the corrected right camera coordinates, respectively. The above calibration parameters are obtained through step S13. The coordinate values ​​of the three-dimensional space point in the left camera coordinate system are further solved by the eight linear equations provided by the left and right cameras. .

[0090] S2, Mask-SAM, a fine-tuned large model deep learning network based on light compensation, segments the seam features on the skin seam image;

[0091] As a preferred implementation of step S2, in step S2, the fine-tuned large model deep learning network Mask-SAM based on light compensation segments the seam features on the skin seam image, and the specific process includes the following steps:

[0092] S21. The large model uses a fine-tuned Mask-SAM network. Based on the SAM pre-trained model, fine-tuning is performed on the aircraft skin seam features. The purpose of fine-tuning is to make a small amount of parameter adjustments on the skin seam detection task to achieve better performance. The image dataset with seam pixel annotation is divided into a training set and a test set to further train the large model.

[0093] S22. In the further training stage of the model, the image encoder part of this example mainly includes block embedding, multi-layer ViT-B and ViT-H. Block embedding is the core step of Vision Transformer (ViT) in processing images. Its function is to convert the input image into a serialized feature vector. ViT-B and ViT-H are variants of ViT models of different scales. ViT-B is used to quickly process images and generate preliminary segmentation results, and deep ViT-H is used to enhance the long-distance dependency segmentation of skin seams. At the same time, it is prompted that the parameters of the encoder are fixed to a non-trainable state, that is, the gradient update function is disabled; training resources are concentrated on the performance improvement of the image encoder and mask decoder, reducing the resource consumption of gradient calculation and parameter optimization;

[0094] S23. Before the image is input into the Mask-SAM network, it needs to be grayscaled and adaptively exposed to reduce the influence of the strong light absorption characteristics of the composite skin. The specific model architecture is as follows: Figure 3 As shown;

[0095] S24, inputting the skin seam image of the area to be tested into the preprocessed and trained large model, the network can identify the skin seam in the left and right camera images and perform image segmentation.

[0096] S3, mapping the seam features on the skin seam image to the point cloud data, obtaining the skin seam pixel area of ​​the image and then performing error verification;

[0097] As a preferred implementation of step S3, step S3 specifically includes:

[0098] S31, let the pixel coordinates of a certain seam pixel point in the left camera image be , the corresponding three-dimensional point cloud coordinates are , Refers to the zoom factor of the left camera, and the corresponding point cloud coordinates can be obtained through the following relationship:

[0099] ;

[0100] in, Refers to the horizontal coordinate of the left camera image, Refers to the vertical axis, Refers to depth, Indicates the focus of the left camera; Indicates the aperture center of the left camera, Represents the z coordinate of the 3D point cloud in the left camera coordinate system;

[0101] S32, projecting the 3D point cloud onto the right camera image for error verification, converting the points in the left camera coordinate system to the right camera coordinate system, extracting the actual feature points in the right image, and calculating the reprojection error between the projection of the 3D point on the right camera image and the actually detected feature points ‌, evaluate the deviation between the projection of the 3D point on the right camera image and the actual detected feature points to verify the quality of the 3D point cloud and the accuracy of the camera calibration. The smaller the reprojection error, the higher the point cloud accuracy. The specific formula is as follows:

[0102] ;

[0103] ;

[0104] in, Refers to the projection point coordinates of the 3D point on the right camera image, , They are the rotation matrix and translation vector from the left camera coordinate system to the right camera coordinate system.

[0105] S4, perform centerline fitting and profile line extraction, such as Figure 4 As shown, the point cloud data distribution of the profile line is obtained;

[0106] As a preferred implementation of step S4, in step S4, centerline fitting and profile line extraction are performed to obtain point cloud data distribution of the profile line. The specific process includes the following steps:

[0107] S41. In the workpiece coordinate system, a section parallel to the skin surface is established, the edge points on both sides of the point cloud are paired, and the midpoint perpendicular line is taken every 0.5 mm along the seam direction of the skin, and the midpoint perpendicular point of the point cloud on both sides of the seam is calculated as the preliminary center point;

[0108] Specifically, in step S41, the midpoint of the point clouds on both sides of the seam is calculated as the preliminary center point; the specific process includes the following steps:

[0109] S411, filter outliers and gap zero points from the original point cloud Q, and smooth the point cloud to obtain the point cloud ;

[0110] S412, point cloud Cluster segmentation is performed using the regional growing clustering method based on Euclidean distance to divide the skin seam point cloud into two categories: left and right. and ;

[0111] S413, calculate point clouds separately and Get their edge points respectively , , calculate point cloud and The curvature changes of all adjacent points in the , and the points with the largest curvature changes on both sides are recorded as the edge point sets on both sides and ;

[0112] S414: Estimation of the local main direction is performed, and the local main direction of each point is calculated using PCA analysis to ensure the connection direction of the paired point pairs The direction is consistent with the extension direction of the seam. If the direction deviation exceeds the angle difference threshold of 15°, the pairing is eliminated;

[0113] S415, perform midpoint calculation, and calculate the effective matching points. , calculate the vertical point , as the preliminary centerline candidate point;

[0114] S416, DBSCAN clustering is used to remove outliers to reduce the interference of bifurcation or noise. The key definition is as follows:

[0115] ;

[0116] in, is a dataset, is the Euclidean distance, As the core point, Indicate point of Neighborhood, that is, all the points in the data set The distance does not exceed the threshold The set of points, It is the basic unit for defining core points and extended clustering in DBSCAN. The neighborhood of contains at least minPts points, including itself. minPts is the threshold for defining the core point density in DBSCAN, which directly affects the number of clusters, noise filtering, and algorithm robustness.

[0117] S42. A uniform cubic B-spline curve is randomly initialized according to the preliminary center point. The total equation of the B-spline curve is:

[0118] ;

[0119] in, are the characteristic points of the control curve, is the K-order B-spline basis function;

[0120] S43. The distance measurement from the perpendicular point in the slit to the spline curve is realized by using the Frenet frame combined with the square distance minimization method. That is, a coordinate system is constructed with a point on the spline curve to be fitted as the origin and the tangent vector and normal vector at the point as the coordinate axes. The square distance analysis function from the perpendicular point in the slit to the spline curve is defined as:

[0121] ;

[0122] in, The vertical point of the joint To the corresponding perpendicular point on the spline curve The distance between is the curvature at the foot point, and The foot points The unit tangent vector and normal vector at ;

[0123] S44, perform endpoint constraints and cusp constraints on the B-spline center curve, use the minimized objective function to adjust the positions of other control points, and minimize the total objective function. It is defined as the sum of the vertical point errors. The specific formula is as follows:

[0124] ;

[0125] S45, repeat step S44 until the objective function value is less than the error threshold Or the number of iterations exceeds a fixed value Finally, the iteratively optimized cubic B-spline center curve and its parameter equation are obtained. ;

[0126] S46, dividing the cubic B-spline center curve obtained in step S45 into segments according to arc length, and then calculating the normal plane and the profile line, and obtaining the two-dimensional point set data distribution of the profile line.

[0127] Specifically, in step S46, the cubic B-spline center curve obtained in step S45 is segmented according to arc length to obtain the normal plane and the profile line, and obtain the two-dimensional point set data distribution of the profile line; the specific process includes the following steps:

[0128] S461, centerline arc length segmentation, for parameterized curves , in the parameters Small increments nearby , the length of the displacement element of the corresponding curve in three-dimensional space is ,right from arrive Integrate to get the total arc length of the curve ,from arrive The calculation formula is:

[0129] ;

[0130] in, , , The curve is derivatives in direction;

[0131] S462, after the center line is divided into sections, there are midpoints, calculate the normal section plane of each midpoint respectively, and suppose a midpoint is ,in It is the first segment after the center line is segmented. The parameter value corresponding to the midpoint is used to locate the specific position of the point on the curve, and the equation of the normal section plane of the midpoint is:

[0132] ;

[0133] in, is the center line of the B-spline Tangent vector at the point;

[0134] S463, respectively calculate the distance between the data points on the point cloud and the normal section plane, and set a distance threshold max to extract the point set of the original skin seam point cloud Q near the normal plane. ;

[0135] S464, point set Plane projection on the normal section plane obtains the two-dimensional point set data distribution of the section line.

[0136] S5. Analyze and calculate the edge chamfer gap and step difference of the obtained profile point cloud data, such as Figure 5 shown.

[0137] As a preferred implementation of step S5, in step S5, the obtained profile point cloud data is subjected to edge chamfer gap and step difference analysis and calculation, and the specific process includes the following steps:

[0138] S51, using the local PCA method to perform cluster segmentation on the two-dimensional point set of the normal section plane based on the curvature threshold and the Euclidean distance, and segmenting the left and right rounded corners of the skin seam and the straight line area;

[0139] S52, use Gaussian mixture model GMM to fit two straight lines in the skin straight line area of ​​the two-dimensional point set of the normal section plane and , then the distance between the two straight lines That is, the difference in the seam steps, where and is a point on the fitted line, and is the direction vector of the line, is the angle between the two straight line direction vectors, and the parameter and is a scalar, representing the direction of the vector along the line and A linear scale of , used to generate all points on the line;

[0140] S53, respectively use the RANSAC fitting method to perform circular fitting on the left and right edge fillet areas of the skin seam to obtain the left fillet equation and , and then the skin gap can be obtained , where the center coordinates and radius of the left fillet are and , the center coordinates and radius of the right fillet are and .

[0141] In summary, the beneficial effects of the present invention are as follows: the method can quickly complete the accurate detection of the gap and step difference of the aircraft skin seam by using the multi-modal combination of large model segmentation analysis of two-dimensional images and surface structured light processing of point clouds, and solve the problems of low efficiency of laser detection and difficulty in edge chamfer seam detection in traditional methods.

[0142] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0144] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting gap and step difference of aircraft skin based on surface structured light, characterized in that: The following steps are involved: S1. Use binocular structured light method to obtain skin seam image and original skin seam point cloud; S2, Mask-SAM, a fine-tuned large model deep learning network based on light compensation, segments the seam features on the skin seam image; S3, mapping the seam features on the skin seam image to the point cloud data, obtaining the skin seam pixel area of ​​the image and then performing error verification; S4, performing centerline fitting and profile line extraction to obtain point cloud data distribution of the profile line; S5. Analyze and calculate the edge chamfer gap and step difference of the obtained profile line point cloud data.

2. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 1, characterized in that: In step S1, a skin seam image and an original skin seam point cloud are obtained using a binocular structured light method. The specific process includes the following steps: S11, a structured light generator capable of emitting surface structured light and two industrial cameras are installed at a certain angle and position according to their optical axes, and a three-step phase-shifted fringe pattern is projected onto an aircraft skin with a seam using the structured light generator; S12. The left and right cameras are symmetrically distributed according to the binocular vision principle, and the structured light generator is placed at the perpendicular midline of the line connecting the optical centers of the two cameras through a supporting frame; S13, using a checkerboard calibration plate to calibrate the binocular structured light system, using the relationship between the pixel coordinates of all corner points and the world coordinates to calculate the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion parameters, completing the single-target positioning, and finally completing the binocular positioning through the relative position relationship between the left and right cameras; S14, synchronously triggering two calibrated industrial cameras to collect skin seam images, projecting structured light on skin seams of different shapes in different aircraft skin regions and taking photos, to obtain clear and visible skin seam images; S15. The original skin seam point cloud Q is obtained by processing the image data through the fringe projection structured light measurement algorithm and the software system.

3. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 2, characterized in that: In step S15, the original skin seam point cloud Q is obtained by processing the image data through the fringe projection structured light measurement algorithm and the software system; the specific process is: First, epipolar correction, phase matching and phase unfolding are performed on the skin seam fringe images collected by the left and right cameras to obtain the disparity map and the left unfolded phase map. Then, the left camera coordinates are used as the measurement coordinates to construct a binocular stereo structured light model and calculate the three-dimensional point cloud information of the seam features. The model is described as: ； Where: , Respectively represent the depth value of the 3D point in the left and right camera coordinate systems, Represents the depth value of a 3D point in the projector coordinate system. It represents the horizontal coordinate of the projector image plane in the binocular structured light system. Its core function is to solve the three-dimensional coordinates of the object surface through phase encoding. Represents the ordinate of the projector image plane in the binocular structured light system; represents the horizontal coordinate of the left camera image, Indicates the ordinate of the left camera image; represents the horizontal coordinate of the right camera image, Indicates the ordinate of the right camera image; and They are the projection matrices of the left and right cameras after image stereo rectification; is the projection matrix of the projector; for ; ω is the pattern image width; is the number of stripes, and They are the transformation matrices from the corrected left camera coordinates to the projector coordinates and the corrected right camera coordinates. The coordinates of the three-dimensional space point in the left camera coordinate system are solved by solving the eight linear equations provided by the left and right cameras. .

4. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 1, characterized in that: In step S2, Mask-SAM, a fine-tuned large model deep learning network based on light compensation, segments the seam features on the skin seam image. The specific process includes the following steps: S21. The large model uses a fine-tuned Mask-SAM network. Based on the SAM pre-trained model, fine-tuning is performed on the aircraft skin seam features. The image dataset with seam pixel annotation is divided into a training set and a test set to train the large model. S22, during the training phase, the encoder parameters are fixed to a non-trainable state, i.e., the gradient update function is disabled; S23. Before the image is input into the Mask-SAM network, it needs to be grayscaled and adaptively exposed to reduce the influence of the strong light absorption characteristics of the composite skin; S24, inputting the skin seam image of the area to be tested into the preprocessed and trained large model, the network can identify the skin seam in the left and right camera images and perform image segmentation.

5. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 1, characterized in that: Step S3 specifically includes: projecting the 3D point cloud onto the right camera image for error verification, converting the points in the left camera coordinate system to the right camera coordinate system, extracting the actual feature points in the right image, and calculating the reprojection error between the projection of the 3D point on the right camera image and the actually detected feature points. .

6. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 1, characterized in that: In step S4, centerline fitting and profile line extraction are performed to obtain the point cloud data distribution of the profile line. The specific process includes the following steps: S41. In the workpiece coordinate system, a section parallel to the skin surface is established, the edge points on both sides of the point cloud are paired, and the midpoint perpendicular line is taken every 0.5 mm along the seam direction of the skin, and the midpoint perpendicular point of the point cloud on both sides of the seam is calculated as the preliminary center point; S42. A uniform cubic B-spline curve is randomly initialized according to the preliminary center point. The total equation of the B-spline curve is: ； in, are the characteristic points of the control curve, is the K-order B-spline basis function; S43. The distance measurement from the perpendicular point in the slit to the spline curve is realized by using the Frenet frame combined with the square distance minimization method. That is, a coordinate system is constructed with a point on the spline curve to be fitted as the origin and the tangent vector and normal vector at the point as the coordinate axes. The square distance analysis function from the perpendicular point in the slit to the spline curve is defined as: ； in, The vertical point of the joint To the corresponding perpendicular point on the spline curve The distance between is the curvature at the foot point, and The foot points The unit tangent vector and normal vector at ; S44, perform endpoint constraints and cusp constraints on the B-spline center curve, use the minimized objective function to adjust the positions of other control points, and minimize the total objective function. It is defined as the sum of the vertical point errors. The specific formula is as follows: ； S45, repeat step S44 until the objective function value is less than the error threshold Or the number of iterations exceeds a fixed value Finally, the iteratively optimized cubic B-spline center curve and its parameter equation are obtained. ; S46, dividing the cubic B-spline center curve obtained in step S45 into segments according to arc length, and then calculating the normal plane and the profile line, and obtaining the two-dimensional point set data distribution of the profile line.

7. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 6, characterized in that: In step S41, the midpoint of the point cloud on both sides of the seam is calculated as the preliminary center point; the specific process includes the following steps: S411, filter outliers and gap zero points from the original point cloud Q, and smooth the point cloud to obtain the point cloud ; S412, point cloud Cluster segmentation is performed using the regional growing clustering method based on Euclidean distance to divide the skin seam point cloud into two categories: left and right. and ; S413, calculate point clouds separately and Get their edge points respectively , , calculate point cloud and The curvature changes of all adjacent points in the , and the points with the largest curvature changes on both sides are recorded as the edge point sets on both sides and ; S414: Estimation of the local main direction is performed, and the local main direction of each point is calculated using PCA analysis to ensure the connection direction of the paired point pairs The direction is consistent with the extension direction of the seam. If the direction deviation exceeds the angle difference threshold of 15°, the pairing will be eliminated; S415, perform midpoint calculation, and calculate the effective matching points. , calculate the vertical point , as the preliminary centerline candidate point; S416, DBSCAN clustering is used to remove outliers to reduce the interference of bifurcation or noise. The key definition is as follows: ； in, is a dataset, is the Euclidean distance, As the core point, Indicate point of Neighborhood, that is, all points in the data set The distance does not exceed the threshold The set of points, It is the basic unit for defining core points and extended clustering in DBSCAN. The neighborhood of contains at least minPts points, including itself. minPts is the threshold for defining the core point density in DBSCAN, which directly affects the number of clusters, noise filtering, and algorithm robustness.

8. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 6, characterized in that: In step S46, the cubic B-spline center curve obtained in step S45 is segmented according to arc length to obtain the normal plane and the profile line, and obtain the two-dimensional point set data distribution of the profile line; the specific process includes the following steps: S461, centerline arc length segmentation, for parameterized curves , in the parameters Small increments nearby , the length of the displacement element of the corresponding curve in three-dimensional space is ,right from arrive Integrate to get the total arc length of the curve ,from arrive The calculation formula is: ； in, , , The curve is derivatives in direction; S462, after the center line is divided into sections, there are midpoints, calculate the normal section plane of each midpoint respectively, and suppose a midpoint is ,in It is the first segment after the center line is segmented. The parameter value corresponding to the midpoint is used to locate the specific position of the point on the curve, and the equation of the normal section plane of the midpoint is: ； in, is the center line of the B-spline Tangent vector at the point; S463, respectively calculate the distance between the data points on the point cloud and the normal section plane, and set a distance threshold max to extract the point set of the original skin seam point cloud Q near the normal plane. ; S464, point set Plane projection on the normal section plane obtains the two-dimensional point set data distribution of the section line.

9. The method for detecting gap and step difference of aircraft skin based on surface structured light according to claim 1, characterized in that: In step S5, the obtained profile point cloud data is subjected to edge chamfer gap and step difference analysis and calculation, and the specific process includes the following steps: S51, using a local PCA method to perform cluster segmentation on the two-dimensional point set of the normal section plane based on a curvature threshold and Euclidean distance, and segmenting the left and right rounded corners of the skin seam and the straight line area; S52, use Gaussian mixture model GMM to fit two straight lines in the skin straight line area of ​​the two-dimensional point set of the normal section plane and , then the distance between the two straight lines That is, the difference in the seam steps, where and is a point on the fitted line, and is the direction vector of the line, is the angle between the two straight line direction vectors, and the parameter and is a scalar, representing the direction of the vector along the line and A linear scale of , used to generate all points on the line; S53, respectively use the RANSAC fitting method to perform circular fitting on the left and right edge fillet areas of the skin seam to obtain the left fillet equation and , and then the skin gap can be obtained , where the center coordinates and radius of the left fillet are and , the center coordinates and radius of the right fillet are and .

Images