Aviation fastener concave-convex amount detection method based on weak supervision optimization

Abstract

The invention relates to the technical field of aviation fastener detection, solves the technical problems that in a traditional method, the segmentation difficulty of a skin and a fastener is large, the detection precision and efficiency are low, and manual marking is difficult, and particularly relates to a weak supervision optimization-based aviation fastener concave-convex amount detection method. Comprising the following steps: acquiring an original image and three-dimensional point cloud data of an aircraft skin fastener; performing segmentation by adopting an SAM network model to obtain a two-dimensional mask image containing fastener features; mapping the pixel region of the fastener feature into the three-dimensional point cloud data; using a NumPy vectorization method to calculate the average deviation of the concave-convex quantity; and carrying out visual output on the average deviation of the concave-convex quantity in a label form. According to the method, rapid and accurate measurement of the concave-convex quantity of the nail head of the aviation fastener can be realized, visual output of the concave-convex quantity detection result is realized by utilizing the two-dimensional image mask and the three-dimensional point cloud mapping, the efficiency and the precision of aviation fastener detection are greatly improved, and a visual basis is provided for assembly quality evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of aviation fastener detection, and particularly to a method for detecting the concave-convex amount of aviation fasteners based on weak supervision optimization. Background Art

[0002] In the field of aviation manufacturing, aircraft fasteners, as key components for aircraft structure connection and aerodynamic shape, their assembly quality will directly affect the performance of aircraft such as safety, stability, flight, and stealth. Aviation fasteners have characteristics such as a large number of shallow features, a wide distribution area, and high precision requirements. For example, a civil airliner has approximately 2 million rivet fasteners, and the concave-convex amount accuracy of fighter jet rivets needs to be lower than 0.1 mm. Traditional detection methods mainly rely on manual visual inspection or auxiliary measurement with a micrometer pad, which have problems such as low efficiency, poor accuracy, and being easily interfered by subjective factors. Especially in the area of complex curved skins, it is difficult to achieve accurate quantitative evaluation of the concave-convex amount by manual detection, and surface damage may be caused by contact measurement.

[0003] In recent years, digital detection technologies based on machine vision have been gradually applied to this field, but there are still the following defects: Detection methods based on lasers or single-line structured light tend to highlight feature targets, resulting in too high a missed detection rate in the multi-rivet area of skins with densely arranged fasteners; Traditional deep learning methods require a large amount of labeled data for support, but the accurate labeling cost of aviation fasteners is high, and it is difficult to obtain data under special working conditions; The algorithm has weak environmental adaptability. When detecting strongly reflective fasteners, the missing rate of point clouds is too high, affecting the integrity of detection.

[0004] Scholars at home and abroad have tried to optimize the efficiency by adopting lightweight networks, such as improving the YOLO series algorithms. However, the problem of fast and accurate detection of aviation fasteners in the weak supervision scenario has not been effectively solved. The detection system based on reinforcement learning still needs to rely on tens of thousands of labeled samples, and it is difficult to adapt to the characteristics of small-batch and multi-variety production in aviation manufacturing. In summary, developing a method for detecting the concave-convex amount of aviation fasteners based on weak supervision, breaking through technical bottlenecks such as instance loss, extraction of a large number of shallow features, and noise interference, and realizing non-contact detection with high precision, high efficiency, and low missed detection rate has become an urgent need to improve the quality control of aviation equipment manufacturing. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a method for detecting the concave-convex amount of aviation fasteners based on weak supervision optimization, which solves the technical problems of large difficulty in skin-fastener segmentation, low detection accuracy and efficiency, and difficult manual labeling in traditional methods.

[0006] To solve the above technical problems, the present invention uses a method of fusing binocular vision and structured light multi-sensors to obtain the original image and three-dimensional point cloud data, and adopts a multi-modal segmentation network of the SAM large model optimized based on weak supervision to achieve rapid and accurate measurement of the concave-convex amount of the head of aviation fasteners, and uses two-dimensional image masks and three-dimensional point cloud mapping to realize the visual output of the concave-convex amount detection results. The technical solution is as follows: A method for detecting the concave-convex amount of aviation fasteners based on weak supervision optimization, the method comprising the following steps:

[0007] S1. Collect the original image of the fasteners on the aircraft skin through a binocular structured light system, and reconstruct the three-dimensional point cloud data of the area of the fasteners on the aircraft skin based on the original image;

[0008] S2. Based on the image dataset containing the image data and three-dimensional point cloud data of the area of the fasteners on the aircraft skin, use the SAM network model for segmentation to obtain a two-dimensional mask image containing the fastener features;

[0009] S3. Map the pixel area of the fastener features in the two-dimensional mask image to the three-dimensional point cloud data, and screen out the point cloud of the skin features belonging to the fasteners from the three-dimensional point cloud data;

[0010] S4. Determine the optimal skin fitting plane equation with the most nail head point cloud and the smallest fitting error in the point cloud of the skin feature area as the optimal skin background plane equation, and use the NumPy vectorization method to calculate the deviation of several nail head point clouds to the skin fitting plane to obtain the average concave-convex deviation of the fasteners;

[0011] S5. Visually output the average concave-convex deviation of the fasteners on the two-dimensional mask image in the form of a label.

[0012] The present invention proposes a complementary fusion strategy of the large model network for the skin and fasteners, which can achieve accurate segmentation of the skin fasteners under weak supervision, and greatly improve the efficiency and accuracy of aviation fastener detection.

[0013] Further, in step S1, the specific process includes the following steps:

[0014] S11. Install the optical axes of two industrial cameras at a preset angle so that the two optical axis beams are simultaneously focused on the surface of the skin to be measured, that is, the area of the fasteners on the aircraft skin, and then calibrate the binocular structured light system using the checkerboard calibration method;

[0015] S12. Project structured light onto different types of fasteners in different areas of the aircraft skin fasteners and take pictures to obtain clear and visible original images, and use the multi-frequency heterodyne method to solve the absolute phase value of the original image and the depth image of the area of the fasteners on the aircraft skin;

[0016] S13. Reconstruct three-dimensional point cloud data of the aircraft skin fastener area based on the obtained depth image.

[0017] Furthermore, in step S13, specifically:

[0018] Phase matching is performed using the absolute phase values ​​of the left and right camera images. The coordinates (x, y, z) of the 3D point cloud data are calculated using the triangulation formula in combination with the camera calibration parameters and phase matching results. The calculation formula is:

[0019]

[0020] Where B is the baseline distance; f is the focal length; d is the disparity; (u0, v0) is the center coordinate of the depth image; (u1, v1) is the pixel coordinate of the target space point in the reference camera image.

[0021] Furthermore, in step S2, the specific process includes the following steps:

[0022] S21, collecting and preprocessing the image data of the aircraft skin fastener area, and combining the three-dimensional point cloud data to form an image data set that resists the high reflective characteristics of the skin and fasteners;

[0023] S22, taking any image in the image data set as the input of the SAM network model based on weakly supervised optimization, calling the 3D superpoint to initialize the candidate points, and initializing the points in the bounding box that may belong to the feature to be tested as candidate points;

[0024] S23, using a greedy view selection algorithm to select a set of image perspectives to achieve observation of all candidate points, and mapping the three-dimensional point cloud data to the two-dimensional image plane to obtain the projected two-dimensional coordinates of each point, the expression is:

[0025]

[0026] Where K and P represent the intrinsic and extrinsic matrix of each industrial camera, respectively; (u, v) represents the unnormalized pixel coordinates of the 3D point on the image plane; d represents the depth value of the 3D point in the camera coordinate system; X represents the 3D coordinate of the 3D point in the world coordinate system (X W ,Y W ,Z W ),(u d ,v d ), represents the normalized two-dimensional pixel coordinates;

[0027] S24. Implement the optimal instance segmentation of the two-dimensional fastener image by adopting a complementary fusion optimization strategy. Calculate the two-dimensional bounding box of the projected pixels as the foreground hint for segmenting the target instance within the bounding box. Meanwhile, sample the pixels around the projection area as the background hint for filtering out the redundant parts irrelevant to the fastener instance in the image plane;

[0028] S25. Fuse the foreground hint and multiple background hints to predict the two-dimensional mask image containing fastener features The fusion formula is:

[0029]

[0030] In the formula, is the foreground hint mask; is the background hint mask; β is the parameter controlling the background suppression intensity;

[0031] S26. Perform superpoint confidence correction on the two-dimensional mask image to obtain accurate point instances and their labels. If the superpoint s is covered by multiple instances k, assign it to the instance k with the highest confidence * .

[0032] Furthermore, in step S26, the superpoint confidence correction formula is as follows:

[0033]

[0034]

[0035] In the formula, C p,k is the confidence of a single candidate point; Φ(p, V k,m ) is the visibility of point p in the m-th view V k,m of instance k; is the SAM heatmap value of point p projected to the pixel coordinates (i, j) of view V k,m ; ∑ m Φ(p, V k,m ) represents the normalized weight of point p, which can effectively avoid the problem of too few projection views; C s,k is the average confidence of each superpoint, and |s| is the number of points included in superpoint s.

[0036] Furthermore, in step S3, the specific process includes the following steps:

[0037] S31. Align the depth image generated by the binocular structured light system with the two-dimensional mask image, and establish the mapping relationship from the image coordinate system to the three-dimensional world coordinate system by using the camera calibration parameters. The specific formula is as follows:

[0038]

[0039] Wherein, (u, v) are the coordinates of the image pixels; (X, Y, Z) are the coordinates of the three-dimensional point cloud; f x and f y and c x and c y are the calibrated internal parameters of the camera;

[0040] S32. Traverse the pixels belonging to the fastener area in the two-dimensional mask image, extract the corresponding depth value Z, and inversely calculate the three-dimensional point cloud coordinates (X, Y, Z) according to the above formula, and screen out the point cloud of the skin feature area belonging to the fastener from the three-dimensional point cloud data.

[0041] Further, in step S4, the specific process includes the following steps:

[0042] S41. Use the RANSAC random three-point method to generate a candidate skin fitting plane equation. By calculating the Euclidean distance from all points in the skin feature area point cloud to the candidate skin fitting plane equation, screen out the inliers smaller than the threshold to form an inlier set. After iterating a preset number N of times, select the skin fitting plane with the most inliers and the smallest fitting error as the optimal skin background plane equation;

[0043] S42. Use the least squares method to recalculate the plane parameters of the optimal skin background plane equation for the selected inlier set;

[0044] S43. Convert the nail head point cloud coordinates into the form of a homogeneous coordinate matrix, and use the broadcasting mechanism and matrix operations of NumPy to quickly calculate the vertical distance d from each nail head point cloud to the skin fitting plane through a vectorized formula to form a distance matrix;

[0045] S44. Distinguish the convex and concave areas according to the positive and negative values of the distance matrix, and calculate the deviation extreme value and the average deviation of the concave and convex amount of the fastener to be measured.

[0046] Further, in step S43, the vectorized formula for calculating the vertical distance d is:

[0047]

[0048] The distance matrix L D is expressed as:

[0049]

[0050] Wherein, A, B, and C are the normal vector parameters of the optimal skin background plane equation, D is a constant term related to the plane spatial position; (x i , y i , z i ) are the nail head point cloud coordinates; d iis the distance from a single point in the point cloud to the plane; L D is the distance matrix containing the vertical distances from all the nail head point clouds to the plane.

[0051] By means of the above technical solution, the present invention provides a method for detecting the concavity and convexity of aviation fasteners based on weakly supervised optimization, which at least has the following beneficial effects:

[0052] 1. The present invention combines the weakly supervised optimized SAM large model network with binocular structured light to analyze the point cloud, which can quickly and accurately detect the concavity and convexity of aviation fasteners, and can achieve precise segmentation of the skin fasteners under weak supervision, greatly improving the efficiency and accuracy of aviation fastener detection.

[0053] 2. The present invention can quickly and accurately measure the concavity and convexity of the nail head of aviation fasteners, and uses two-dimensional image masks and three-dimensional point cloud mapping to realize the visual output of the concavity and convexity detection results, thereby providing a visual basis for assembly quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0055] Figure 1 is the flow chart of the method for detecting the concavity and convexity of aviation fasteners in the present invention;

[0056] Figure 2 is the structural schematic diagram of the binocular structured light system in the present invention;

[0057] Figure 3 is the network structure diagram of the aviation fastener concavity and convexity detection system in the present invention;

[0058] Figure 4 is the schematic diagram of the measurement result of the concavity and convexity of aviation fasteners in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments. Thereby, a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects can be obtained and implemented accordingly.

[0060] In order to solve the technical defects existing in the traditional methods, this embodiment adopts a method of fusing binocular vision and structured light multi-sensors to obtain the original images and three-dimensional point cloud data, and uses a multi-modal segmentation network of the SAM large model optimized based on weak supervision to achieve rapid and accurate measurement of the concave-convex amount of the aviation fastener head, and uses two-dimensional image masks and three-dimensional point cloud mapping to realize the visual output of the concave-convex amount detection results. Please refer to Figures 1-4 , this embodiment proposes a method for detecting the concave-convex amount of aviation fasteners based on weak supervision optimization. Aiming at the problems of difficult skin-fastener segmentation, low detection accuracy and efficiency, and difficult manual annotation, a complementary fusion strategy of large model networks is provided. This method can quickly complete the accurate detection of the concave-convex amount of aviation fasteners by combining the weakly supervised optimized SAM large model network with binocular structured light analysis of point clouds. As Figure 1 shown, this method includes the following steps:

[0061] S1. Use a calibrated binocular structured light system to obtain the original images and three-dimensional point cloud data. Control the structured light projector to project the stripe image onto the aircraft skin with fasteners, and synchronously trigger two calibrated industrial cameras to collect the original images of the aircraft skin fasteners. Use software algorithms to process the original images to obtain three-dimensional point cloud data. The structural composition of the binocular structured light system is as Figure 2 shown, and the specific steps include:

[0062] S11. Install the optical axes of the two industrial cameras at a preset angle so that the two optical axis beams are simultaneously focused on the surface of the skin to be measured, that is, the aircraft skin fastener area. In this example, the preset angle for installing the two industrial cameras is set to 55°, and the checkerboard calibration method is used to calibrate the binocular structured light system.

[0063] S12. Project structured light onto fasteners of different models in different aircraft skin fastener areas and take pictures to obtain clear and visible original images, that is, aircraft skin fastener images. This embodiment uses "binocular vision + surface structured light projection" to obtain the original images and three-dimensional point cloud data. The projection pattern is encoded with "4-step sine phase-shifted stripes + 4-order Gray code", and the multi-frequency heterodyne method is used to solve the absolute phase value and the depth image of the aircraft skin fastener area.

[0064] S13. Reconstruct the three-dimensional point cloud data of the aircraft skin fastener area according to the obtained depth image. Use the absolute phase values of the left and right camera images for phase matching. Combine the camera calibration parameters and the phase matching results, and calculate the coordinates (x, y, z) of the three-dimensional point cloud data through the triangulation formula. The formula derivation is as follows:

[0065]

[0066] Wherein, B is the baseline distance; f is the focal length; d is the parallax; (u0, v0) are the central coordinates of the depth image; (u1, v1) are the pixel coordinates of the target spatial point in the reference camera image.

[0067] S2. Based on the image dataset including the image data of the aircraft skin fastener area and the three-dimensional point cloud data, use the SAM network model for segmentation to obtain a two-dimensional mask image containing fastener features. In this embodiment, the network structure of the aviation fastener concavity and convexity detection system based on the SAM network model is as Figure 3 shown, and the specific steps include:

[0068] S21. Collect sufficient image data of the aircraft skin fastener area, and perform preprocessing operations such as Gaussian denoising, Gamma transform enhancement, and normalization, and combine the three-dimensional point cloud data to form an image dataset that resists the high reflectivity characteristics of the skin and fasteners.

[0069] S22. Take any image in the image dataset as the input of the SAM network model based on weakly supervised optimization. The SAM network model can significantly reduce the dependence on fastener annotation data, and at the same time maintain high-precision segmentation under working conditions such as noise interference and high reflectivity characteristics.

[0070] S23. Call 3D superpoints for candidate point initialization. This method can effectively filter out other points outside the feature to be measured, so as to initialize the points that may belong to the feature to be measured within the bounding box as candidate points. Superpoints are small clusters of three-dimensional point clouds representing local geometric continuity formed by graph clustering technology based on normals. If any point in the superpoint is outside the bounding box, this embodiment will filter out the entire superpoint.

[0071] S24. Use the greedy view selection algorithm to efficiently select a set of image views to observe all candidate points, providing complete coverage data for subsequent two-dimensional image segmentation. The specific algorithm process is as follows:

[0072]

[0073] Among them, the input of the algorithm includes the number of instances M, the preprocessed image dataset V, and the candidate point set P (|P| = M), and the output includes the selected instance view set V`. Wherein, V i 、V j are the i-th and j-th images in the image dataset V respectively, P i is the i-th candidate point in the candidate point set, and S j represents the subset of candidate points that the view V j can observe; where S is the set of candidate points covered by all views, used for greedy selection of the optimal view, that is, select the view that covers the most remaining candidate points.

[0074] S25. Map the 3D point cloud data onto the 2D image plane using the calibrated camera view information. According to the pinhole camera model, the projected 2D coordinates of each point are calculated by the following formula:

[0075]

[0076] where K and P represent the internal and external parameter matrices of each industrial camera respectively, P is the input point position vector in the 3D world coordinates, and is the projected 2D pixel coordinates; (u, v) represents the unnormalized pixel coordinates of the 3D point on the image plane; d represents the depth value of the 3D point in the camera coordinate system; X represents the 3D coordinates (X W , Y W , Z W ), (u d , v d ) represents the normalized 2D pixel coordinates.

[0077] S26. Adopt a complementary fusion optimization strategy to achieve the best instance segmentation of the 2D fastener image. Calculate the 2D bounding box of the projected pixels as the foreground cue, and sample the pixels around the projected area as the background cue at the same time. The foreground cue can segment the target instance within the bounding box, while the background cue helps to filter out the redundant parts in the image plane that are not relevant to the fastener instance.

[0078] S27. Fuse the foreground cue (box) with multiple background cues (points) to predict the 2D mask image containing fastener features Generate the corresponding 2D mask heatmap by separately calling SAM for each cue. At the same time, the background predictions are merged into a single background mask. By subtracting the highest response in the background area, the false detection parts in the foreground mask can be effectively suppressed. The fusion formula is as follows:

[0079]

[0080] where is the foreground cue mask; is the background cue mask; β is a parameter controlling the background suppression intensity. In this embodiment, the optimal value of the parameter β obtained through experiments is 0.5.

[0081] S28. Perform 3D confidence integration and label correction on the 2D mask image . If the superpoint s is covered by multiple instances k, assign it to the instance k with the highest confidence *By calculating the average confidence of each superpoint, low-confidence superpoints (average confidence less than 0) are filtered out. At the same time, a superpoint voting mechanism is established to handle the special case where a superpoint is covered by multiple instances. In this way, 3D point clouds and 2D images can be effectively combined to obtain accurate point instances and their labels. The superpoint confidence correction formula is as follows:

[0082]

[0083] where C p,k is the confidence of a single candidate point; Φ(p, V k,m ) is the visibility of point p in the m-th view V k,m of instance k (1 means visible, 0 means invisible); is the SAM heatmap value at the pixel coordinates (i, j) where point p is projected onto view V k,m ; the denominator ∑ m Φ(p, V k,m ) represents the normalized weight of point p, which can effectively avoid the problem of too few projection views; C s,k is the average confidence of each superpoint, and |s| is the number of points included in superpoint s.

[0084] S3. Map the pixel region of the fastener feature in the 2D mask image to the 3D point cloud data, and filter out the skin feature region point cloud belonging to the fastener from the 3D point cloud data. The specific steps are as follows:

[0085] S31. Strictly align the depth image generated by the binocular structured light system with the 2D segmentation result (i.e., the 2D mask image) to ensure that the RGB value and depth value of each pixel correspond one by one. Use the camera calibration parameters (intrinsic matrix, extrinsic matrix) to establish the mapping relationship from the image coordinate system to the 3D world coordinate system. The specific formula is as follows:

[0086]

[0087] In the formula, (u, v) are the image pixel coordinates; (X, Y, Z) are the 3D point cloud coordinates; f x , f y , c x , c y are the calibrated camera intrinsics.

[0088] S32. Traverse the pixels in the 2D mask image belonging to the fastener region, extract their corresponding depth values Z, and reverse calculate the 3D point cloud coordinates (X, Y, Z) according to the above formula. Filter out the skin feature region point cloud belonging to the fastener from the 3D point cloud data.

[0089] S4. Perform fastener head extraction and concavity / convexity measurement. In this embodiment, a local area RANSAC fitting algorithm is used to obtain the optimal skin background plane equation, and the NumPy vectorization method is used to quickly calculate the deviation of the fastener head point cloud of the fastener to the skin fitting plane. The calculation results of the fastener concavity / convexity are as Figure 4 shown. The specific steps are as follows:

[0090] S41. Use the RANSAC random three-point method to generate a candidate skin fitting plane equation. By calculating the Euclidean distance from all points in the skin feature area point cloud to the candidate skin fitting plane equation, the inliers smaller than the threshold are filtered to form an inlier set. After N iterations of the preset number, the skin fitting plane with the most inliers and the smallest fitting error is selected as the optimal skin background plane equation Ax + By + Cz + D = 0. The threshold is set to t = 2σ = 0.1 mm based on the point cloud noise and sub-millimeter accuracy requirements, where σ is the noise standard deviation of the point cloud. The preset number N = 200 is adjusted according to the actual application.

[0091] S42. Use the least squares method to recalculate the plane parameters of the optimal skin background plane equation for the selected inlier set, thereby improving the plane fitting accuracy. Among them, A, B, and C are the normal vector parameters of the optimal skin background plane equation, and D is the constant term related to the plane spatial position.

[0092] S43. Convert the fastener head point cloud coordinates into the form of a homogeneous coordinate matrix. Using the broadcasting mechanism and matrix operations of NumPy, the perpendicular distance d from each fastener head point cloud to the skin fitting plane is quickly calculated through the vectorized formula i to form a distance matrix L D . In this example, this calculation method is about 50 times faster than loop traversal. The vectorized distance calculation formula is as follows:

[0093]

[0094] The distance matrix L D is expressed as:

[0095]

[0096] In the formula, A, B, and C are the normal vector parameters of the optimal skin background plane equation, and D is the constant term related to the plane spatial position; (x i , y i , z i ) are the fastener head point cloud coordinates; d i is the distance from a single point in the point cloud to the plane; L D is the distance matrix containing the perpendicular distances from all fastener head point clouds to the plane.

[0097] S44. Distinguish the convex (positive value) and concave (negative value) regions according to the positive and negative values of the distance matrix, use the software system to calculate the extreme deviation value and the average deviation of the convex and concave amounts of the fastener to be measured, and output the measurement results in the form of a report.

[0098] S5. Visualize and output the average deviation of the convex and concave amounts of the fastener in the form of a label on the two-dimensional mask image, providing a visual basis for the assembly quality assessment. This embodiment can achieve the rapid and accurate measurement of the convex and concave amounts of the head of the aviation fastener, and use the two-dimensional image mask and three-dimensional point cloud mapping to realize the visual output of the detection results of the convex and concave amounts, thereby providing a visual basis for the assembly quality assessment.

[0099] Those of ordinary skill in the art can understand that all or part of the steps in implementing the above method embodiments can be completed by instructing relevant hardware through a program. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The above embodiments have introduced the present invention in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting the concavity and convexity of aviation fasteners based on weak supervision optimization, characterized in that The method includes the following steps: S1. Collect the original images of the aircraft skin fasteners through a binocular structured light system, and reconstruct the three-dimensional point cloud data of the aircraft skin fastener area based on the original images; S2. Based on the image dataset including the image data and three-dimensional point cloud data of the aircraft skin fastener area, use the SAM network model for segmentation to obtain a two-dimensional mask image containing fastener features; S3. Map the pixel area of the fastener features in the two-dimensional mask image to the three-dimensional point cloud data, and filter out the skin feature area point cloud belonging to the fastener from the three-dimensional point cloud data; S4. Determine the skin fitting plane with the most nail head point cloud and the smallest fitting error in the skin feature area point cloud as the optimal skin background plane equation, and use the NumPy vectorization method to calculate the deviation of several nail head point clouds to the skin fitting plane to obtain the average deviation of the concavity and convexity of the fastener; S5. Visually output the average deviation of the concavity and convexity of the fastener on the two-dimensional mask image in the form of a label.

2. The method for detecting the concavo-convex amount of the aviation fastener according to claim 1, wherein In step S1, the specific process includes the following steps: S11. Install the optical axes of two industrial cameras at a preset angle so that the two optical axis beams are simultaneously focused on the surface of the skin to be measured, that is, the aircraft skin fastener area, and then calibrate the binocular structured light system using the checkerboard calibration method; S12. Project structured light onto fasteners of different models in different aircraft skin fastener areas and take pictures to obtain clear and visible original images, and use the multi-frequency heterodyne method to solve the absolute phase value of the original images and the depth image of the aircraft skin fastener area; S13. Reconstruct the three-dimensional point cloud data of the aircraft skin fastener area according to the obtained depth image.

3. The method for detecting the concavo-convex amount of the aviation fastener according to claim 2, wherein In step S13, specifically: Use the absolute phase values of the left and right camera images for phase matching. Combine the camera calibration parameters and the phase matching results, and calculate the coordinates (x, y, z) of the three-dimensional point cloud data through the triangulation formula. The calculation formula is: In the formula, B is the baseline distance; f is the focal length; d is the parallax; (u0, v0) is the central coordinate of the depth image; (u1, v1) is the pixel coordinate of the target space point in the reference camera image.

4. The method for detecting the concave-convex amount of an aviation fastener according to claim 1, wherein In step S2, the specific process includes the following steps: S21. Collect the image data of the aircraft skin fastener area and perform preprocessing, and combine the three-dimensional point cloud data to form an image dataset that resists the high reflective characteristics of the skin and fasteners; S22. Take any image in the image dataset as the input of the SAM network model based on weakly supervised optimization, and call 3D superpoints for candidate point initialization, and initialize the points that may belong to the feature to be measured within the bounding box as candidate points; S23. Use the greedy view selection algorithm to select a set of image perspectives to observe all candidate points, and map the three-dimensional point cloud data to the two-dimensional image plane to obtain the projected two-dimensional coordinates of each point. The expression is: where K and P represent the internal and external parameter matrices of each industrial camera respectively; (u, v) represents the unnormalized pixel coordinates of a 3D point on the image plane; d represents the depth value of the 3D point in the camera coordinate system; X represents the 3D coordinates (X W , Y W , Z W ), (u d , v d ) represent the normalized two-dimensional pixel coordinates; S24. Adopt a complementary fusion optimization strategy to achieve optimal 2D image instance segmentation of fasteners. Calculate the 2D bounding box of the projection pixels as the foreground hint for segmenting the target instance within the bounding box, and simultaneously sample the pixels around the projection area as the background hint for filtering out the redundant parts irrelevant to the fastener instance in the image plane. S25. Fuse the foreground prompt with multiple background prompts to predict a two-dimensional mask image containing fastener features The fusion formula is as follows: In the formula, is the foreground hint mask; is the background hint mask; β is a parameter for controlling the background suppression intensity; S26. For the two-dimensional mask image perform superpoint confidence correction to obtain accurate point instances and their labels. If the superpoint s is covered by multiple instances k, assign it to the instance k with the highest confidence * .

5. The method for detecting the concave-convex amount of an aviation fastener according to claim 4, wherein In step S26, the superpoint confidence correction formula is as follows: Where C p,k is the confidence of a single candidate point; Φ(p, V k,m ) is the visibility of point p in the m-th view V k,m of instance k; is the SAM heatmap value at the pixel coordinates (i, j) where point p is projected onto view V k,m ; ∑ m Φ(p, V k,m ) represents the normalized weight of point p, which can effectively avoid the problem of too few projection views; C s,k is the average confidence of each superpoint; |s| is the number of points contained in superpoint s.

6. The method for detecting the concave-convex amount of an aviation fastener according to claim 2, wherein, In step S3, the specific process includes the following steps: S31. Align the depth image generated by the binocular structured light system with the 2D mask image, and establish the mapping relationship from the image coordinate system to the 3D world coordinate system using the camera calibration parameters. The specific formula is as follows: where (u, v) are the coordinates of an image pixel; (X, Y, Z) are the coordinates of a 3D point cloud; f x and f y and c x and c y are the calibrated internal parameters of the camera; S32. Traverse the pixels in the 2D mask image that belong to the fastener area, extract their corresponding depth values Z, and inversely calculate the 3D point cloud coordinates (X, Y, Z) according to the above formula. Select the point cloud of the skin feature area belonging to the fastener from the 3D point cloud data.

7. The method for detecting the concave-convex amount of an aviation fastener according to claim 1, wherein In step S4, the specific process includes the following steps: S41. Use the RANSAC random three-point method to generate a candidate skin fitting plane equation. Calculate the Euclidean distance from all points in the skin feature area point cloud to the candidate skin fitting plane equation, and filter out the inliers smaller than the threshold to form an inlier set. After iterating a preset number N of times, select the skin fitting plane with the most inliers and the smallest fitting error as the optimal skin background plane equation. S42. Use the least squares method to recalculate the plane parameters of the optimal skin background plane equation for the selected inlier set. S43. Convert the nail head point cloud coordinates into the form of a homogeneous coordinate matrix. Using the broadcasting mechanism and matrix operations of NumPy, quickly calculate the vertical distance d from each nail head point cloud to the skin fitting plane through a vectorized formula to form a distance matrix L D ; S44. Distinguish the convex and concave areas according to the positive and negative values of the distance matrix, and calculate the deviation extreme value and average deviation of the convexity and concavity of the fastener to be measured.

8. The method for detecting the concave-convex amount of an aviation fastener according to claim 7, characterized in that, In step S43, the calculation formula for vectorizing the vertical distance d is: Distance matrix L D The expression is as follows: where A, B, and C are the normal vector parameters of the optimal skin background plane equation, and D is a constant term related to the plane's spatial position; (x i , y i , z i ) are the nail head point cloud coordinates; d i is the distance from a single point in the point cloud to the plane; L D is the distance matrix containing the perpendicular distances from all nail head point clouds to the plane.

Images