Rivet convex-concave quantity feature accurate extraction method based on point cloud dimension reduction

Abstract

A rivet convex-concave quantity feature accurate extraction method based on point cloud dimensionality reduction comprises the following steps that S1, point cloud curved surface variation is calculated, and color mapping is carried out on curved surface normal variation after mathematical processing; s2, performing point cloud dimension reduction by adopting a method of combining a principal component analysis (PCA) projection technology and two-dimensional gridding, and generating an image; s3, carrying out rivet region point cloud coarse segmentation by adopting a related image processing algorithm and an image point cloud mapping relation; and S4, accurately extracting the rivet head point cloud and the skin point cloud based on layered structure fitting, and calculating the convex-concave quantity of the rivet. According to the method, the convex-concave quantity characteristics of the rivet can be effectively, accurately and efficiently extracted from the three-dimensional point cloud.

Description

Technical Field

[0001] The present invention relates to a method for accurately extracting convex and concave features of rivets based on point cloud dimensionality reduction, and particularly proposes a point cloud gridding dimensionality reduction and a rivet area point cloud coarse segmentation method based on point cloud and image mapping, and a rivet extraction method based on hierarchical structure fitting to achieve efficient and accurate feature extraction, belonging to the field of computer vision. Background Art

[0002] In the field of aircraft manufacturing, riveting quality is an important factor affecting the overall aerodynamic performance and long fatigue life of the aircraft, so the detection of riveting quality is essential. The convexity and concavity of rivets is an important indicator in riveting quality detection. The traditional method of rivet convexity and concavity detection mainly relies on visual or touch inspection, lacks quantitative data, has low accuracy, low efficiency and poor reliability. In view of the continuous improvement of the level of automation and digitalization in the production process of modern aircraft, it is urgent to study the automation and digitalization methods of riveting quality detection. Summary of the invention

[0003] The purpose of the present invention is to solve the problems of low efficiency and low precision in extracting rivet features and calculating rivet convexity and concavity in three-dimensional point cloud data, and propose a method for accurately extracting rivet convexity and concavity features based on point cloud dimensionality reduction. The rivet area is roughly segmented by processing the image generated by point cloud dimensionality reduction and two-dimensional grid division, thereby improving the detection efficiency; the rivet head point cloud and the skin point cloud are accurately segmented by hierarchical fitting, thereby improving the calculation accuracy of the rivet convexity and concavity.

[0004] The technical solution of the present invention is:

[0005] A method for accurately extracting convex and concave features of rivets based on point cloud dimensionality reduction. The specific steps are as follows:

[0006] S1: Calculate the surface variation of the point cloud, and perform color mapping on the surface normal variation after mathematical processing;

[0007] S2: The principal component analysis (PCA) projection technique is combined with two-dimensional gridding to reduce the dimension of the point cloud and generate an image;

[0008] S3: Use relevant image processing algorithms and image point cloud mapping relationships to perform rough segmentation of the rivet area point cloud;

[0009] S4: Based on hierarchical structure fitting, the rivet head point cloud and skin point cloud are accurately extracted, and the rivet convexity and concavity are calculated.

[0010] The calculation of the point cloud surface variation and color mapping of the surface normal variation after mathematical processing include the following steps:

[0011] Step 1: Point cloud surface variation estimation. Surface variation is an indicator that describes the local geometric features of the surface. It is similar to surface curvature. Compared with curvature estimation based on function fitting, it is more suitable for simplifying point sampling surfaces and is widely used in areas such as point cloud data simplification. For any point Pi (xi, yi, zi) in a discrete point cloud P, the set of points in its r neighborhood is:

[0012] N r (P i )={P l ,P2,P3,…P i ,…P m} (1)

[0013] Where: P i is a point in the point cloud; N r (P i ) is P i is the center; r is the point set in the spherical area with radius; m is the number of points in the neighborhood of r.

[0014] Calculate the center point of the neighborhood point set:

[0015]

[0016] Where: P i is a point in the point cloud; is the center point of the domain point set; m is the number of points in the neighborhood of r.

[0017] Construct the point cloud covariance matrix and solve it:

[0018]

[0019] Where: C is a symmetric semi-positive definite matrix; λ = {λ0,λ1,λ2} is the eigenvalue of C, and λ0≤λ1≤λ2; is j The corresponding feature vector.

[0020] Define point P i The surface variation in the neighborhood of r is:

[0021]

[0022] Where: σ(P i ) represents the degree of flatness of the point in the local area, and its value range is (0~1 / 3). The smaller the value, the flatter the local area. When the value is 0, it represents a plane. λ0,λ1,λ2 are the eigenvalues ​​of C in (3).

[0023] Step 2: Calculate the logarithm of the rivet surface variation. Since the curvature of the aircraft skin is very small and the riveted area is also very small, the surface variation of the riveted area is very small and not very distinct, so the logarithm of the surface variation is calculated:

[0024] The step S2 uses a method combining principal component analysis (PCA) projection technology with two-dimensional gridding to reduce the dimension of the point cloud on the basis of calculating the surface variation logarithm, and generates an image including the following steps:

[0025] Step 1: Use the principal component analysis (PCA) method to determine the projection plane:

[0026] (1) Construct the covariance matrix of the entire point cloud P:

[0027]

[0028] Where: N is the number of the entire point cloud, and the other variables have the same meaning as above. and is the matrix C P The eigenvectors corresponding to the two largest eigenvalues ​​represent the main direction of the point cloud and the direction vector, respectively, so we can and The plane where it is located is used as the point cloud projection plane. The eigenvector corresponding to the smallest eigenvalue λ0 is the plane normal vector.

[0029] (2) Calculate the coordinates of the projected point cloud:

[0030]

[0031] Where: Q i is the coordinate of the projected point, P i is a point in the original point cloud, is the center point of the point cloud, and U is a column full rank matrix.

[0032] Step 2: Mesh the plane point cloud and map the pixel information:

[0033] (1) For a two-dimensional point cloud Q = {Q i (x,y),i=1,2,...,N}, where N is the number of point clouds. First, calculate the number of grid rows and columns:

[0034]

[0035] Where: step is the grid step size, Row is the number of image pixel rows, Col is the number of columns, Y max , Y min , X max , X minare the maximum and minimum values ​​of the X and Y coordinates in the point cloud Q, respectively. Indicates rounding up.

[0036] Then calculate any point Q i The row and column number:

[0037]

[0038] Where: rowID, colID represent point Q i The row and column number of the grid. Indicates rounding down.

[0039] (2) Pixel information mapping: Each pixel of a two-dimensional image has only one RGB value corresponding to it. However, after rasterization, there may be three situations: there is only one point in a grid; there are multiple points in a grid; there is no point in the grid. When there is only one point in the grid, the RGB value corresponding to the surface variation of the point is used as the pixel RGB value; when there are multiple points in the grid, the average value of the RGB values ​​of each point cloud inside is calculated as the pixel RGB value; when there is no point in the grid, the eight-neighborhood average interpolation method is used to calculate, and the empty cell is filled by calculating the average value of the eight neighboring cells of the target cell (pixel, grid).

[0040] The step S3 uses the generated image to perform rough segmentation of the rivet area point cloud using the relevant image processing algorithm and the image point cloud mapping relationship, and includes the following steps:

[0041] Step 1: Rough extraction of rivet circle contour based on image:

[0042] (1) Grayscale the RGB image and perform Gaussian filtering to remove noise, and use the Canny algorithm to extract edge contours.

[0043] (2) The pixel perimeter of the calculated contour is S. To ensure the accuracy of the fitting, we only select circular contours with a radian greater than π / 4. Therefore, we restrict the above contour features and retain contours that meet the conditions.

[0044] (3) Perform Hough circle fitting on the extracted contour to obtain the rivet circle contour and the center and radius of each rivet circle

[0045] Step 2: Rough segmentation of the rivet area point cloud based on point cloud and image mapping: After obtaining the rivet circle contour parameters (center position, radius), inversely map it back to the point cloud coordinate system, construct an annular circle bounding box, and extract the rivet area point cloud:

[0046] (1) Inversely rasterize the circle parameters in the image coordinate system and map them to the projection plane. If the center of the circle has coordinates C in the image coordinate system, img (x img ,yimg ), with a radius of r img , since the origin of the image coordinate system is generally in the upper left corner, the projection coordinate system is:

[0047]

[0048] Where: C proj (x proj ,y proj ) is the coordinate of the center of the circle in the projection coordinate system, r proj is the radius in the projected coordinate system,

[0049] X min , Y max , step have the same meaning as above.

[0050] (2) Through inverse projection, C proj (x proj ,y proj ) is mapped to the point cloud coordinate system:

[0051]

[0052] C p (x, y, z) are the coordinates of the center of the circle in the point cloud coordinate system, U R is the right inverse matrix of the projection matrix U mentioned above, U, Consistent with the above meaning, U T is the U transposed matrix.

[0053] (3) When mapping two-dimensional data back to three-dimensional space, information will be lost, so C p It will fall on a straight line passing through the true center of the circle and in the same direction as the normal vector of the projection plane. With this straight line as the axis, concentric annular cylinders are constructed with radii r proj +2mm, r proj / 2. Extract the point cloud inside the annular cylinder as the rivet area point cloud to complete the rivet area

[0054] Coarse segmentation of domain point cloud.

[0055] In the step S4, a rivet convex-concave amount calculation model based on three-dimensional point cloud is designed, and then hierarchical fitting is performed according to the roughly extracted point cloud to obtain three parts, namely, rivet head point cloud, skin plane point cloud, and rivet contour point cloud, so as to accurately calculate the convex-concave amount.

[0056] Step 1: Design a calculation model for the convex and concave amount of rivets:

[0057]

[0058] Where: d flush is the convexity and concavity of the rivet, d maxis the maximum value from the rivet head point cloud to the skin plane, d min It is the minimum value from the rivet head point cloud to the skin plane.

[0059] Step 2: Rivet layered fitting:

[0060] (1) Fit the curvature variation with a Gaussian distribution and use the value of μ+2σ as the threshold to extract the rivet contour point cloud;

[0061] (2) The extracted rivet contour points are fitted into a spatial circle and the rivet area is divided into two parts, the inner green part is

[0062] The candidate point cloud of the rivet head and the candidate point cloud of the skin on the outside;

[0063] (3) Using the Ransac plane fitting algorithm, the rivet head candidate point cloud and the skin candidate point cloud are selected with appropriate bandwidths for plane fitting, respectively, to obtain the layered rivet head point cloud and skin plane point cloud;

[0064] Step 3: Calculate the distance from the rivet head point cloud to the skin plane, and use the rivet convexity and concave amount calculation model proposed in step 1 to calculate the rivet convexity and concave amount.

[0065] The beneficial effects of the present invention are:

[0066] The method for accurately extracting the convex-concave features of the rivet of the present invention first calculates the surface variation of the point cloud, and performs color mapping on the surface normal variation after mathematical processing. Then, a method combining principal component analysis (PCA) projection technology with two-dimensional gridding is used to reduce the dimension of the point cloud, and an image is generated, and a related image processing algorithm is used to perform a rough segmentation of the rivet area point cloud. Finally, based on hierarchical structure fitting, the rivet head point cloud and the skin point cloud are accurately extracted, and the rivet convex-concave amount is calculated. This method can effectively solve the problem of low calculation accuracy and efficiency of rivet convex-concave amount in the case of a large amount of point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Graph of point cloud surface variation calculation results.

[0068] Figure 2 Calculation results of the variational logarithm of the point cloud surface.

[0069] Figure 3 Schematic diagram of point cloud meshing.

[0070] Figure 4 Point cloud dimension reduction generates RGB image.

[0071] Figure 5 Image-based rough extraction process of rivet circle contour.

[0072] Figure 6Schematic diagram of rough segmentation of rivet area point cloud based on inverse mapping of image point cloud.

[0073] Figure 7 Schematic diagram of the rivet convexity and concaveness calculation model.

[0074] Figure 8 Diagram of the rivet layered fitting process. DETAILED DESCRIPTION

[0075] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0076] like Figure 1-5 shown.

[0077] A method for accurately extracting convex and concave features of rivets based on point cloud dimensionality reduction includes the following steps:

[0078] S1: Calculate the surface variation of the point cloud, and perform color mapping on the surface normal variation after mathematical processing;

[0079] S2: The principal component analysis (PCA) projection technique is combined with two-dimensional gridding to reduce the dimension of the point cloud and generate an image;

[0080] S3: Use relevant image processing algorithms and image point cloud mapping relationships to perform rough segmentation of the rivet area point cloud;

[0081] S4: Based on hierarchical structure fitting, the rivet head point cloud and skin point cloud are accurately extracted, and the rivet convexity and concavity are calculated.

[0082] Reference Figure 1-2 : In S1, the point cloud surface variation is calculated, and the surface normal variation is color mapped after mathematical processing, including the following steps:

[0083] Step 1: Figure 1 As shown in the figure, the surface variation estimation of point cloud is similar to the surface curvature as an indicator to describe the local geometric features of the surface. Compared with the curvature estimation based on function fitting, it is more suitable for simplifying point sampling surfaces and is widely used in the fields of point cloud data simplification. For any point Pi(xi,yi,zi) in the discrete point cloud P, the set of points in its r neighborhood is:

[0084] N r (P i )={P l ,P2,P3,…P i ,…P m} (1)

[0085] Where: P i is a point in the point cloud; N r (P i ) is P iis the center; r is the point set in the spherical area with radius; m is the number of points in the neighborhood of r.

[0086] Calculate the center point of the neighborhood point set:

[0087]

[0088] Where: P i is a point in the point cloud; is the center point of the domain point set; m is the number of points in the neighborhood of r.

[0089] Construct the point cloud covariance matrix and solve it:

[0090]

[0091] Where: C is a symmetric semi-positive definite matrix; λ = {λ0,λ1,λ2} is the eigenvalue of C, and λ0≤λ1≤λ2; is j The corresponding feature vector.

[0092] Define point P i The surface variation in the neighborhood of r is:

[0093]

[0094] Where: σ(P i ) represents the degree of flatness of the point in the local area, and its value range is (0~1 / 3). The smaller the value, the flatter the local area. When the value is 0, it represents a plane. λ0,λ1,λ2 are the eigenvalues ​​of C in (3).

[0095] Step 2: Figure 2 As shown, the logarithm of the rivet surface variation is calculated. Since the curvature of the aircraft surface skin is very small and the riveted area is also very small, the surface variation of the riveted area is very small and not very different, so the logarithm of the surface variation is calculated:

[0096] Reference Figure 3-4 :S2 uses the principal component analysis (PCA) projection technology combined with two-dimensional gridding to reduce the dimension of point cloud based on the calculation of surface variation logarithm, and generates images including the following steps:

[0097] Step 1: Use the principal component analysis (PCA) method to determine the projection plane:

[0098] (1) Construct the covariance matrix of the entire point cloud P:

[0099]

[0100] Where: N is the number of the entire point cloud, and the other variables have the same meaning as above. and is the matrix C P The eigenvectors corresponding to the two largest eigenvalues ​​represent the main direction of the point cloud and the direction vector, respectively, so we can and The plane where it is located is used as the point cloud projection plane. The eigenvector corresponding to the smallest eigenvalue λ0 is the plane normal vector.

[0101] (2) Calculate the coordinates of the projected point cloud:

[0102]

[0103] Where: Q i is the coordinate of the projected point, P i is a point in the original point cloud, is the center point of the point cloud, and U is a column full rank matrix.

[0104] Step 2: Mesh the plane point cloud and map the pixel information:

[0105] (1) Figure 3 As shown, for a two-dimensional point cloud Q = {Q i (x,y),i=1,2,...,N}, where N is the number of point clouds. First, count

[0106] Calculate the number of grid rows and columns:

[0107]

[0108] Where: step is the grid step size, Row is the number of image pixel rows, Col is the number of columns, Y max , Y min , X max , X min are the maximum and minimum values ​​of the X and Y coordinates in the point cloud Q, respectively. Indicates rounding up.

[0109] Then calculate any point Q i The row and column number:

[0110]

[0111] Where: rowID, colID represent point Q i The row and column number of the grid. Indicates rounding down.

[0112] (2) Figure 4As shown in the figure, pixel information mapping: Each pixel of a two-dimensional image has only one RGB value corresponding to it, but after rasterization, there may be the following three situations: there is only one point in a grid; there are multiple points in a grid; there is no point in the grid. For the above situations, when there is only one point in the grid, the RGB value corresponding to the surface variation of the point is used as the pixel RGB value; when there are multiple points in the grid, the average value of the RGB values ​​of each internal point cloud is calculated as the pixel RGB value; when there is no point in the grid, the eight-neighborhood average interpolation method is used to calculate, and the empty cells are filled by calculating the average value of the eight neighboring cells of the target cell (pixel, grid).

[0113] Reference Figure 5-6 : S3 uses the generated image to use the relevant image processing algorithm and the image point cloud mapping relationship to perform rough segmentation of the rivet area point cloud, including the following steps:

[0114] Step 1: Figure 5 As shown, the rough extraction of rivet circle contour based on image:

[0115] (1) Grayscale the RGB image and perform Gaussian filtering to remove noise, and use the Canny algorithm to extract edge contours.

[0116] (2) The pixel perimeter of the calculated contour is S. To ensure the accuracy of the fitting, we only select circular contours with a radian greater than π / 4. Therefore, we restrict the above contour features and retain contours that meet the conditions.

[0117] (3) Perform Hough circle fitting on the extracted contour to obtain the rivet circle contour and the center and radius of each rivet circle

[0118] Step 2: Figure 6 As shown in the figure, the rough segmentation of the rivet area point cloud based on point cloud and image mapping: after obtaining the rivet circle contour parameters (center position, radius), it is inversely mapped back to the point cloud coordinate system, and the annular circle bounding box is constructed to extract the rivet area point cloud:

[0119] (1) Inversely rasterize the circle parameters in the image coordinate system and map them to the projection plane. If the center of the circle has coordinates C in the image coordinate system, img (x img ,y img ), with a radius of r img , since the origin of the image coordinate system is generally in the upper left corner, the projection coordinate system is:

[0120]

[0121] Where: C proj (x proj ,y proj ) is the coordinate of the center of the circle in the projection coordinate system, r projis the radius in the projection coordinate system, X min , Y max , step have the same meaning as above.

[0122] (2) Through inverse projection, C proj (x proj ,y proj ) is mapped to the point cloud coordinate system:

[0123]

[0124] Where: C p (x, y, z) are the coordinates of the center of the circle in the point cloud coordinate system, U R is the right inverse matrix of the projection matrix U mentioned above, U, Consistent with the above meaning, U T is the U transposed matrix.

[0125] Mapping two-dimensional data back to three-dimensional space will lose information, so C p It will fall on a straight line passing through the true center of the circle and in the same direction as the normal vector of the projection plane. With this straight line as the axis, concentric annular cylinders are constructed with radii r proj +2mm, r proj / 2. Extract the point cloud inside the annular cylinder as the point cloud of the rivet area and complete the rough segmentation of the point cloud of the rivet area.

[0126] Reference Figure 7-8 : In S4, a rivet convexity and concavity calculation model based on three-dimensional point cloud is designed, and then the point cloud obtained by rough extraction is hierarchically fitted to obtain three parts, namely, rivet head point cloud, skin plane point cloud, and rivet contour point cloud, to accurately calculate the convexity and concavity.

[0127] Step 1: Figure 7 As shown in the figure, the calculation model of rivet convexity and concaveness is designed:

[0128]

[0129] Where: d flush is the convexity and concavity of the rivet, d max is the maximum value from the rivet head point cloud to the skin plane, d min It is the minimum value from the rivet head point cloud to the skin plane.

[0130] Step 2: Figure 8 As shown, the rivet is layered fitted:

[0131] (4) Fit the curvature variation with a Gaussian distribution and use the value of μ+2σ as the threshold to extract the rivet contour point cloud;

[0132] (5) The extracted rivet contour points are fitted into a spatial circle and the rivet area is divided into two parts, the inner green part is

[0133] The candidate point cloud of the rivet head and the candidate point cloud of the skin on the outside;

[0134] (6) Using the Ransac plane fitting algorithm, the rivet head candidate point cloud and the skin candidate point cloud are selected with appropriate bandwidths for plane fitting, respectively, to obtain the layered rivet head point cloud and skin plane point cloud;

[0135] Step 3: Calculate the distance from the rivet head point cloud to the skin plane, and use the rivet convexity and concave amount calculation model proposed in step 1 to calculate the rivet convexity and concave amount.

[0136] The present invention does not involve parts that are the same as the prior art or can be implemented by using the prior art.

Claims

1. A method for accurately extracting convex and concave features of rivets based on point cloud dimensionality reduction, characterized in that: The steps of the method for accurately extracting the convex and concave amount of the rivet are as follows: S1: Calculate the surface variation of the point cloud, and perform color mapping on the surface normal variation after mathematical processing; S2: The principal component analysis (PCA) projection technique is combined with two-dimensional gridding to reduce the dimension of the point cloud and generate an image; S3: Use image processing algorithm and image point cloud mapping relationship to perform rough segmentation of rivet area point cloud; S4: Based on hierarchical structure fitting, the rivet head point cloud and skin point cloud are accurately extracted, and the rivet convexity and concavity are calculated.

2. The method according to claim 1, characterized in that: Calculating the point cloud surface variation and color mapping the surface normal variation after mathematical processing includes the following steps: Step 1: Estimation of point cloud surface variation. Surface variation is an indicator that describes the local geometric features of the surface and is similar to the surface curvature. For any point Pi(xi,yi,zi) in the discrete point cloud P, the point set in its r neighborhood is: N r (P i )={P l ,P2,P3,…P i ,…P m } (1) Where: P i is a point in the point cloud; N r (P i ) is P i is the center; r is the point set in the spherical area with radius; m is the number of points in the neighborhood of r; Calculate the center point of the neighborhood point set: Where: P i is a point in the point cloud; is the center point of the domain point set; m is the number of points in the neighborhood of r; Construct the point cloud covariance matrix and solve it: Where: C is a symmetric semi-positive definite matrix; λ = {λ0,λ1,λ2} is the eigenvalue of C, and λ0≤λ1≤λ2; is j The corresponding eigenvector; Define point P i The surface variation in the neighborhood of r is: Where: σ(P i ) represents the degree of flatness of the point in the local area, and its value range is (0~1 / 3). The smaller the value, the flatter the local area. When the value is 0, it represents a plane. λ0,λ1,λ2 are the eigenvalues ​​of C in formula (3); Step 2: Calculate the logarithm of the rivet surface variation. Since the curvature of the aircraft skin is very small and the riveted area is also very small, the surface variation of the riveted area is very small and not very distinct, so the logarithm of the surface variation is calculated:

3. The method according to claim 1, characterized in that: The step S2 uses a method combining principal component analysis (PCA) projection technology with two-dimensional gridding to reduce the dimension of the point cloud on the basis of calculating the surface variation logarithm, and generates an image including the following steps: Step 1: Use the principal component analysis (PCA) method to determine the projection plane: (1) Construct the covariance matrix of the entire point cloud P: Where: N is the number of the entire point cloud, and the other variables have the same meaning as in formula (3); and is the matrix C P The eigenvectors corresponding to the two largest eigenvalues ​​represent the main direction of the point cloud and the direction vector, respectively, so we can and The plane where it is located is used as the point cloud projection plane; the eigenvector corresponding to the smallest eigenvalue λ2 is the plane normal vector. (2) Calculate the coordinates of the projected point cloud: Where: Q i is the coordinate of the point after projection, P i is a point in the original point cloud, is the center point of the point cloud, and U is a column full rank matrix; Step 2: Mesh the plane point cloud and map the pixel information: (1) For a two-dimensional point cloud Q = {Q i (x,y),i=1,2,...,N}, where N is the number of point clouds. First, calculate the number of grid rows and columns: Where: step is the grid step size, Row is the number of image pixel rows, Col is the number of columns, Y max , Y min , X max , X min are the maximum and minimum values ​​of the X and Y coordinates in the point cloud Q, respectively. Indicates rounding up; Then calculate any point Q i The row and column number: Where: rowID, colID represent point Q i The row and column number of the grid. Indicates rounding down. (2) Pixel information mapping: Each pixel in a two-dimensional image has only one RGB value corresponding to it. However, after rasterization, there are three situations: there is only one point in a grid; there are multiple points in a grid; there is no point in the grid; when there is only one point in the grid, the RGB value corresponding to the surface variation of the point is used as the pixel RGB value; when there are multiple points in the grid, the average RGB value of each point cloud inside is calculated as the pixel RGB value. When there is no point in the grid, the eight-neighborhood average interpolation method is used to calculate the target cell. Empty cells are filled with the average of the eight neighboring cells of the pixel and grid.

4. The method according to claim 1, characterized in that: The step S3 uses the image processing algorithm and the image point cloud mapping relationship to roughly segment the rivet area point cloud of the generated image, including the following steps: Step 1: Rough extraction of rivet circle contour based on image: (1) Grayscale the RGB image, perform Gaussian filtering to remove noise, and use the Canny algorithm to extract edge contours; (2) The pixel perimeter of the contour is calculated as S; to ensure the accuracy of the fitting, a circular contour with a radian greater than π / 4 is selected, so the above contour features are restricted and contours that meet the conditions are retained; (3) Performing Hough circle fitting on the extracted contour to obtain the rivet circle contour and the center and radius of each rivet circle; Step 2: Rough segmentation of the rivet area point cloud based on point cloud and image mapping: After obtaining the rivet circle contour parameters including the center position and radius, inversely map them back to the point cloud coordinate system, construct a circular bounding box, and extract the rivet area point cloud; (1) Inversely rasterize the circle parameters in the image coordinate system and map them to the projection plane. If the center of the circle in the image coordinate system is C img (x img ,y img ), with a radius of r img , since the origin of the image coordinate system is generally in the upper left corner, the projection coordinate system is: Where: C proj (x proj ,y proj ) is the coordinate of the center of the circle in the projection coordinate system, r proj is the radius in the projection coordinate system, X min , Y max , step have the same meaning as above. (2) Through inverse projection, C proj (x proj ,y proj ) is mapped to the point cloud coordinate system: C p (x, y, z) are the coordinates of the center of the circle in the point cloud coordinate system, U R is the right inverse matrix of the projection matrix U in equation (6), U, Consistent with the above meaning, U T is the U transposed matrix; (3) When mapping two-dimensional data back to three-dimensional space, information will be lost, so C p It will fall on a straight line passing through the true center of the circle and in the same direction as the normal vector of the projection plane. With this straight line as the axis, concentric annular cylinders are constructed with radii r proj +2mm, r proj / 2. Extract the point cloud inside the annular cylinder as the rivet area point cloud to complete the rivet area Point cloud coarse segmentation.

5. The method according to claim 1, characterized in that: In step S4, the rivet head point cloud and the skin point cloud are accurately extracted based on hierarchical structure fitting, and the rivet convexity and concavity are calculated, which includes the following steps: Step 1: Design a rivet convexity and concaveness calculation model: Where: d flush is the convexity and concavity of the rivet, d max is the maximum value from the rivet head point cloud to the skin plane, d min It is the minimum value from the rivet head point cloud to the skin plane; Step 2: Rivet layered fitting: (1) Fit the curvature variation with a Gaussian distribution and use the value of μ+2σ as the threshold to extract the rivet contour point cloud; (2) Perform spatial circle fitting on the extracted rivet contour points and divide the rivet area into two parts: the inner part is the rivet head candidate point cloud and the outer part is the skin candidate point cloud; (3) Using the Ransac plane fitting algorithm, the rivet head candidate point cloud and the skin candidate point cloud are selected with appropriate bandwidths for plane fitting, respectively, to obtain the layered rivet head point cloud and skin plane point cloud; Step 3: Calculate the distance from the rivet head point cloud to the skin plane, and use the rivet convexity and concave amount calculation model proposed in step 1 to calculate the rivet convexity and concave amount.

Images