A rivet concave-convex measurement method based on binocular face structured light

Abstract

The application discloses a rivet concave-convex quantity measurement method based on binocular face structure light, which comprises the following steps: collecting a structure light projection image on an aircraft skin by using a binocular camera; identifying a rivet from the image, extracting a region of interest with the rivet as the center and solving a phase; carrying out region segmentation according to a rivet profile, generating a point cloud of the relevant region; and calculating the concave-convex quantity of the rivet according to the point cloud. The application has the beneficial effects that a rivet concave-convex quantity measurement method based on binocular face structure light is proposed, compared with a traditional measurement method using a depth ruler, a plug gauge and other measuring tools, the method is simpler and faster under the premise of ensuring high precision; a three-dimensional topography reconstruction method is improved, and the problem of large redundant point cloud data and long time consumption in a conventional three-dimensional scanning measurement mode is overcome; and a rivet profile point cloud processing method based on local mean is proposed, which can realize accurate and reliable measurement of the concave-convex quantity of multiple rivets simultaneously.

Description

TECHNICAL FIELD

[0001] The present application relates to a rivet concave-convex measurement technology based on binocular face structure light, and belongs to the technical field of visual three-dimensional measurement. BACKGROUND

[0002] The concave-convex amount of a rivet refers to the maximum distance from a point on the head plane of a countersunk rivet to the surrounding structure surface, and is one of important indicators for measuring the riveting quality of an aircraft skin and is a key factor affecting the overall aerodynamic performance and component fatigue life of an aircraft.

[0003] The traditional measurement method mainly relies on a feeler gauge to touch and compare or uses a depth gauge with an auxiliary support to contact measure, the traditional method is inconvenient to use tools, and manual judgment may exist human error, and the contact measurement is easy to scratch the surface of the aircraft. The method using a scanning device needs to scan the entire surface without distinction, and a large amount of irrelevant point cloud data is included, and after collecting data, special software is still needed to process and analyze the massive point cloud data, which is low in efficiency and cannot be applied to real-time measurement. Although the measurement method based on line structure light is high in efficiency, only the point cloud data on the structure light line can be collected, the contour information of the entire rivet region cannot be obtained, and the measurement result is not accurate enough.

[0004] Therefore, the present application proposes a rivet concave-convex measurement method based on binocular face structure light. Unlike the traditional three-dimensional measurement method of first scanning to generate a three-dimensional point cloud and then analyzing and extracting relevant information from the point cloud, the present application first analyzes the region related to the rivet from the image and then generates the point cloud of the relevant region, and the measurement efficiency is greatly improved through optimization of the phase solving method and the same name point matching method. In addition, the present application proposes a rivet contour recognition algorithm and a rivet profile point cloud processing algorithm, and the accuracy and stability of the measurement result are improved. SUMMARY

[0005] TECHNICAL PROBLEM

[0006] The present application aims to provide a convenient, accurate and reliable rivet concave-convex measurement method for an aircraft to solve the problems in the background.

[0007] TECHNICAL SCHEME

[0008] To achieve the above object, the present application provides the following technical scheme: a rivet concave-convex measurement method based on binocular face structure light, the method comprising the following steps:

[0009] S1: synchronously collecting the stripe images continuously projected on the aircraft skin by an industrial projector using a calibrated binocular industrial camera, detecting the contour of the rivet in the left and right camera images, and extracting the region of interest centered on the rivet;

[0010] S2: processing and analyzing the fringe image in the region of interest, calculating phase information and generating a phase map, correcting the phase map and the white light projection image;

[0011] S3: segmenting the rivet head region and the adjacent skin region according to the rivet contour, matching the homonymic points in the relevant regions, and generating a point cloud according to the homonymic points;

[0012] S4: filtering the gross error points in the point cloud by using the RANSAC method, and calculating the concave-convex amount of the rivet to be measured.

[0013] As an improvement of the present application, in step S1, the fringe image projected by the industrial projector on the aircraft skin continuously includes one white light image, N1 sinusoidal fringe images and N2 Gray code images. The rivet contour in the white light image is not disturbed by the fringe pattern, which is used to identify and locate the rivet, extract the region of interest, and segment the rivet head region and the adjacent skin region. The sinusoidal fringe image and the Gray code image are used to calculate the phase and generate a phase map.

[0014] As an improvement of the present application, in step S1, the detection of the rivet contour in the left and right camera images includes the following steps:

[0015] S11: performing median filtering and Gaussian filtering on the white light projection image collected by the left and right cameras to remove the impulse noise, salt and pepper noise and Gaussian noise caused by the image sensor and the environment light, enhancing the image contrast of the rivet contour by Gamma transformation, and preliminarily extracting the edge skeleton line in the image by Canny operator;

[0016] S12: searching each skeleton line, calculating the length L and the width H of the minimum circumscribed rectangle of the skeleton line, and obtaining the aspect ratio Ratio of the minimum circumscribed rectangle as follows:

[0017] Ratio=L / H

[0018] Calculating the number of pixels I occupied by each skeleton line and the area Area of the minimum circumscribed rectangle thereof:

[0019] Area=LH

[0020] According to the characteristics of the rivet contour approximating to a circular arc, the curvature of the skeleton line involved in fitting the rivet circular contour should not be too small, i.e. the aspect ratio Ratio of the minimum circumscribed rectangle thereof should not be too large, therefore a threshold value T0 is set for Ratio. According to the characteristics of the fixed size specification of the aviation rivet, a lower limit I min and an upper limit I max are set for I. A lower limit S min and an upper limit S max are set for Area. The rivet edge skeleton line should satisfy the following conditions:

[0021]

[0022] Skeleton lines not meeting the above conditions are removed.

[0023] S13: According to the characteristics that the skeleton line may have a branch or a backhand, search the pixels on each skeleton line one by one, analyze whether the pixel is a branch point or an inflection point, and set the gray value of the pixel determined as a branch point or an inflection point to 0;

[0024] S14: Repeat step S12, and use the RANSAC algorithm to fit the contour circle for the retained skeleton line, calculate the pixel coordinates of the contour circle center, and the distance D between any two contour circles is ij :

[0025] 1≤i,j≤M1 and i≠j.

[0026] In the formula, O i is the fitting center of the i th skeleton line, O j is the fitting center of the j th skeleton line, and M1 is the total number of retained skeleton lines.

[0027] A distance threshold d is set, the retained skeleton lines are classified, the skeleton lines with a distance between contour circle centers ≤d are put into the same set as the edge skeleton lines of the same rivet. After classification, if there are K sets, it indicates that there are K rivets in the camera field of view. Then, the least square method is used to fit the contour circle for the skeleton lines in each set as the contour of the corresponding rivet, and the center of the contour circle is determined.

[0028] The advantage of the above-mentioned contour circle detection algorithm compared with the conventional image circle detection algorithm is that whether each skeleton line is similar to an arc is analyzed according to the arc characteristic, the non-arc skeleton line is removed through detection of branch points and inflection points, the data calculation amount is smaller, the running efficiency is higher, and the algorithm is more suitable for the scene of detecting rivets on a skin surface.

[0029] As an improvement of the present application, in step S1, the extracted region of interest centered on the rivet is:

[0030] E k ={(x,y)|x k -R k -b≤x≤x k +R k +b,y k -R k -b≤y≤y k +R k +b},k=1,2,…,K.

[0031] In the formula, (x k ,y k ) is the contour circle center coordinate of the k th rivet, R kR is the radius of the profile circle of the kth rivet, b is based on the profile radius R k Expanded width, b = 0.5R k .

[0032] As an improvement of the present application, in step S2, the processing and analysis of the fringe image in the region of interest, the calculation of the phase information and the generation of the phase map means that only the sinusoidal fringe and the Gray code image in the region of interest extracted in step S1 are analyzed and calculated to generate the phase value, and only the phase information in the region of interest is contained in the generated phase map, and the phase value outside the region of interest is set to 0.

[0033] As an improvement of the present application, in step S3, the rivet head region and the adjacent skin region are segmented according to the rivet profile, which requires that the rivet be re-detected using the rivet profile detection algorithm in step S1 on the corrected white light projection image, the rivet profile circle be fitted, the center of the profile circle be determined, and the left image be segmented with the profile circle as the boundary, and the segmented rivet head region is:

[0034]

[0035] In the formula, (x′ k_L ,y′ k_L ) is the center coordinate of the profile circle of the kth rivet in the corrected left image, and the corresponding profile circle radius is R′ k_L . R′ ak_L = Round (0.6R′ k_L ), R′ bk_L = Round (0.9R′ k_L ), and E′ top_k_L is an annular band formed by the circles concentric with the profile circle and with radii of R′ ak_L and R′ bk_L .

[0036] The segmented adjacent skin region is:

[0037]

[0038] In the formula, R′ ck_L = Round (1.1R′ k_L ), R′ dk_L = Round (1.4R′ k_L ), and E′ bottom_k_L is an annular band formed by the circles concentric with the profile circle and with radii of R′ ck_L and R′ dk_L .

[0039] As an improvement of the present application, in step S3, the feature points in the matching related area are matched, in order to further improve the matching efficiency, the disparity constraint is added on the basis of the epipolar constraint to reduce the search range of the homonym point. The process is as follows:

[0040] S31: the disparity of the same rivet in the corrected left image and right image is calculated, for the kth rivet, the disparity is:

[0041]

[0042] In the formula, (x′ k_L ,y′ k_L ) is the center coordinate of the profile circle of the rivet in the kth interested area E′ k_L of the left image after correction, (x′ k_R ,y′ k_R ) is the center coordinate of the profile circle of the rivet in the kth interested area E′ k_R of the right image after correction.

[0043] S32: according to the corrected phase diagram, the pixels in the rivet head area E′ top_k_L or the adjacent skin area E′ bottom_k_L search the homonym pixels in the right image. The corrected image is subjected to the epipolar constraint, that is, the homonym points in the left and right images are located at the same image height. In order to further reduce the search range of the homonym point, the disparity dis(x′ k_L ,y′ k_L ) is constrained. For any pixel P′ L (x′ L ,y′ L ) in the left phase diagram, the search is started from the pixel P′ R_start in the right phase diagram along the image x-axis direction, and the coordinate of P′ R_start is:

[0044]

[0045] In the formula, Δdis is a set redundancy value.

[0046] The pixels meeting the following conditions are put into the set of right homonym points to be matched:

[0047] |At(x′ L ,y′ L )-At(x′ Rt ,y′ Rt )|≤ΔAt

[0048] In the formula, At(x′ L ,y′ L ) is the phase value of the pixel P′ L , At(x′ Rt ,y′Rt ) is the phase value of the tth searched pixel in the right phase map, and ΔAt is a set threshold value.

[0049] The search is completed when any of the following criteria is met:

[0050] (1) |At(x′ L ,y′ L )-At(x′ Rt ,y′ Rt )|>ΔAt and |At(x′ L ,y′ L )-At(x′ Rt ,y′ Rt )|>|At(x′ L ,y′ L )-At(x′ R(t-1) ,y′ R(t-1) )|;

[0051] (2) or x′ RT >W-2, W is the image width.

[0052] A reference matching frame is set with the pixel P′ L and its 8-neighborhood pixels, and a matching frame is set with the pixel P′ Rt (x′ Rt ,y′ RT ) in the set of pixels to be matched as the center of the matching frame containing 8-neighborhood pixels, the matching frame is compared with the reference matching frame, and the phase difference value is calculated according to the following formula:

[0053]

[0054] The pixel to be matched with the smallest phase difference value is selected as the coarsely selected right homonym.

[0055] S33: The three times spline curve interpolation is performed on the coarsely selected right homonym and the two pixels before and after the coarsely selected right homonym, and the horizontal coordinate x′ R of the right homonym with sub-pixel accuracy is calculated.

[0056] As an improvement of the present application, in step S3, the specific implementation of generating the point cloud according to the homonym pairs is that the homonym pairs have a mapping relationship with the space point cloud, the relevant calibration parameters are obtained by calibrating the binocular camera, and the re-projection matrix Q representing the mapping relationship is calculated by the calibration parameters. The spatial coordinates of the point cloud data are calculated by the following formula:

[0057] [X Y Z W] T =Q*[x′ L y′ L dis(x′ Ly' L ) 1] T

[0058] dis(x' L ,y' L ) is the disparity of the homonym point pair, dis(x' L ,y' L ) = x' L -x' R , (X, Y, Z) is the three-dimensional coordinates of the space point cloud corresponding to the homonym point pair.

[0059] The method has the advantages that compared with the conventional structured light three-dimensional reconstruction method, the range of phase solving is reduced by extracting the region of interest, the region of the generated point cloud is locked to the rivet-related profile by region segmentation, and the search range of the homonym point is reduced by increasing the disparity constraint, so that the data calculation amount in the program running process is greatly reduced, and the efficiency of the whole algorithm running is significantly improved.

[0060] As an improvement of the application, in step S4, in the process of calculating the concave-convex amount of the measured rivet, for the case that the single point cloud is not reliable, the local mean method is used to improve the stability of the measurement result. A fan-shaped sliding window is set for the annular point cloud belt of the rivet head area E' top_k_L , a certain number of point clouds are contained in the window, and the window slides in the counterclockwise direction. When the angle is θ, the left image pixel set corresponding to the point cloud in the window is denoted as:

[0061]

[0062]

[0063] The rivet head area is traversed through the sliding window, the Euclidean distance mean of the point cloud in the window to the installation plane under each sliding is calculated as the height difference of this small area to the skin. The distance of a single point cloud in the window to the skin is expressed by the following formula:

[0064]

[0065] In the formula, d j is the distance of the jth point cloud P j in the sliding window to the skin, (x i ,y i ,z i ) is the space point cloud coordinate corresponding to the point P j , M θ is the total number of point clouds contained in the window, and Ax+By+Cy+D=0 is the plane equation fitted according to the point clouds in the adjacent skin area E' bottom_k_L .

[0066] Distance d of the area where the single sliding window is located relative to the skin θ :

[0067]

[0068] By calculating the distance d of the local area within each window relative to the skin θ , the change range of the distance from the rivet head to the skin is obtained, thereby determining the concave-convex amount. The local mean is used instead of the distance value of a single point in the scheme, which improves the stability of the measurement result.

[0069] Advantages:

[0070] Compared with the prior art, the advantages of the present application are:

[0071] 1) The traditional manual contact measurement mode can be replaced, and compared with the traditional measurement method relying on a depth gauge or a depth gauge with an auxiliary support, the present application uses visual measurement, which does not cause the problem of scratching the surface of the measured part by a hard tool, and does not exist the case of manual misjudgment, and under the premise of ensuring high measurement accuracy, the operation is more simple and efficient;

[0072] 2) The profile area related to riveting is segmented from the image, and the point cloud of the related area is generated, which reduces the range of phase calculation and the search range of the same name point, and compared with the method of scanning the contour by using a point cloud scanning device and then analyzing and processing all point cloud data, the measurement algorithm used in the present application has a significantly improved efficiency, and the measurement result can be obtained after the image is collected, which meets the demand of instant measurement;

[0073] 3) A riveting profile point cloud processing method based on local mean is proposed, compared with the line structured light measurement method, the rivet contour data obtained by the present application is more comprehensive and accurate, and the measurement result is more accurate when the rivet head plane is relatively large relative to the skin surface. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a schematic diagram of a riveting test piece measured by a binocular face structured light measurement system.

[0075] Figure 2 is a schematic diagram of a white light projection measured skin surface collected by a binocular camera, wherein (a) is a left camera collection diagram, and (b) is a right camera collection diagram.

[0076] Figure 3 is a schematic diagram of a sinusoidal fringe and a Gray code projected by an industrial projector in the area where the rivet is located.

[0077] Figure 4is the schematic diagram of identifying rivet edge fitting contour circle, wherein (a) is the original drawing, (b) is the skeleton line thinning, (c) is the skeleton line filtering, (d) is the broken bifurcation point and inflection point, (e) is the skeleton line filtering again, and (f) is the fitting contour circle.

[0078] Figure 5 is the schematic diagram of extracting the region of interest centered on the rivet in the left and right camera images, wherein (a) is the schematic diagram of the region of interest in the left camera image, and (b) is the schematic diagram of the region of interest in the right camera image.

[0079] Figure 6 is the schematic diagram of phase information in the left and right camera images, wherein (a) is the left phase diagram, and (b) is the right phase diagram.

[0080] Figure 7 is the schematic diagram of the rivet head region and the adjacent skin region.

[0081] Figure 8 is the schematic diagram of matching feature points in the phase diagram by using a sliding window, wherein (a) is the local enlarged schematic diagram of the left phase diagram, and the square in it is the reference matching box, (b) is the local enlarged schematic diagram of the right phase diagram, and the square in it is the to-be-matched box.

[0082] Figure 9 is the point cloud visualization schematic diagram of the extracted rivet head region and the adjacent skin region, wherein (a) is the point cloud overhead view,

[0083] (b) is the point cloud side view.

[0084] List of reference signs:

[0085] 1-industrial projector, 2-industrial camera, 3-riveted test piece, 4-rivet head region, 5-adjacent skin region, 6-sliding window, 7-adjacent skin region, 8-rivet head region. DETAILED DESCRIPTION

[0086] The application will be described in more detail below with reference to the drawings and examples.

[0087] Example 1:

[0088] Figure 1 An application schematic diagram of a rivet concave-convex measurement method for an aircraft based on binocular face structured light provided by the application is shown, and the method comprises the following steps:

[0089] S1: using the calibrated binocular industrial camera, the industrial projector is used to synchronously collect the stripe images continuously projected on the aircraft skin, the profile of the rivet in the left and right camera images is detected, and the region of interest centered on the rivet is extracted;

[0090] S2: processing and analyzing the fringe images in the region of interest, calculating the phase information and generating a phase map, correcting the phase map and the white light projection image;

[0091] S3: segmenting the rivet head region and the adjacent skin region according to the rivet contour, matching the homonymic points in the relevant regions, and generating a point cloud according to the homonymic points;

[0092] S4: filtering the gross error points in the point cloud using the RANSAC method, and calculating the concave-convex amount of the rivet to be measured.

[0093] In step S1, the fringe images projected by the industrial projector on the aircraft skin include one white light image, N1 sinusoidal fringe images, and N2 Gray code images. This embodiment uses 4-step sinusoidal phase shift fringe combined with 4-order Gray code, and additionally supplements one frame of inverse code, so N1 is 4 and N2 is 5. The rivet contour in the white light image (such as Figure 2 ) is not disturbed by the fringe pattern, which is used to identify and locate the rivet, and to extract the rivet head region and the adjacent skin region in the image. The sinusoidal fringe image and the Gray code image (such as Figure 3 ) are used for processing and calculation to generate a phase map.

[0094] In step S1, the detection of the rivet contour in the left and right camera images includes the following steps:

[0095] S11: performing median filtering and Gaussian filtering on the white light projection images collected by the left and right cameras to remove impulse noise, salt and pepper noise, and Gaussian noise caused by the image sensor and the environment light, enhancing the image contrast of the rivet contour using Gamma transformation, and using the Canny operator to preliminarily extract the edge skeleton lines in the image;

[0096] S12: searching for each skeleton line, calculating the length L and the width H of the minimum circumscribed rectangle of the skeleton line, and obtaining the aspect ratio Ratio of the minimum circumscribed rectangle as follows:

[0097] Ratio = L / H

[0098] Calculating the number of pixels I occupied by each skeleton line and the area Area of the minimum circumscribed rectangle thereof:

[0099] Area = LH

[0100] According to the characteristics of the rivet edge contour being approximately a circular arc, the curvature of the skeleton line involved in fitting the rivet circular contour should not be too small, i.e., the aspect ratio Ratio of the minimum circumscribed rectangle should not be too large, so a threshold value T0 is set for Ratio. According to the fixed size specification characteristics of the aviation rivet, a lower limit I min and an upper limit I max are set for I, and a lower limit S min and an upper limit S maxThe rivet edge skeleton line should satisfy the following conditions:

[0101]

[0102] In this embodiment, the minimum radian of the skeleton line meeting the conditions is π / 4, the corresponding T0 is 4.83, I min is 100, I max is 2500, S min is 20, S max is 40000. The skeleton line not meeting the above conditions is removed.

[0103] S13: According to the characteristics that the skeleton line may have bifurcation or inflection, search the pixels on each skeleton line one by one, analyze whether the pixel is a bifurcation point or an inflection point, and set the gray value of the pixel determined as a bifurcation point or an inflection point to 0;

[0104] S14: Repeat step S12, and fit the contour circle for the retained skeleton line by using the RANSAC algorithm to calculate the pixel coordinates of the contour circle center, and the distance D ij between any two contour circles is:

[0105] 1≤i,j≤M1 and i≠j

[0106] In the formula, O i is the fitting center of the i th skeleton line, O j is the fitting center of the j th skeleton line, and M1 is the total number of retained skeleton lines.

[0107] A distance threshold d is set, the retained skeleton lines are classified, the skeleton lines with a distance between contour circle centers ≤d are put into the same set, and are taken as the edge skeleton lines of the same rivet. In this embodiment, d is set to 200. There are K rivets in the projection range, so there are K sets after classification. In this embodiment, there are four rivets in the projection range, that is, K is 4. Then, the least square method is used to fit the contour circle of the skeleton line in the k th set, as the contour of the corresponding rivet (such as Figure 4 ), and the center of the contour circle of the corresponding rivet is determined.

[0108] In step S1, the extracted region of interest (such as Figure 5 ) centered on the rivet is:

[0109] E k ={(x,y)|x k -R k -n≤x≤x k +R k +n,y k -R k -b≤y≤y k +R k+ b}, k = 1, 2, …, k.

[0110] where (x k ,y k ) is the center coordinate of the profile circle of the kth rivet, R k is the radius of the profile circle of the kth rivet, and b is the width based on the profile radius R k , b = 0.5R k . In this embodiment, the center coordinate of the profile circle of the 1st rivet in the left image is (417, 240), the radius is 90, the center coordinate of the profile circle of the 2nd rivet is (1125, 254), the radius is 74, the center coordinate of the profile circle of the 3rd rivet is (415, 788), the radius is 89, and the center coordinate of the profile circle of the 4th rivet is (1120, 781), the radius is 73; the center coordinate of the profile circle of the 1st rivet in the right image is (195, 261), the radius is 86, the center coordinate of the profile circle of the 2nd rivet is (911, 252), the radius is 79, the center coordinate of the profile circle of the 3rd rivet is (196, 794), the radius is 85, and the center coordinate of the profile circle of the 4th rivet is (908, 808), the radius is 76.

[0111] In step S2, the processing and analysis of the fringe image in the region of interest, the calculation of the phase information and the generation of the phase map refer to the analysis and calculation of the phase value only for the sinusoidal fringe image and the Gray code image in the region of interest extracted in step S1, and the generated phase map (such as Figure 6 ) only contains the phase information in the region of interest, and the phase values outside the region of interest are all set to 0.

[0112] In step S3, the rivet head region and the adjacent skin region (such as Figure 7 ) are segmented according to the rivet profile, which requires that the rivet profile detection algorithm in step S1 is used to re-detect the rivet, fit the profile circle, and determine the center of the profile circle for the corrected white light projection image. The segmented rivet head region for the left image is:

[0113]

[0114] where (x′ k_L ,y′ k_L ) is the center coordinate of the profile circle of the kth rivet in the corrected left image, and the corresponding profile circle radius is R′ k_L . R′ ak_L = Round(0.6R′ k_L ), R′ bk_L = Round(0.9R′ k_L ), and E′ top_k_L is a circle concentric with the profile circle and with a radius of R′ ak_L .and R′ bk_L The annular band formed by the circle. In this embodiment, the radius of the first rivet after correction is 47, corresponding to R′ a1_L =28, R′ b1_L =42, the radius of the second rivet after correction is 41, corresponding to R′ a2_L =24, R′ b2_L =36, the radius of the third rivet after correction is 46, corresponding to R′ a3_L =27, R′ b3_L =41, the radius of the 4th rivet after correction is 40, corresponding to R′ a4_L =24, R′ b4_L =36.

[0115] The separated adjacent skin areas are:

[0116]

[0117] In the formula, R′ ck_L =Round(1.1R′) k_L ), R′ dk_L =Round(1.4R′) k_L ), E′ bottom_k_L It is concentric with the outline circle and has a radius of R′. ck_L and R′ dk_L A ring-shaped band formed by circles. In this embodiment, R′ corresponds to the first rivet. c1_L =51, R′ d1_L =65, R′ corresponding to the second rivet c2_L =45, R′ d2_L =57, R′ corresponding to the 3rd rivet c3_L =50, R′ d3_L =64, R′ corresponding to the 4th rivet c4_L =44, R′ d4_L =56.

[0118] In step S3, feature points within the relevant regions are matched. To further improve matching efficiency, a disparity constraint is added to the epipolar constraint to narrow the search range of corresponding points. The process is as follows:

[0119] S31: Calculate the disparity of the same rivet in the corrected left and right images. For the k-th rivet, its disparity is:

[0120]

[0121] In the formula, (x′ k_L ,y′ k_L E′ represents the k-th region of interest in the corrected left image. k_L The coordinates of the center of the outline circle of the rivet, (x′k_R ,y′ k_R E′ represents the k-th region of interest in the corrected right image. k_R The coordinates of the center of the outline circle of the rivet. In this example, the disparity dis(x′) of the first rivet. 1_L ,y′ 1_L =627, the disparity dis(x′) of the second rivet 2_L ,y′ 2_L =628, the disparity dis(x′) of the 3rd rivet 3_L ,y′ 3_L =625, the parallax of the 4th rivet is dis(x′) 4_L ,y′ 4_L =628.

[0122] S32: Search for the rivet head region E′ based on the corrected phase map. top_k_L Or adjacent skin zone E′ bottom_k_L The pixel in the first image has its corresponding pixel in the second image. The corrected image is constrained by epipolar lines, meaning that corresponding points in the left and right images lie at the same image height. To further narrow down the search range for corresponding points, disparity dis(x′) is used. k_L ,y′ k_L Constraints are applied to any pixel P′ in the left phase image. L (x′ L ,y′ L ), from pixel P′ in the right phase image R_start Start the search along the x-axis of the image, P′ R_start The coordinates are:

[0123]

[0124] In the formula, Δdis is the set redundancy value. In this embodiment, Δdis is 10.

[0125] Pixels that meet the following conditions will be added to the set of right-named pixels to be matched:

[0126] |At(x′ L ,y′ L )-At(x′ Rt ,y′ Rt )|≤ΔAt

[0127] In the formula, At(x′) L ,y′ L ) represents pixel P′ L The phase value, At(x′) Rt ,y′ Rt Let ΔAt be the phase value of the t-th pixel searched in the right phase image, and let ΔAt be the set threshold. In this embodiment, ΔAt is 0.2.

[0128] The search is completed when any of the following criteria are met:

[0129] (1)|At(x′ L ,y′ L )-At(x′ Rt ,y′ Rt )|>ΔAt

[0130] And |At(x′) L ,y′ L )-At(x′ Rt ,y′ Rt )|>|At(x′ L ,y′ L )-At(x′ R(t-1) ,y′ R(t-1) )|;

[0131] (2) or x′ Rt >W-2, where W is the image width.

[0132] With pixel P′ L The reference matching box is formed by the pixel P' and its 8 neighboring pixels. Pixel P' is the pixel in the set of right-side corresponding points to be matched. Rt (x′ Rt ,y′ Rt ) Set up a matching bounding box centered on the pixel and containing 8 neighboring pixels for each pixel, and compare the matching bounding box with the reference matching bounding box (e.g. Figure 8 The phase difference is calculated using the following formula:

[0133]

[0134] The right pixel with the smallest phase difference is selected as the coarse right corresponding point.

[0135] S33: Select the coarsely selected right corresponding point and the two pixels before and after it, perform cubic spline curve interpolation, and calculate the x′ of the right corresponding point with sub-pixel precision. R .

[0136] In step S3, the specific implementation of generating a point cloud based on corresponding point pairs lies in the existence of a mapping relationship between the corresponding point pairs and the spatial point cloud. Camera calibration parameters are obtained by calibrating the stereo camera, and the reprojection matrix Q representing the mapping relationship is calculated using these parameters. The spatial coordinates of the point cloud data are calculated using the following formula:

[0137] [XYZW] T =Q*[x′ L y′ L dis(x′ L ,y′ L ) 1]T

[0138] where dis(x', y') is the disparity of the same point pair, dis(x', y') = x' - x', (x, y, z) is the three-dimensional coordinate of the corresponding space point cloud. L L L L L R where dis(x', y') is the disparity of the same point pair, dis(x', y') = x' - x', (x, y, z) is the three-dimensional coordinate of the corresponding space point cloud.

[0139] In step S4, in the process of calculating the concave-convex amount of the rivet to be measured, the local mean method is used to improve the stability of the measurement result in the case of low reliability of a single point cloud. A fan-shaped sliding window (such as top_k_L ) is set for the annular point cloud belt of the rivet head area E' Figure 9 , a certain number of point clouds are contained in the window, and the window slides counterclockwise from the 3 o'clock direction. When the angle is θ, the left image pixel set corresponding to the point cloud in the window is denoted as:

[0140]

[0141]

[0142] The rivet head area is traversed through the sliding window, the average of the Euclidean distances of the point clouds in the window to the installation plane at each sliding is calculated, and the height difference of the small area to the skin is taken as the average. The distance of a single point cloud in the window to the skin is expressed by the following formula:

[0143]

[0144] where (x i ,y i ,z i ) is the three-dimensional coordinate of the jth point cloud in the window, M θ is the total number of point clouds contained in the window, and Ax+By+Cy+D=0 is the plane equation fitted according to the point clouds in the adjacent skin area E' bottom_k_L .

[0145] The distance d θ of the area where the single sliding window is located to the skin:

[0146]

[0147] The distance d θ ​​​​​That is, the range of the distance between the rivet head and the skin is obtained, and the concave-convex amount is determined. In this embodiment, the range of the distance between the head of the first rivet and the skin is -0.43mm to -0.019mm, the range of the distance between the head of the second rivet and the skin is 0.050mm to 0.076mm, the range of the distance between the head of the third rivet and the skin is -0.087mm to -0.065mm, and the range of the distance between the head of the fourth rivet and the skin is 0.142mm to 0.161mm. The range of the concave-convex amount of the aviation flat head rivet is ±0.1mm, so the first to third rivets are qualified, and the fourth rivet is unqualified.

[0148] The application discloses a rivet concave-convex amount measurement method based on binocular face structure light, which is simpler and faster than the traditional measurement method using a depth ruler, a plug gauge and the like, and ensures high precision; the three-dimensional appearance reconstruction method is improved, and the problem of large redundant point cloud data and long time consumption in the conventional three-dimensional scanning measurement mode is solved; a riveting profile point cloud processing method based on local mean is provided, and accurate and reliable measurement of the concave-convex amount of multiple rivets can be realized.

[0149] The technical means disclosed in the application scheme is not limited to the technical means disclosed in the above embodiments, and also includes technical solutions composed of any combination of the above technical features. It should be pointed out that, for ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements are also considered to be within the protection scope of the application.

Claims

1. A rivet concave-convex measurement method for an aircraft based on binocular face structured light, characterized in that, The method comprises: S1: using the calibrated binocular industrial camera to synchronously collect the stripe images continuously projected on the aircraft skin by the industrial projector, detecting the profile of the rivet in the left and right camera images, and extracting the region of interest centered on the rivet; S2: processing and analyzing the stripe images in the region of interest, calculating the phase information and generating a phase map, and correcting the phase map and the white light projection image; S3: dividing the rivet head region and the adjacent skin region according to the rivet profile, matching the homonymic points in the relevant regions, and generating a point cloud according to the homonymic points; S4: filtering the coarse error points in the point cloud by using the RANSAC method, and calculating the concave-convex amount of the rivet to be measured; In step S4, in the process of calculating the concave-convex amount of the rivet to be measured, the local mean method is used to improve the stability of the measurement results in the case of low reliability of a single point cloud. A fan-shaped sliding window is set for the annular point cloud belt of the rivet head region E' top_k_L , a certain number of point clouds are contained in the window, and the window slides counterclockwise from the 3 o'clock direction. When the angle is θ, the left image pixel set corresponding to the point clouds in the window is denoted as: By traversing the rivet head region through a sliding window, the average of the Euclidean distances of the point cloud in the window to the installation plane at each sliding is calculated as the height difference of the small region to the skin; the distance of a single point cloud in the window to the skin is represented by the following formula: In the formula, (x i ,y i ,z i ) is the three-dimensional coordinate of the jth point cloud in the window, M θ is the total number of point clouds contained in the window, Ax+By+Cy+D=0 is the plane equation fitted according to the point clouds in the adjacent skin area E′ bottom_k_L ; Distance d of the area where the single sliding window is located relative to the skin θ : By calculating the distance d of the local area in each window to the skin θ That is, the distance change range of the rivet head to be tested from the assembly surface is obtained, so as to determine the concave-convex amount.

2. The binocular structured light based rivet bump measurement method for aircraft according to claim 1, wherein, In step S1, the stripe images continuously projected on the aircraft skin by the industrial projector include one frame of white light image, N1 frames of sinusoidal stripe image and N2 frames of Gray code image; the white light image is used for identifying and positioning the rivet, extracting the region of interest, and dividing the rivet head region and the adjacent skin region; the sinusoidal stripe image and the Gray code image are used for calculating the phase and generating a phase map.

3. The binocular structured light based rivet bump measurement method for aircraft according to claim 1, wherein, In step S1, the detection of the profile of the rivet in the left and right camera images comprises the following steps: S11: performing median filtering and Gaussian filtering on the white light projection images collected by the left and right cameras to remove the impulse noise, salt and pepper noise and Gaussian noise caused by the image sensor and the environmental light, enhancing the image contrast of the rivet profile by using Gamma transformation, and preliminarily extracting the edge skeleton lines in the image by using the Canny operator; S12: searching each skeleton line to obtain the length L and the width H of the minimum circumscribed rectangle of the skeleton line, and obtaining the aspect ratio Ratio of the minimum circumscribed rectangle as follows: Ratio = L / H calculating the number I of pixels occupied by each skeleton line and the area Area of the minimum circumscribed rectangle thereof: Area = LH According to the characteristics that the rivet profile approximates to a circular arc, a threshold value T0 is set for Ratio; according to the characteristics that the size specification of the aviation rivet is fixed, a lower limit I min and an upper limit I max are set for I; a lower limit S min and an upper limit S max are set for Area; and the rivet edge skeleton line should satisfy the following conditions: removing the skeleton lines that do not meet the above conditions; S13: according to the characteristics that the skeleton lines may have bifurcations or inflection points, searching the pixels on each skeleton line one by one, analyzing whether the pixels are bifurcation points or inflection points, and setting the gray value of the pixels determined as bifurcation points or inflection points to 0; S14: repeat step S12, use RANSAC algorithm to fit contour circle for the reserved skeleton line, calculate the pixel coordinates of the contour circle center, and the distance D between any two contour circles ij is: In the formula, O i is the fitting circle center of the i-th skeleton line, O j is the fitting circle center of the j-th skeleton line, and M1 is the total number of retained skeleton lines. setting a distance threshold d, classifying the retained skeleton lines, and putting the skeleton lines with a distance between the profile centers of ≤d into the same set as the edge skeleton lines of the same rivet; if there are K sets after the classification, it indicates that there are K rivets in the field of view of the camera; then, the least square method is used to fit the profile circle of the skeleton lines in each set to obtain the profile of the corresponding rivet and determine the center of the profile circle.

4. The binocular structured light based rivet bump measurement method for aircrafts according to claim 1, wherein, In step S1, the extracted region of interest centered on the rivet is: E k = {(x, y) | x k -R k -b≤ x≤ x k + R k +b, y k -R k -b≤ y≤ y k + R k +b}, k = 1, 2,..., K. where (x k ,y k ) is the center coordinate of the profile circle of the kth rivet, R k is the radius of the profile circle of the kth rivet, and b is the width based on the profile radius R k , b = 0.5R k .

5. The binocular structured light based rivet bump measurement method for aircrafts according to claim 1, wherein, In step S2, the processing and analysis of the stripe images in the region of interest, the calculation of the phase information and the generation of a phase map refer to the calculation of the phase value only for the sinusoidal stripe and Gray code images in the region of interest extracted in step S1, and the phase map generated only has the phase information in the region of interest, and the phase values outside the region of interest are all set to 0.

6. The binocular structured light based rivet bump measurement method for aircrafts according to claim 1, wherein, In step S3, the rivet head region and the adjacent skin region are segmented according to the rivet profile, which requires that the rivet is detected again by using the rivet profile detection algorithm in step S1, the rivet profile circle is fitted, the center of the profile circle is determined, and the left image is segmented with the profile circle as the boundary, so that the rivet head region is segmented as follows: In the formula, (x′ k_L ,y′ k_L Let R' be the coordinates of the center of the contour circle of the k-th rivet in the corrected left image, and let R' be the radius of the corresponding contour circle. k_L ;R′ ak_L =Round(0.6R′) k_L ), R′ bk_L =Round(0.9R′) k_L ), E′ top_k_L It is concentric with the outline circle and has a radius of R′. ak_L and R′ bk_L A ring-shaped band formed by circles; The adjacent skin region segmented is as follows: In the formula, R′ ck_L =Round(1.1R′) k_L ), R′ dk_L =Round(1.4R′) k_L ), E′ bottom_k_L It is concentric with the outline circle and has a radius of R′. ck_L and R′ dk_L A ring-shaped band formed by circles.

7. The binocular structured light based rivet bump measurement method for aircrafts according to claim 1, wherein, In step S3, the feature points in the matching region are matched, in order to further improve the matching efficiency, the disparity constraint is added on the basis of the epipolar constraint to reduce the search range of the corresponding point, and the process is as follows: S31: Calculate the disparity of the same rivet in the corrected left image and the right image, and the disparity of the kth rivet is as follows: dis(x′ k_L ,y′ k_L )=x′ k_L -x′ kR ,k=1,2,…,K. In the formula, (x′ k_L ,y′ k_L E′ represents the k-th region of interest in the corrected left image. k_L The coordinates of the center of the outline circle of the rivet, (x′ k_R ,y′ k_R E′ represents the k-th region of interest in the corrected right image. k_R The coordinates of the center of the outline circle of the central rivet; S32: search the rivet head region E' according to the corrected phase map top_k_L or the pixel in the adjacent skin region E' bottom_k_L in the right image; the corrected image is subject to epipolar constraint, i.e. the homonymic points in the left and right images are located at the same image height; to further reduce the search range of the homonymic points, the disparity dis(x' k_L ,y' L ) is used for constraint; for any pixel P' L (x' L ,y' R_start ) in the left phase map, search from the pixel P' R_start in the right phase map in the image x-axis direction, the coordinates of P' L are: In the formula, Δdis is a set redundancy value; Pixels meeting the following conditions are put into the set of right pixels to be matched: | At(x', y' ) - At(x', y' ) | < ΔAt L L Rt Rt | At(x', y' ) - At(x', y' ) | < ΔAt​​​ In the formula, At(x′) L ,y′ L ) represents pixel P′ L The phase value, At(x′) Rt ,y′ Rt ) represents the phase value of the t-th pixel searched in the right phase image, and ΔAt is the set threshold. When any of the following criteria is met, the search is ended: (1) |At(x', y' ) - At(x, y) | < ΔAt L L Rt Rt |At(x', y' ) - At(x, y) | > ΔAt​​​ and |At(x′ L ,y′ L ) - At(x′ Rt ,y′ Rt )| > |At(x′ L ,y′ L ) - At(x′ R(t-1) ,y′ R(t-1) )|; (2) or x' Rt W-2, W is the image width; The pixel P' L and its 8-neighbor pixels constitute a reference matching frame, and the pixel P' Rt (x' Rt ,y' Rt ) in the right homonymy point set to be matched is set to be the center of a matching frame containing 8-neighbor pixels, the matching frame to be matched is compared with the reference matching frame, and a phase difference value is calculated according to the following formula: The right pixel to be matched with the smallest phase difference value is screened out as the coarsely selected right corresponding point; S33: selecting the coarse selection right homonym point and each two pixels before and after it to perform cubic spline curve interpolation, and calculating the horizontal coordinate x' of the right homonym point with sub-pixel level accuracy R .

8. The binocular structured light based rivet bump measurement method for aircrafts according to claim 1, wherein, In step S3, the specific implementation of generating the point cloud according to the corresponding point pair lies in that the corresponding point pair has a mapping relationship with the space point cloud, the relevant calibration parameters are obtained by calibrating the binocular camera, the re-projection matrix Q representing the mapping relationship is calculated through the calibration parameters, and the space coordinates of the point cloud data are calculated through the following formula: [X Y Z W] T = Q * [x' y' z' w'] L y' L dis(x' y' z' w') L ,y' L ) 1] T In the formula, dis(x′) L ,y′ L ) represents the disparity of a pair of points with the same name, dis(x′) L ,y′ L )=x′ L -x′ R (X,Y,Z) are the three-dimensional coordinates of the corresponding spatial point cloud.

Images