An industrial robot vision positioning and grasping method for aircraft ribs

Abstract

The present invention discloses an industrial robot visual positioning and grasping method for aircraft ribs, comprising: collecting left and right visual images of aircraft ribs, using a pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib category and rib feature circle in the image and perform mask segmentation of the aircraft rib feature circle; performing contour pixel point separation, skeleton refinement, smoothing and ellipse fitting on the mask to locate the aircraft rib feature ellipse contour and center; performing binocular depth estimation on the aircraft rib feature ellipse contour and center to reconstruct the three-dimensional space point of the rib feature circle in the camera coordinate system, and estimating the space normal vector of the actual rib feature circle; establishing an aircraft rib workpiece coordinate system in the robot base coordinate system, and visually guiding the industrial robot to perform positioning and grasping. The present invention can solve the problems of low accuracy in aircraft rib feature recognition and difficulty in positioning, thereby improving the automation and accuracy of the assembly process.

Description

Technical Field

[0001] The invention belongs to the technical field of industrial robot visual positioning and grasping, and in particular relates to an industrial robot visual positioning and grasping method for aircraft ribs. Background Art

[0002] Aircraft assembly is a core step that determines the efficiency and quality of aircraft manufacturing. The process is complex and the quality requirements are very high. Ribs are key components of aircraft structures, and their precise positioning, grasping and assembly have a vital impact on aircraft performance and safety. Traditional positioning methods often rely on manual operations, which are not only costly and inefficient, but also easily affected by human factors, resulting in insufficient assembly accuracy.

[0003] With the widespread development of industrial robots, their flexibility and high repeatability enable them to achieve automated assembly in complex production environments. Therefore, using industrial robots for automated grasping and assembly of aircraft ribs is an important means of intelligent assembly in the aviation field. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide an industrial robot visual positioning and grasping method for aircraft ribs. Aiming at the visual identification, positioning, grasping and assembly requirements of aircraft ribs, a visual camera is combined with an industrial robot, and high-resolution cameras and advanced image processing algorithms are used to realize visual inspection instance segmentation and spatial positioning of different aircraft ribs, ensuring that the industrial robot can position and grasp the aircraft ribs with high precision and high efficiency for intelligent assembly. This not only reduces the workload of workers and reduces work costs, but also greatly improves the accuracy and efficiency of aircraft assembly, and can solve the problems of low accuracy in feature recognition and difficult positioning of aircraft ribs, thereby improving the degree of automation and accuracy of the assembly process.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] An industrial robot visual positioning and grasping method for aircraft ribs, comprising:

[0007] Step 1: Collect the left and right visual images of the aircraft ribs, use the pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib categories and rib feature circles in the left and right visual images, and perform aircraft rib feature circle mask segmentation;

[0008] Step 2: Perform contour pixel separation, skeleton refinement, smoothing and ellipse fitting on the aircraft rib feature circle mask obtained by segmentation to locate the contour and center of the aircraft rib feature ellipse;

[0009] Step 3: Perform binocular depth estimation on the aircraft rib feature ellipse contour and center of the left and right visual images to reconstruct the 3D space points of the rib feature circle in the camera coordinate system, and estimate the space normal vector of the actual rib feature circle based on the 3D space points of the rib feature circle in the camera coordinate system;

[0010] Step 4: Based on the spatial normal vector of the actual rib feature circle, the aircraft rib workpiece coordinate system is established in the robot base coordinate system, and the industrial robot is visually guided to perform positioning and grasping.

[0011] To optimize the above technical solutions, the specific measures taken also include:

[0012] The pre-training steps of the YOLOV5-SEG detection segmentation model described in step 1 above are as follows:

[0013] The Basler industrial camera is rigidly connected to the end of the industrial robot flange, and the robot end and the aircraft rib are assisted in positioning using a zero-point locator to build an experimental platform;

[0014] Use a circular calibration plate to perform internal calibration of the Basler industrial camera and hand-eye calibration between the camera coordinate system and the robot end coordinate system;

[0015] Drive the industrial robot to ensure that the aircraft ribs are within the visual range of the industrial camera, and continuously adjust the robot end position to collect aircraft rib images, which are used as the aircraft rib dataset;

[0016] The collected aircraft rib data set is feature labeled, and the i-th aircraft rib is labeled as Rib using polygons. i , mark the rib feature circle as ellipse, and use the labeled data set to train the YOLOV5-SEG model.

[0017] The above step 1 uses the pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib category and rib feature circle in the image and perform aircraft rib feature circle mask segmentation, including:

[0018] The actual collected aircraft rib visual image is adjusted to 640 pixels × 640 pixels and then input into the pre-trained YOLOV5-SEG detection and segmentation model. The model extracts the features of the aircraft ribs and rib feature circles, and outputs the predicted target category, target confidence of various candidate boxes and ROI position information. The aircraft ribs and rib feature circle ROIs are identified by labels through non-maximum suppression, and the rib feature circles corresponding to each rib are found by position matching. The aircraft ribs and rib feature circle masks are obtained through mask cropping and binarization.

[0019] In the above step 2, morphological preprocessing is performed on the aircraft rib feature circle mask to separate the contour pixel point set, and then the contour skeleton is refined to remove the interference pixel point set. Finally, the contour pixel point set is regionally smoothed and ellipse fitted to locate the aircraft rib feature ellipse contour and center.

[0020] The above-mentioned morphological preprocessing of the aircraft rib feature circle mask to separate the contour pixel point set includes:

[0021] The rib feature circle mask obtained by segmentation is subjected to Gaussian filtering to remove noise in the mask image. Then, the Canny edge detection algorithm is used to segment the rib feature circle contour and the background noise edge. The edge pixels in the image are connected into edge segments through the eight-connection method. The edge segments in the image are further processed using the contour detection algorithm to find the set of pixels in the image that contain the interference contour and the rib feature circle contour.

[0022] The above-mentioned contour skeleton refinement removes the interfering pixel point set, and finally performs regional smoothing and ellipse fitting on the contour pixel point set to locate the characteristic ellipse contour and center of the aircraft rib plate, including:

[0023] The pixel point set containing the interference contour and the rib characteristic circle contour is removed with the contour point set whose number is less than the threshold N; the contour point set whose ROI area is less than the threshold S in the remaining contour is removed; the rib characteristic circle contour is subjected to skeleton refinement, and a 5×5 pixel area is taken to traverse the entire contour to perform regional least squares straight line approximation smoothing processing; the smoothed rib characteristic circle contour pixel point set is subjected to ellipse fitting and the voting strategy is used to remove duplicates to obtain the final aircraft rib characteristic ellipse contour and circle center.

[0024] The above-mentioned binocular depth estimation of the aircraft rib characteristic ellipse contour and the center of the circle corresponding to the left and right visual images to reconstruct the three-dimensional space point of the rib characteristic circle in the camera coordinate system includes:

[0025] For the characteristic ellipse contour and center of the aircraft rib corresponding to the left and right visual images, the rib feature points are estimated in depth using the angle error minimization method to reconstruct the three-dimensional space points of the rib feature circle in the camera coordinate system; the angle error minimization method is as follows:

[0026] The three-dimensional coordinate point of the characteristic circle center of the rib in the world coordinate system is regarded as the theoretical point P of the target to be determined. Under the observation of the two optical axis centers O1 and O2 of the binocular camera, P2=RP1+t, where R and t are the rotation matrix and translation matrix of the camera O1 to O2 respectively; P i Corresponding to camera O i Point P below;

[0027] Assume that camera O i The calibration matrix is ​​Ki , then the normalized image coordinates f i =K i -1 u i , where u i is point P on camera O i The homogeneous pixel coordinate vector under ;

[0028] The optical axis rays f1 and f2 of the two cameras are corrected to f1' and f2', so that they satisfy f2'·(t×Rf1')=0. The angle reprojection error is defined as:

[0029] α i =∠(f i ,f i ')=∠(K -1 u i ,K -1 u i ')i=1,2 (1)

[0030] Therefore, the minimized objective function is:

[0031]

[0032] Solving equation (2) obtains the optimal homogeneous pixel coordinate vectors f1' and f2', and then obtains the optimal target estimation points P1' and P2' in the camera O1 and O2 coordinate systems:

[0033]

[0034] Where β = f2' × Rf1'.

[0035] After obtaining the optimal target estimation points P1' and P2' in the camera O1 and O2 coordinate systems, the three-dimensional space points of the rib characteristic circle in the camera coordinate system can be obtained.

[0036] The above step 3 estimates the spatial normal vector of the actual rib characteristic circle according to the three-dimensional spatial points of the rib characteristic circle in the camera coordinate system, including:

[0037] First, three points are randomly selected from the three-dimensional space point set of the rib characteristic circle in the camera coordinate system, and the SVD decomposition method is used to fit the space circle. The normal error T = (A, B, C) of the plane where the two points are located is obtained, and the center of the space circle O c (x0, y0, z0) and radius R. Next, find the distance from all points in the point set to the plane and the difference between the distance to the center of the circle and the radius, which are denoted by M i ,N i ,i=1,2,...,n, then:

[0038] M i =|Axi +By i +Cz i -K| (4)

[0039]

[0040] Set the distance threshold M from the point to the plane th And the difference threshold N between the distance from the point to the center of the circle and the radius th ;

[0041] Traversing all points in the point set will satisfy M at the same time i <M th , N i <N th The points that meet the requirements are determined as in-game points and retained, and the points that do not meet the requirements are determined as out-game points and removed;

[0042] If the number of points retained in the game finally reaches more than 90% of the total number, the point set is retained; otherwise, 3 points are reselected to perform the above calculation;

[0043] Finally, after finding the point set that meets the total number requirement, the retained points are used to perform spatial circle fitting, extract feature parameters, and obtain the spatial point parameter set of the center of the circular outer contour of the large end of the conical pin in the camera base coordinate system, which is denoted as O j =(x 0j ,y 0j ,z 0j ,A j ,B j ,C j ,R j ),j=1,2, then the normal vector of the circular plane is Therefore, the aircraft rib attitude unit normal vector It is expressed as the mean of the unit normal vectors of the two characteristic circles of the rib:

[0044]

[0045] The above-mentioned step 4 establishes the aircraft rib workpiece coordinate system in the robot base coordinate system based on the space normal vector of the actual rib characteristic circle, including:

[0046] Take the mean of the spatial coordinates of the centers of the two characteristic circles of the same aircraft rib as the origin O of the rib workpiece coordinate system c , the direction vector of the line connecting the two circle centers is used as the X direction of the workpiece coordinate system, and the mean value of the spatial normal vectors of the two rib feature circles is used as the Z direction of the rib workpiece coordinate system to establish the aircraft rib workpiece coordinate system in the camera coordinate system;

[0047] The homogeneous transformation matrix between the camera coordinate system and the robot end coordinate system, and the homogeneous transformation matrix between the robot base coordinate system and the aircraft rib workpiece coordinate system obtained by hand-eye calibration are used to calculate the transformation relationship between the aircraft rib workpiece coordinate system in the robot base coordinate system.

[0048] The present invention has the following beneficial effects:

[0049] The present invention firstly adopts the YOLOV5-SEG detection and segmentation model to remove a large amount of complex background noise, realizes the detection and recognition of aircraft ribs and the instance segmentation of rib feature circles; secondly, the rib feature circle mask obtained by segmentation is preprocessed through morphological operations such as filtering and edge detection, and the preprocessed mask is further subjected to ellipse detection and circle center positioning based on the theoretical geometric features of the ribs; then, based on the angle error minimization method, the identified characteristic ellipse contour and circle center are subjected to binocular depth estimation to reconstruct the three-dimensional spatial point of the rib feature circle in the camera coordinate system; the spatial normal vector of the actual rib feature circle is estimated based on the acquired three-dimensional point cloud based on the RANSAC method; finally, the aircraft rib workpiece coordinate system can be established by combining the spatial coordinates of the rib feature circle and its normal vector, thereby guiding the robot to perform positioning and grasping, thereby solving the problems of positioning difficulty and low feature recognition accuracy in the process of intelligent assembly of aircraft ribs by industrial robots;

[0050] The present invention adopts the YOLOV5-SEG detection and segmentation model to realize the detection and identification of aircraft ribs and the instance segmentation of rib feature circles; the rib feature circle mask preprocessing obtained by segmentation through morphological operations such as skeleton refinement, Gaussian filtering, edge detection, etc. realizes rib feature circle detection and circle center positioning; the present invention performs binocular three-dimensional reconstruction of the identified characteristic ellipse contour and circle center based on the angle error minimization method, and calculates the three-dimensional spatial points of the rib feature circle in the camera coordinate system with high precision and high efficiency; the present invention estimates the spatial normal vector of the actual rib feature circle; and establishes the aircraft rib workpiece coordinate system, thereby realizing the high-precision positioning and grasping of the aircraft rib by the industrial robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 It is to collect images of aircraft ribs in different environments, different postures, and different categories;

[0053] Figure 3 This is a flow chart of the aircraft rib recognition and feature segmentation method based on YOLOV5-SEG;

[0054] Figure 4 This is the result diagram of the characteristic circle center and contour positioning of the aircraft rib;

[0055] Figure 5It is a schematic diagram of the characteristic circle center and contour of the aircraft rib plate reconstructed by the industrial camera depth estimation;

[0056] Figure 6 This is the flow chart of the industrial robot actually positioning and grabbing the aircraft ribs. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0059] The present invention discloses an industrial robot visual positioning and grasping method for aircraft ribs, which solves the problems of difficult positioning and low feature recognition accuracy in the process of intelligent assembly of aircraft ribs by industrial robots. The algorithm first adopts the YOLOV5-SEG detection and segmentation model to remove a large amount of complex background noise, and realizes the detection and recognition of aircraft ribs and the instance segmentation of rib feature circles; secondly, the rib feature circle mask obtained by segmentation is preprocessed by morphological operations such as filtering and edge detection, and the preprocessed mask is further subjected to ellipse detection and circle center positioning based on the theoretical geometric features of the ribs; then, based on the angle error minimization method, the recognized characteristic ellipse contour and circle center are subjected to binocular depth estimation to reconstruct the three-dimensional spatial point of the rib feature circle in the camera coordinate system; the spatial normal vector of the actual rib feature circle is estimated based on the RANSAC method according to the obtained three-dimensional point cloud; finally, the aircraft rib workpiece coordinate system can be established by combining the spatial coordinates of the rib feature circle and its normal vector, so as to guide the robot to perform positioning and grasping. For details, see Figure 1 , the method comprising:

[0060] Step 1: Collect the left and right visual images of the aircraft ribs, use the pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib categories and rib feature circles in the left and right visual images, and perform aircraft rib feature circle mask segmentation;

[0061] Step 2: Perform contour pixel separation, skeleton refinement, smoothing and ellipse fitting on the aircraft rib feature circle mask obtained by segmentation to locate the contour and center of the aircraft rib feature ellipse;

[0062] Step 3: Perform binocular depth estimation on the aircraft rib feature ellipse contour and center of the left and right visual images to reconstruct the 3D space points of the rib feature circle in the camera coordinate system, and estimate the space normal vector of the actual rib feature circle based on the 3D space points of the rib feature circle in the camera coordinate system;

[0063] Step 4: Based on the spatial normal vector of the actual rib feature circle, the aircraft rib workpiece coordinate system is established in the robot base coordinate system, and the industrial robot is visually guided to perform positioning and grasping.

[0064] In the embodiment, step 1 rigidly connects the Basler industrial camera to the end of the industrial robot flange, and uses a zero-point locator to assist in positioning the robot end and the aircraft rib to build an experimental platform. Drive the robot to the visual range of the aircraft rib, collect images, and use the YOLOV5-SEG detection and segmentation model to visually identify the aircraft rib category and the rib feature circle, and perform mask segmentation of the aircraft rib feature circle;

[0065] Specifically, the circular calibration plate is first used to calibrate the internal parameters of the Basler industrial camera and the hand-eye calibration between the camera coordinate system and the robot end coordinate system. The industrial robot is driven to ensure that the aircraft rib is within the visual range of the industrial camera, and the robot end position is continuously adjusted to collect the aircraft rib image, which is used as a data set, such as Figure 2 As shown in the figure, the collected aircraft rib plate data set is labeled with features and the Rib i The rib of the aircraft is recorded as the i-th type, and ellipse is recorded as the characteristic circle of the rib. Both are annotated with polygons. The annotated data set is used to train the YOLOV5-SEG model.

[0066] The actual collected aircraft rib image is adjusted to 640 pixels × 640 pixels and then input into the YOLOV5-SEG model feature extraction fusion network to extract the important features of the aircraft rib and rib feature circle, and output the predicted target category, target confidence of various candidate frames and ROI position information; the module outputs the image segmentation mask of the aircraft rib and rib feature circle; the aircraft rib and rib feature circle ROI are identified by label through non-maximum suppression, and the rib feature circle corresponding to each rib is found by position matching. The aircraft rib and rib feature circle mask are obtained through mask cropping and binarization. The method flow is as follows: Figure 3 shown.

[0067] In the embodiment, step 2 performs morphological preprocessing on the aircraft rib feature circle mask to separate the contour pixel point set, then performs contour skeleton refinement to remove the interference pixel point set, and finally performs regional filtering and smoothing processing on the contour pixel point set to perform ellipse fitting to achieve the positioning of the aircraft rib in the camera coordinate system;

[0068] Specifically, the rib feature circle mask obtained by segmentation is subjected to Gaussian filtering to remove noise in the mask image. Then, the Canny edge detection algorithm is used to accurately segment the rib feature circle contour and the background noise edge. The edge pixels in the image are connected into edge segments through the eight-connection method. The contour detection algorithm is used to further process the edge segments in the image to find the set of pixel points in the image that contain the interference contour and the rib feature circle contour.

[0069] Then, the pixel point set containing the interference contour and the rib plate characteristic circle contour is further accurately extracted. First, the contour point set with a number less than the threshold N is removed; the contour point set with an ROI area less than the threshold S in the remaining contour is removed; the rib plate characteristic circle contour is subjected to skeleton refinement, and a small 5×5 pixel area is taken to traverse the entire contour for regional least squares linear approximation smoothing. The processed rib plate characteristic circle contour pixel point set is subjected to ellipse fitting and the voting strategy is used to remove duplicates to obtain the final rib plate characteristic circle center and contour, as shown in Figure 4 shown.

[0070] In the embodiment, step 3 performs binocular depth estimation on the identified and located aircraft rib characteristic elliptical contour and circle center to reconstruct the three-dimensional space point of the rib characteristic circle in the camera coordinate system, and estimates the space normal vector of the actual rib characteristic circle;

[0071] Specifically, the above steps are performed on the left and right views captured by the industrial camera to obtain the center and contour of the rib feature circle, and the angle error minimization method is used to estimate the depth of the rib feature points based on the triangulation principle, and the spatial point set in the camera coordinate system is reconstructed in three dimensions. The schematic diagram is shown in FIG. Figure 5 As shown. The three-dimensional coordinate point of the center point of the characteristic circle of the aircraft rib located in step 2 in the camera coordinate system is regarded as the theoretical point P of the target to be determined. Under the observation of the two optical axis centers O1 and O2 of the binocular camera, P2=RP1+t, where R and t are the rotation matrix and translation matrix of the camera O1 to O2 respectively; P i Corresponding to camera O i Assume that the camera O i The calibration matrix is ​​K i , then the normalized image coordinates f i =K i -1 u i , where u i is point P on camera O i Homogeneous pixel coordinate vector under. Due to the systematic error caused by camera calibration distortion and the measurement error caused by image noise, the optical axis rays of the two cameras may not intersect. In order to satisfy the epipolar constraint, the two optical axis rays f1 and f2 must be corrected to f1' and f2' so that they satisfy f2'·(t×Rf1')=0. The angular reprojection error can be defined as:

[0072] α i =∠(f i ,f i ')=∠(K -1 u i ,K -1 u i ')i=1,2 (1)

[0073] Therefore, the minimized objective function is:

[0074]

[0075] Solving equation (2) can obtain the optimal homogeneous pixel coordinate vectors f1' and f2', and thus the optimal target estimation points P1' and P2' in the camera O1 and O2 coordinate systems can be obtained:

[0076]

[0077] Where β = f2' × Rf1'. After obtaining the optimal target estimation points P1' and P2' in the camera O1 and O2 coordinate systems, the three-dimensional space points of the rib characteristic circle in the camera coordinate system can be obtained.

[0078] The RANSAC method is used to estimate the spatial normal vector of the actual rib characteristic circle for the spatial points of the rib characteristic circle contour in the reconstructed camera coordinate system. Three points are randomly selected from the point cloud after depth estimation, and the SVD decomposition method is used to fit the spatial circle. The normal loss T = (A, B, C) of the plane where the two points are located is obtained, and the center of the spatial circle O c (x0, y0, z0) and radius R. Next, find the distance of all points in the point set to the plane and the difference between the distance to the center of the circle and the radius, denoted by M i ,N i ,i=1,2,...,n, then:

[0079] M i =|Ax i +By i +Cz i -K| (4)

[0080]

[0081] Set the distance threshold M from the point to the plane th And the difference threshold N between the distance from the point to the center of the circle and the radius th . Traversing all points in the point set will satisfy M at the same time i <M th , N i <N thThe points that meet the requirements are determined as internal points and retained, and the points that do not meet the requirements are determined as external points and removed. If the number of internal points retained in the end reaches more than 90% of the total, the point set is retained; if not, 3 points are reselected for the above calculation. Finally, after finding a point set that meets the requirements, the retained points are used to perform spatial circle fitting and extract feature parameters. In summary, the spatial point parameter set of the center of the circular outer contour of the large end of the conical pin in the camera base coordinate system can be obtained, which is denoted as O j =(x 0j ,y 0j ,z 0j ,A j ,B j ,C j ,R j ),j=1,2, then the normal vector of the circular plane is Therefore, the aircraft rib attitude unit normal vector It can be expressed as the mean of the unit normal vectors of the two characteristic circles of the rib:

[0082]

[0083] In the embodiment, step 4 establishes the aircraft rib workpiece coordinate system in the robot base coordinate system, and visually guides the industrial robot to perform positioning and grasping.

[0084] Specifically, the mean of the spatial coordinates of the centers of the two characteristic circles of the same aircraft rib is taken as the origin O of the rib workpiece coordinate system. c The direction vector of the line connecting the two circle centers is used as the X direction of the workpiece coordinate system, and the average of the normal vectors of the two rib feature circles is used as the Z direction of the rib workpiece coordinate system to establish the aircraft rib workpiece coordinate system in the camera coordinate system. The conversion relationship of the aircraft rib workpiece coordinate system in the robot base coordinate system can be calculated by using the homogeneous conversion matrix between the camera coordinate system and the robot end coordinate system obtained by hand-eye calibration, and the homogeneous conversion matrix between the robot base coordinate system and the aircraft rib workpiece coordinate system.

[0085] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0086] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An industrial robot visual positioning and grasping method for aircraft ribs, characterized in that: include: Step 1: Collect the left and right visual images of the aircraft ribs, use the pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib categories and rib feature circles in the left and right visual images, and perform aircraft rib feature circle mask segmentation; The pre-training steps of the YOLOV5-SEG detection segmentation model are as follows: The Basler industrial camera is rigidly connected to the end of the industrial robot flange, and the robot end and the aircraft rib are assisted in positioning using a zero-point locator to build an experimental platform; Use a circular calibration plate to perform internal calibration of the Basler industrial camera and hand-eye calibration between the camera coordinate system and the robot end coordinate system; Drive the industrial robot to ensure that the aircraft ribs are within the visual range of the industrial camera, and continuously adjust the robot end position to collect aircraft rib images, which are used as the aircraft rib dataset; The collected aircraft rib dataset is feature labeled and polygons are used to mark the first The ribs of this type of aircraft are marked as , mark the rib feature circle as , use the labeled data set to train the YOLOV5-SEG model; The method of using the pre-trained YOLOV5-SEG detection and segmentation model to identify the aircraft rib category and the rib feature circle in the image and perform mask segmentation of the aircraft rib feature circle includes: The actual collected aircraft rib visual image is adjusted to 640 pixels × 640 pixels and then input into the pre-trained YOLOV5-SEG detection and segmentation model. The model extracts the features of the aircraft ribs and rib feature circles, and outputs the predicted target category, target confidence of various candidate frames, and ROI position information; the aircraft ribs and rib feature circle ROIs are identified by labels through non-maximum suppression, and the rib feature circles corresponding to each rib are found through position matching. The aircraft ribs and rib feature circle masks are obtained through mask cropping and binarization processing; Step 2: Perform contour pixel separation, skeleton refinement, smoothing and ellipse fitting on the aircraft rib feature circle mask obtained by segmentation to locate the contour and center of the aircraft rib feature ellipse; Step 3: Perform binocular depth estimation on the aircraft rib feature ellipse contour and center of the left and right visual images to reconstruct the 3D space points of the rib feature circle in the camera coordinate system, and estimate the space normal vector of the actual rib feature circle based on the 3D space points of the rib feature circle in the camera coordinate system; Step 4: Based on the spatial normal vector of the actual rib feature circle, the aircraft rib workpiece coordinate system is established in the robot base coordinate system, and the industrial robot is visually guided to perform positioning and grasping.

2. The method for grasping an aircraft rib by an industrial robot visual positioning according to claim 1, characterized in that: In step 2, the aircraft rib feature circle mask is subjected to morphological preprocessing to separate the contour pixel point set, and then the contour skeleton is refined to remove the interference pixel point set, and finally the contour pixel point set is subjected to regional smoothing and ellipse fitting to locate the aircraft rib feature ellipse contour and circle center.

3. The method for grasping an aircraft rib by an industrial robot visual positioning according to claim 2, characterized in that: The step of performing morphological preprocessing on the aircraft rib feature circle mask to separate the contour pixel point set includes: The rib feature circle mask obtained by segmentation is subjected to Gaussian filtering to remove noise in the mask image. Then, the Canny edge detection algorithm is used to segment the rib feature circle contour and the background noise edge. The edge pixels in the image are connected into edge segments through the eight-connection method. The edge segments in the image are further processed using the contour detection algorithm to find the set of pixels in the image that contain the interference contour and the rib feature circle contour.

4. The method for grasping an aircraft rib by an industrial robot visual positioning according to claim 2, characterized in that: The process of performing contour skeleton refinement to remove interfering pixel point sets, and finally performing regional smoothing and ellipse fitting on the contour pixel point sets to locate the characteristic ellipse contour and center of the aircraft rib plate includes: The pixel point set containing the interference contour and the rib characteristic circle contour is removed with the contour point set whose number is less than the threshold N; the contour point set whose ROI area is less than the threshold S in the remaining contour is removed; the rib characteristic circle contour is subjected to skeleton refinement, and a 5×5 pixel area is taken to traverse the entire contour to perform regional least squares straight line approximation smoothing processing; the smoothed rib characteristic circle contour pixel point set is subjected to ellipse fitting and the voting strategy is used to remove duplicates to obtain the final aircraft rib characteristic ellipse contour and circle center.

5. The method for industrial robot visual positioning and grasping of aircraft ribs according to claim 1, characterized in that: Step 3 performs binocular depth estimation on the aircraft rib feature ellipse contour and the center of the circle corresponding to the left and right visual images to reconstruct the three-dimensional space point of the rib feature circle in the camera coordinate system, including: For the characteristic ellipse contour and center of the aircraft rib corresponding to the left and right visual images, the rib feature points are estimated in depth using the angle error minimization method to reconstruct the three-dimensional space points of the rib feature circle in the camera coordinate system; the angle error minimization method is as follows: The three-dimensional coordinate point of the characteristic circle center of the rib in the world coordinate system is regarded as the target theoretical point to be determined. , at the center of the two optical axes of the binocular camera and Under observation, there are ,in , Camera arrive The rotation matrix and translation matrix of Corresponding to the camera The next point ; Assume the camera The calibration matrix is , then the normalized coordinates of the image are ,in For point On Camera The homogeneous pixel coordinate vector under ; Two camera optical axis rays and Corrected to and , so that it satisfies , the angular reprojection error is defined as: （1） Therefore, the minimized objective function is: （2） Solve equation (2) to obtain the optimal homogeneous pixel coordinate vector and , and then find the and The optimal target estimation point in the coordinate system and : （3） The parameters ; Get the camera and The optimal target estimation point in the coordinate system and After that, the three-dimensional space point of the rib characteristic circle in the camera coordinate system can be obtained.

6. The method for industrial robot visual positioning and grasping of aircraft ribs according to claim 1, characterized in that: Step 3, estimating the spatial normal vector of the actual rib characteristic circle according to the three-dimensional spatial points of the rib characteristic circle in the camera coordinate system, includes: Firstly, three points are randomly selected from the three-dimensional space point set of the rib characteristic circle in the camera coordinate system, and the SVD decomposition method is used to fit the space circle to obtain the normal error of the plane where the two points are located. , the center of the space circle And the radius Three parameters, secondly, find the distance of all points in the point set to the plane and the difference between the distance to the center of the circle and the radius, respectively , then: （4） （5） Set the distance threshold from point to plane And the difference threshold between the distance from the point to the center of the circle and the radius ; Traversing all points in the point set will satisfy , The points that meet the requirements are determined as in-game points and retained, and the points that do not meet the requirements are determined as out-game points and removed; If the number of points retained in the game finally reaches more than 90% of the total number, the point set is retained; otherwise, three points are reselected to perform the above calculation; Finally, after finding the point set that meets the total number requirement, the retained points are used to perform spatial circle fitting, extract feature parameters, and obtain the spatial point parameter set of the center of the circular outer contour of the large end of the conical pin in the camera base coordinate system, which is recorded as , then the normal vector of the circular plane is , so the aircraft rib attitude unit normal vector It is expressed as the mean of the unit normal vectors of the two characteristic circles of the rib: （6）。 7. The method for grasping an aircraft rib by an industrial robot visual positioning according to claim 1, characterized in that: Step 4, based on the space normal vector of the actual rib feature circle, establishes the aircraft rib workpiece coordinate system in the robot base coordinate system, including: Take the mean of the spatial coordinates of the centers of the two characteristic circles of the same aircraft rib as the origin of the rib workpiece coordinate system , the direction vector of the line connecting the two centers is used as the workpiece coordinate system The mean of the space normal vectors of the two rib feature circles is used as the rib workpiece coordinate system. To establish the aircraft rib workpiece coordinate system under the camera coordinate system; The homogeneous transformation matrix between the camera coordinate system and the robot end coordinate system, and the homogeneous transformation matrix between the robot base coordinate system and the aircraft rib workpiece coordinate system obtained by hand-eye calibration are used to calculate the transformation relationship between the aircraft rib workpiece coordinate system in the robot base coordinate system.

Images