Reference hole pose measurement system and method based on monocular double-line laser

Abstract

The invention relates to a reference hole pose measurement system and method based on monocular double-line laser, the system comprises a monocular camera, two line laser generators, a checkerboard calibration plate, a robot and an upper computer, and the upper computer executes the following processes: controlling the monocular camera to shoot and calibrate an image, performing distortion correction on the image, and then performing calibration on the image; the two-dimensional coordinates of the four feature points and the two sets of laser center lines are extracted, the two-dimensional coordinates are converted into three-dimensional coordinates under a robot coordinate system, the corresponding circle center coordinates and the circle center normal direction are obtained according to the three-dimensional coordinates under the robot coordinate system, the position and posture of the reference hole are determined, and the process is achieved through the method. The monocular camera is matched with the two line lasers to complete three-dimensional coordinate positioning and circle center normal calculation of the hole center of the reference hole, the three-dimensional position of the reference hole can be accurately obtained, the theoretical hole position and the normal direction of the reference hole are compared to calculate the deviation, the hole to be machined is compensated, and the hole forming quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of reference hole pose measurement, and particularly to a reference hole pose measurement system and method based on monocular double-line laser. Background Art

[0002] During the production cycle of an aircraft, the assembly of the aircraft is a key link, and its assembly connection method is mainly riveting. There are installation holes for connection fasteners of different specifications and functions during the assembly process of an aircraft, with the number exceeding one million, and the hole-making takes a long time. At the same time, the hole-making accuracy requirements for aviation connection fastener installation holes are relatively high, which directly affects the overall structural strength of the aircraft and the fatigue life of the connection. Therefore, the hole-making quality is crucial.

[0003] The invention patent with the publication number CN113203369B discloses a method for measuring the normal perpendicularity of robot hole-making. By establishing a hole-making plane model on the hole-making surface and a hole-making normal path model according to the movement path of the drilling actuator, measuring the angle between the hole-making plane model and the hole-making normal path model, and then obtaining the normal perpendicularity of the robot system for hole-making, the detection efficiency and detection accuracy of the normal perpendicularity of robot hole-making are improved; this patent obtains the hole-making point normal by fitting the coordinates of at least three non-collinear plane measurement points to obtain the hole-making plane model, and the measurement method of the normal is relatively complex and cannot take into account both the measurement position and the measurement normal.

[0004] Therefore, providing a method that can both obtain the three-dimensional coordinate position of the reference hole and measure the normal of the reference hole is an urgent problem to be solved at present. Summary of the Invention

[0005] The purpose of the present invention is to provide a reference hole pose measurement system and method based on monocular double-line laser to overcome the defects existing in the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] According to one aspect of the present invention, a reference hole pose measurement system based on monocular double-line laser is provided. The system includes a monocular camera, two line laser generators, a checkerboard calibration board, a robot, and a host computer. The monocular camera, the line laser generator, and the robot are respectively communicatively connected to the host computer. The laser center line emitted by the line laser generator is on the checkerboard calibration board. The robot is connected to the monocular camera. The host computer performs the following processes:

[0008] The monocular camera is controlled to capture images and calibrate them. After the image is distorted, the two-dimensional coordinates of four feature points and two sets of laser center lines are extracted. The two-dimensional coordinates are converted into three-dimensional coordinates in the robot coordinate system. According to the three-dimensional coordinates in the robot coordinate system, the corresponding center coordinates and center normal are obtained respectively, and the reference hole position is determined.

[0009] According to another aspect of the present invention, a method for measuring the position and posture of a reference hole based on a monocular two-line laser as described above is provided, wherein the method specifically comprises:

[0010] S1. Use a camera to capture an image and use Zhang Zhengyou calibration method to calibrate the camera and obtain internal parameters and external parameters, perform light plane calibration and obtain light plane equations, and perform hand-eye calibration and obtain a hand-eye transformation matrix;

[0011] S2, correcting the image distortion according to internal parameters;

[0012] S3, performing Hough circle detection after the distortion correction, then binarizing the image and extracting the laser center line;

[0013] S4, according to the laser center line detected and extracted by the Hough circle, four feature points and two sets of two-dimensional coordinates of the laser center line are obtained;

[0014] S5. Obtain the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system according to the two-dimensional coordinates of the four feature points and the two groups of laser center lines and the light plane equation; convert the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system into the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the robot base coordinate system according to the hand-eye transformation matrix;

[0015] S6. Perform circle fitting according to the three-dimensional coordinates of the four feature points in the robot base coordinate system to obtain the three-dimensional coordinates of the center of the circle; perform curve fitting according to the three-dimensional coordinates of the two sets of laser center lines in the robot base coordinate system and calculate the intersection of the two curves and the tangent vector at the intersection, thereby calculating the normal direction of the center of the circle;

[0016] S7. Determine the reference hole position according to the circle center coordinates and the circle center normal.

[0017] As a preferred technical solution, the camera taking images in S1 specifically includes:

[0018] S111, drag the robot so that the intersection of the two groups of laser center lines is in the reference hole and the reference hole is in the camera field of view, and take an image of the reference hole.

[0019] As a preferred technical solution, the light plane equation obtained in S1 specifically includes:

[0020] S121, projecting two sets of laser center lines onto a checkerboard calibration plate, and using a camera to capture images of the checkerboard calibration plate with the laser center lines from different angles;

[0021] S122, extracting the coordinates of the intersection of the laser center line and the checkerboard calibration plate, and the coordinates of the marking points on the checkerboard calibration plate from the acquired image;

[0022] S123, establishing a laser line model according to the intersection point coordinates, and establishing a calibration plate model according to the marking point coordinates;

[0023] S124. Solve the light plane equation according to the laser line model and the calibration plate model.

[0024] As a preferred technical solution, the hand-eye transformation matrix obtained in S1 specifically includes:

[0025] S131, fix the position of the checkerboard calibration plate relative to the robot coordinate system, and control the robot to carry a camera to shoot the checkerboard calibration plate, obtain the posture relationship of the checkerboard calibration plate relative to the camera, the posture relationship of the robot end relative to the robot base coordinate system, and the posture relationship of the robot base coordinate system relative to the calibration plate;

[0026] S132. Calculate a hand-eye transformation matrix according to the posture relationship.

[0027] As a preferred technical solution, the Hough circle detection in S3 specifically includes:

[0028] S311, using Hough circle detection to detect the reference hole in the image, and finding the two-dimensional coordinates and radius of the center of the reference hole.

[0029] As a preferred technical solution, the laser centerline extraction in S3 specifically includes:

[0030] S321, set a grayscale threshold, perform binarization on the image, assign a value of 255 to pixels greater than the grayscale threshold, and assign a value of 0 to pixels less than the grayscale threshold;

[0031] S322, extracting the laser center line of the binarized image, using the Steger method for extracting the center of the light stripe to extract the pixel points of the laser center line, and obtaining the two-dimensional coordinates of the laser center line.

[0032] As a preferred technical solution, S4 specifically includes:

[0033] S41. Obtain the two-dimensional coordinates and radius of the circle center according to the Hough circle detection, obtain the two-dimensional coordinates of the laser center line according to the extracted laser center line, divide the laser center line into two groups according to the circle center and the radius and obtain four feature points.

[0034] As a preferred technical solution, the four feature points obtained in S41 specifically include:

[0035] S411, calculating the two-dimensional coordinate distance from the extracted two-dimensional coordinate of the laser center line to the center of the reference hole, where the distance is recorded as D;

[0036] S412, based on the distance D being equal to the radius of the reference hole, four pixel points on the contour of the reference hole are selected from the two-dimensional coordinates of the extracted laser center line, that is, four feature points are obtained.

[0037] As a preferred technical solution, the center normal in S6 specifically includes:

[0038] S61, fitting the parametric equations of the two curves using a quadratic polynomial according to the three-dimensional coordinates of the centers of the two sets of laser lines;

[0039] S62. Calculate the intersection of the two curves and calculate the tangent vector at the intersection according to the parametric equation. Perform a cross product operation on the tangent vector to obtain the normal at the intersection and obtain the normal to the center of the circle.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention uses a monocular camera in conjunction with two line lasers to obtain corresponding two-dimensional coordinates, and after converting them into corresponding three-dimensional coordinates, completes the three-dimensional coordinate positioning of the center of the reference hole and the calculation of the normal of the center of the circle, and can accurately obtain the three-dimensional position of the reference hole, compare the theoretical hole position and normal of the reference hole to calculate the deviation, compensate for the hole to be processed, and improve the quality of hole making.

[0042] 2. The present invention uses the Hough circle detection technology to accurately identify the center and radius of the reference hole, and can maintain high recognition accuracy even under complex backgrounds and noise interference.

[0043] 3. The present invention uses camera calibration technology to correct image distortion, ensuring that the acquired image data truly reflects the actual scene, thereby improving the accuracy of hole positioning.

[0044] 4. The present invention automates the entire process from image acquisition and processing to hole position determination, greatly shortening the processing time; at the same time, it has fast-response real-time processing capabilities, and can complete reference hole positioning and normal measurement in a short time, meeting the high efficiency requirements of aviation manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a practical diagram of the overall structure of the present invention;

[0046] Figure 2 It is a schematic diagram of the calculation process of the present invention;

[0047] Figure 3 The circle center normal calculation principle diagram of the present invention;

[0048] 1. Monocular camera; 2. Lens; 3. First-line laser generator; 4. Second-line laser generator; 5. Reference hole. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] During the production cycle of an aircraft, the assembly of the aircraft is a key link, and the main assembly connection method is riveting. During the assembly process of an aircraft, there are more than a million holes for fasteners of different specifications and functions, and the hole making process takes a long time. At the same time, the hole making accuracy requirements for the installation holes of aviation fasteners are relatively high, which directly affects the overall structural strength of the aircraft and the fatigue life of the joints.

[0051] When using automated equipment to make holes, in order to ensure the quality of the holes, reference holes are set on the workpiece in advance. The actual hole position and normal of the reference hole are measured and compared with the theoretical hole position and normal of the reference hole in the product digital model to calculate the deviation, and compensate for the hole to be processed to improve the quality of the hole.

[0052] The present invention provides a reference hole position measurement system and method based on a monocular dual-line laser. The present invention uses a monocular camera in conjunction with two line lasers to obtain corresponding two-dimensional coordinates, and after converting them into corresponding three-dimensional coordinates, completes the three-dimensional coordinate positioning of the center of the reference hole and the calculation of the normal direction of the center of the circle, and can accurately obtain the three-dimensional position of the reference hole, compare the theoretical hole position and normal direction of the reference hole to calculate the deviation, compensate for the hole to be processed, and improve the quality of hole making. The present invention can accurately identify the center and radius of the reference hole through the Hough circle detection technology, and can maintain high recognition accuracy even under complex backgrounds and noise interference. The present invention uses camera calibration technology to correct image distortion to ensure that the acquired image data truly reflects the actual scene, thereby improving the accuracy of hole positioning. The present invention realizes automation of the entire process from image acquisition, processing to hole position determination, greatly shortening the processing time; at the same time, it has a fast-response real-time processing capability, and can complete the reference hole positioning and normal measurement in a short time, meeting the demand for high efficiency in aviation manufacturing.

[0053] Example 1

[0054] like Figure 1As shown, a reference hole position measurement system based on monocular dual-line laser, the system includes a monocular camera 1, two line laser generators, a checkerboard calibration board, a robot and a host computer, the monocular camera 1, the line laser generator and the robot are respectively connected to the host computer for communication, the line laser generator emits a laser center line on the checkerboard calibration board, the robot is connected to the monocular camera, and the host computer performs the following process:

[0055] The monocular camera 1 is controlled to capture images and calibrate them. After the image is distorted, the two-dimensional coordinates of four feature points and two groups of laser center lines are extracted, and the two-dimensional coordinates are converted into three-dimensional coordinates in the robot coordinate system. According to the three-dimensional coordinates in the robot coordinate system, the corresponding center coordinates and center normal are obtained respectively, and the position and posture of the reference hole 5 are determined.

[0056] In this embodiment, the system further includes a lens 2 , and the monocular camera 1 obtains an image through the lens 2 .

[0057] Monocular camera 1: obtains the reference hole 5 and laser stripe images and is installed on the hole-making end effector. The images taken by the camera installed on the end effector may be distorted, affecting recognition, and distortion correction is required according to the algorithm.

[0058] Line laser generator: It is composed of two line laser generators, namely the first line laser generator 3 and the second line laser generator 4, which are installed on the end effector at a certain angle. The intersection of the two line lasers emitted by the two laser generators is located on the axis of the camera optical axis.

[0059] The robot carries a monocular camera 1 and aims it at the reference hole 5 . When the intersection of the two line lasers is located in the reference hole 5 , the field of view of the monocular camera 1 covers the reference hole 5 .

[0060] Line laser stripes impose feature points on the workpiece surface, and the pixel coordinates on the center line are extracted according to the laser center line extraction algorithm. The pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system according to the monocular line laser principle.

[0061] Host computer: The host computer contains the monocular line laser algorithm and is used to control the camera shooting.

[0062] Example 2

[0063] like Figure 2 and Figure 3 As shown, a method for measuring the position and posture of a reference hole based on a monocular two-line laser, the method specifically includes:

[0064] S1. Use a camera to capture an image and use Zhang Zhengyou calibration method to calibrate the camera and obtain internal parameters and external parameters, perform light plane calibration and obtain light plane equations, and perform hand-eye calibration and obtain a hand-eye transformation matrix;

[0065] S2, correcting the image distortion according to internal parameters;

[0066] S3, performing Hough circle detection after the distortion correction, then binarizing the image and extracting the laser center line;

[0067] S4, according to the laser center line detected and extracted by the Hough circle, four feature points and two sets of two-dimensional coordinates of the laser center line are obtained;

[0068] S5. Obtain the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system according to the two-dimensional coordinates of the four feature points and the two groups of laser center lines and the light plane equation; convert the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system into the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the robot base coordinate system according to the hand-eye transformation matrix;

[0069] S6. Perform circle fitting according to the three-dimensional coordinates of the four feature points in the robot base coordinate system to obtain the three-dimensional coordinates of the center of the circle; perform curve fitting according to the three-dimensional coordinates of the two sets of laser center lines in the robot base coordinate system and calculate the intersection of the two curves and the tangent vector at the intersection, thereby calculating the normal direction of the center of the circle;

[0070] S7. Determine the position and orientation of the reference hole 5 according to the coordinates of the circle center and the normal direction of the circle center.

[0071] The camera image capture in S1 specifically includes:

[0072] S111 . Drag the robot so that the intersection of the two groups of laser center lines is in the reference hole 5 and the reference hole 5 is in the camera field of view, and take an image of the reference hole 5 .

[0073] The light plane equation obtained in S1 specifically includes:

[0074] S121, projecting two sets of laser center lines onto a checkerboard calibration plate, and using a camera to capture images of the checkerboard calibration plate with the laser center lines from different angles;

[0075] S122, extracting the coordinates of the intersection of the laser center line and the checkerboard calibration plate, and the coordinates of the marking points on the checkerboard calibration plate from the acquired image;

[0076] S123, establishing a laser line model according to the intersection point coordinates, and establishing a calibration plate model according to the marking point coordinates;

[0077] S124. Solve the light plane equation according to the laser line model and the calibration plate model.

[0078] The hand-eye transformation matrix obtained in S1 specifically includes:

[0079] S131, fix the position of the checkerboard calibration plate relative to the robot coordinate system, and control the robot to carry a camera to shoot the checkerboard calibration plate, obtain the posture relationship of the checkerboard calibration plate relative to the camera, the posture relationship of the robot end relative to the robot base coordinate system, and the posture relationship of the robot base coordinate system relative to the calibration plate;

[0080] S132. Calculate a hand-eye transformation matrix according to the posture relationship.

[0081] The Hough circle detection in S3 specifically includes:

[0082] S311 , use Hough circle detection to detect the reference hole 5 in the image, and find the two-dimensional coordinates and radius of the center of the reference hole 5 .

[0083] The laser centerline extraction in S3 specifically includes:

[0084] S321, set a grayscale threshold, perform binarization on the image, assign a value of 255 to pixels greater than the grayscale threshold, and assign a value of 0 to pixels less than the grayscale threshold;

[0085] S322, extracting the laser center line of the binarized image, using the Steger method for extracting the center of the light stripe to extract the pixel points of the laser center line, and obtaining the two-dimensional coordinates of the laser center line.

[0086] The S4 specifically includes:

[0087] S41. Obtain the two-dimensional coordinates and radius of the circle center according to the Hough circle detection, obtain the two-dimensional coordinates of the laser center line according to the extracted laser center line, divide the laser center line into two groups according to the circle center and the radius and obtain four feature points.

[0088] The four feature points obtained in S41 specifically include:

[0089] S411, calculating the distance between the extracted two-dimensional coordinate of the laser center line and the center of the reference hole 5, where the distance is recorded as D;

[0090] S412, based on the distance D being equal to the radius of the reference hole 5, four pixel points on the contour of the reference hole 5 are selected from the two-dimensional coordinates of the extracted laser center line, that is, four feature points are obtained.

[0091] The center normal direction in S6 specifically includes:

[0092] S61, fitting the parametric equations of the two curves using a quadratic polynomial according to the three-dimensional coordinates of the centers of the two sets of laser lines;

[0093] S62. Calculate the intersection of the two curves and calculate the tangent vector at the intersection according to the parametric equation. Perform a cross product operation on the tangent vector to obtain the normal at the intersection and obtain the normal to the center of the circle.

[0094] In this embodiment, the working principle of the present invention specifically includes the following steps:

[0095] (1) Camera calibration

[0096] The effective parameters of the internal and external parameters (intrinsic matrix, lens distortion coefficient, rotation matrix, translation vector) are obtained through calibration. Using Zhang Zhengyou's calibration method, multiple images are taken of a chessboard of a specific size, and the internal and external parameters of the camera are calculated through OpenCV.

[0097] Camera calibration is a fundamental and critical technology in the field of computer vision. It involves the process of determining the internal and external parameters of the camera so that the 3D structure of the scene can be accurately restored from the captured image. Internal parameters (intrinsic parameters): refers to the parameters related to the physical characteristics of the camera itself, including the following: Focal length: usually expressed as f x and f y , representing the horizontal and vertical focal lengths, respectively, which determine the magnification capability of the camera lens. 主 ,v 主 ) is represented by, which is a point on the imaging plane, that is, the intersection of the optical axis and the imaging plane, which represents the coordinates of the center of the image. Distortion parameters: describes the image distortion caused by the non-ideal characteristics of the lens, such as radial distortion and tangential distortion. The purpose of intrinsic calibration is to determine these parameters through a specific calibration method. Once these parameters are determined, they can be used for image correction and eliminate distortion to obtain more accurate image data. External parameters (extrinsic parameters): refers to the position and orientation of the camera relative to a reference coordinate system (usually the world coordinate system), including the following: Rotation matrix: describes the rotation relationship between the camera coordinate system and the world coordinate system. Translation vector: describes the position of the origin of the camera coordinate system in the world coordinate system.

[0098] The Zhang Zhengyou calibration method is used to calibrate the camera. The Zhang Zhengyou calibration method uses a black and white chessboard calibration plate as shown in the figure. After obtaining an image of the calibration plate, the Harris corner detection algorithm is used to obtain the pixel coordinates (u, v) of each corner point. The world coordinate system is fixed on the chessboard, and the physical coordinates of any point on the chessboard are w = 0. Since the world coordinate system of the calibration plate is artificially defined in advance, the size of each grid on the calibration plate is known, and we can calculate the physical coordinates of each corner point in the world coordinate system (u, v, w = 0). The camera calibration is performed by calling the calibration toolbox provided by matlab.

[0099] (2) Light plane calibration

[0100] A checkerboard calibration pattern with known geometric features is used to capture the laser lines during the calibration process.

[0101] A line laser is projected onto the calibration plate, and a monocular camera is used to capture images of the calibration plate with the laser line from different angles.

[0102] In the acquired image, the coordinates of the intersection of the laser line and the mark on the calibration plate, as well as the coordinates of the mark point on the calibration plate, are extracted.

[0103] Build the geometry model:

[0104] Laser line model: The equation of the laser line is fitted using the least squares method or other mathematical methods through the extracted point coordinates on the laser line.

[0105] Calibration plate model: Use the extracted coordinates of the marking points on the calibration plate to build the geometric model of the calibration plate.

[0106] Solve the light plane equation:

[0107] Using the laser line model and the calibration plate model, the equation of the laser plane is solved through geometric relationships. Usually, the laser plane equation can be expressed as Ax+By+Cz+D=0. For multiple images, an iterative or optimization algorithm (such as gradient descent) can be used to minimize the error and improve the accuracy of the calibration. Here A, B, C, and D are the coefficients of the laser plane equation, and they have the following meanings:

[0108] A, B and C: These three coefficients form the normal vector of the laser plane, which is perpendicular to the laser plane. D: This coefficient determines the position of the laser plane relative to the origin. Specifically for monocular line laser applications, this equation describes the plane formed in space by the light emitted by the laser transmitter. During the calibration process, we need to determine these coefficients so that we can subsequently calculate the corresponding 3D spatial coordinates based on the laser line information in the image.

[0109] (3) Camera hand-eye calibration

[0110] The chessboard calibration plate is fixed relative to the robot base coordinate system, and the robot is controlled to carry a camera to shoot the chessboard.

[0111] Use Matlab's camera calibration toolbox to obtain the position relationship between the calibration plate and the camera The position relationship of the robot end relative to the robot base coordinate system Obtained from the end-of-arm pose parameters; the pose relationship of the robot base coordinate system relative to the calibration plate Since the calibration plate is fixed in one position throughout the process, the transformation matrix is ​​the same for each set of images.

[0112] In order to solve the transformation matrix from the end of the robot arm to the camera coordinate system Based on the following formula

[0113]

[0114] Derived

[0115]

[0116] Use each set of pictures Unchanged

[0117]

[0118] definition:

[0119]

[0120] Solve the equation AX=XB based on multiple groups of pictures, that is,

[0121] (4) Hough circle detection to identify contour points

[0122] Use Hough circle detection to detect the reference hole 5 in the image and find the pixel coordinates of the center of the reference hole 5 (u o ,v o ) and radius r.

[0123] The captured image of the reference hole 5 is filtered using a median filter. The value of the pixel domain is replaced by the median value in a predefined pixel domain, that is,

[0124] I median (x,y)=median{I(x+i,y+u)|(i,j)∈W}

[0125] Where I(x,y) is the original pixel value at position (x,y), W is a window (usually a square or rectangular area), (i,j) is the offset within the window, and median means taking the median of all pixel values ​​in the window. Median filtering can effectively remove salt and pepper noise and speckle noise in an image without losing too much image details and edge information.

[0126] You can use the median filter function "cv2.medianBlur()" in the OpenCV library for filtering

[0127] The Canny edge detector is used to detect the filtered image. The tool for edge mutation strength and direction at any position (x, y) in the image p is the gradient defined by the following formula

[0128]

[0129] Gradient Represented by and defined as a vector, the gradient direction is the direction at the point (x, y) defined by the following formula.

[0130]

[0131] The Canny edge detection function "cv2.Canny()" in the OpenCV library can be used for processing.

[0132] The Hough circle transform is based on converting an image from the spatial domain to the parameter space. In the spatial domain, a circle can be represented by three parameters: the coordinates of the center of the circle and the radius. In the parameter space, each point corresponds to a possible circle. Considering a point (x, y) in the spatial domain, we can get the equation of a circle:

[0133] (x) 2 +(yb) 2 =r 2

[0134] Where (a, b) is the center of the circle and r is the radius. Then search for possible combinations of (a, b, r) in the Hough three-dimensional space with the coordinates of the center and radius as parameters, and create a three-dimensional accumulator array in the parameter space based on the point (x, y), where each element corresponds to a possible combination of the center and radius. Each element value in the accumulator array represents the number of intersections between the circle and the point through this parameter combination. The corresponding parameter combinations (a, b, r) in the accumulator array that are higher than the set threshold are mapped back to the spatial domain to obtain the corresponding circles. These circles are the circular objects in the image detected by the Hough circle transform.

[0135] The center pixel coordinates (u o , v o ) and radius r of the reference hole 5 can be detected by using the Hough circle detection function "cv2.HoughCircles()" in the OpenCV library and setting the maximum and minimum circle radii.

[0136] (5) Image acquisition and laser center line extraction

[0137] Drag the robot so that the intersection of the line laser is in the reference hole 5 and the reference hole 5 is in the camera's field of view, and take an image of the reference hole 5.

[0138] Set the gray threshold α, and perform binary processing on the image. Pixel points greater than the threshold are assigned a value of 255, and pixel points less than the threshold are assigned a value of 0. Since the line laser is very bright, the line laser can be distinguished from the background.

[0139] Extract the laser center line from the binary image, and use the Steger method to extract the pixel points of the laser center line. When extracting the center of the light strip of structured light, the Steger algorithm is based on the Hessian matrix. Its basic steps are as follows:

[0140] 1. Calculate the normal direction of the line laser stripe from the Hessian matrix

[0141] 2. Expand its gray level distribution according to the Taylor polynomial in the normal direction of the light stripe, and the obtained maximum value is the sub-pixel coordinate of the light stripe in this normal direction. For a two-dimensional discrete image I(u, v), the Hessian matrix can be expressed as:

[0142]

[0143] Here, u and v represent the row and column coordinates of the pixel, and I uv represents the gray level of the pixel (u, v), which can also be called the gray level distribution function. And I uu , I uv and I vv can all be obtained through the convolution operation of I uv and the two-dimensional Gaussian function G(u, v).

[0144]

[0145]

[0146] Here, the two-dimensional Gaussian function is mainly used to make the gray level distribution characteristics of the light stripe more obvious. In the expression of G(u, v), σ is the standard deviation, generally taken as W represents the width of the light strip. The Hessian matrix at the pixel (u,v) has two eigenvectors, one of which has a larger absolute value and is the normal direction vector at the pixel, while the other is the tangent direction vector. Therefore, the normal direction can be calculated by obtaining the eigenvector of the Hessian matrix. 0 ,v 0 ), the Hessian matrix of the second-order Taylor expansion is:

[0147]

[0148] The eigenvalue and eigenvector obtained from the Hessian matrix of the point correspond to the normal direction of the point and the second-order directional derivative of the direction, respectively. The unit vector in the normal direction is: e = [e u ,e v ], and the pixel point in the normal direction of the light stripe (u 0 +t·e u ,v 0 +t·e v ), and the pixel point I in the normal direction of the light stripe is I=(u 0 +t·e u ,v 0 +t·e v ) can be represented by pixels (u 0 ,v 0 ) grayscale I(u 0 ,v 0 ) and the second-order Taylor expansion polynomial are expressed as:

[0149] I(u 0 +t·e u ,v 0 +t·e v )=I(u 0 ,v 0 )+t·e·H(u,v)·e T

[0150]

[0151] Substituting t (i.e., Taylor expansion) into it, the sub-pixel coordinates (u, v) of the center of the light stripe can be obtained.

[0152] (6) Divide the pixel coordinates into two groups and convert them into the robot base coordinate system

[0153] The feature points on the edge of the reference hole 5 are found by using the center coordinates, radius and extracted laser centerline coordinates of the reference hole 5. Since two optical plane equations are calibrated using two laser lines, the extracted two-dimensional laser centerline pixel coordinates need to be divided into two categories and brought into the corresponding optical plane equations to obtain the three-dimensional laser centerline camera coordinates (x c ,y c ,z c ).

[0154] Since the position between the camera and the line laser generator is fixed, the line laser projected by one line laser generator is distributed in the horizontal direction within the camera field of view, and the other line laser is distributed in the vertical direction. Based on this feature, the center coordinates (u, v) of the extracted line laser are calculated according to the center pixel coordinates (u o ,v o ) were divided into two groups.

[0155] According to u 0 ±threshold d can divide the extracted line laser center coordinates (u, v) into two groups, satisfying u∈u 0 ±d line laser center coordinates (u 1 ,v 1 ) Substitute the laser plane equation A along the vertical direction into the line laser distribution 1 x+B 1 y+C 1 z+D 1 = 0, and convert to the camera coordinate system coordinates (x c1 ,y c1 ,z c1 ).

[0156] x c1 =((u 1 -u 主 ) / f x )

[0157]

[0158] z c =(D 1 -A 1 × c1 -B 1 ×y c1 ) / C 1

[0159] Where (u 主 ,v 主 ), f x and f yare the principal point coordinates and the focal lengths in the x and y directions in the camera internal parameter matrix obtained during camera calibration. Similarly, the coordinates of the line laser distributed along the horizontal direction in the camera coordinate system (x c2 ,y c2 ,z c2 ).

[0160] Matrix calibrated by hand and eye And the transformation matrix from the robot base coordinate system to the robot end Convert pixel coordinates to robot base coordinate system coordinates (x b ,y b ,z b ).

[0161]

[0162] (7) Circle center recognition

[0163] Calculate the extracted line laser center coordinates (u, v) to the center pixel coordinates (u o ,v o ) distance D.

[0164]

[0165] According to D being equal to the radius r of the reference hole 5, four pixel points (u′) on the contour of the reference hole 5 can be screened out from the extracted line laser center coordinates. i ,v′ i )i=1,2,3,4, converted to the coordinates of the robot base coordinate system according to step 6

[0166] Using the least squares method, Fit the equation of the circle (x b -x 0 ) 2 +(y b -y 0 ) 2 +(z b -z 0 ) 2 =r 2 , and obtain the center coordinates O of the robot base coordinate system b (x 0 ,y 0 ,z 0 ).

[0167] (8) Normal measurement

[0168] According to the two sets of laser line center coordinates (x c1 ,y c1 ,z c1 ) and (xc2 ,y c2 ,z c2 ) Use a quadratic polynomial to fit two curves L 1 and L 2 The parametric equation of .

[0169]

[0170] Where t is the parameter, a i ,b i ,c i (i=1,2,3) are the coefficients of the equation. Even if the intersection of the two space curves is calculated by taking the derivative of the parametric equation, L 1 and L 2 The tangent vector at the intersection and Finally, the normal at the intersection can be obtained by vector cross product operation

[0171]

[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A reference hole position measurement system based on monocular dual-line laser, the system comprising a monocular camera (1), two line laser generators, a checkerboard calibration board, a robot and a host computer, the monocular camera (1), the line laser generator and the robot are respectively connected to the host computer for communication, the line laser generator emits a laser center line on the checkerboard calibration board, the robot is connected to the monocular camera, and is characterized in that: The host computer performs the following process: A monocular camera (1) is controlled to capture images and calibrate them. After the image is distorted, the two-dimensional coordinates of four feature points and two groups of laser center lines are extracted. The two-dimensional coordinates are converted into three-dimensional coordinates in a robot coordinate system. The corresponding center coordinates and center normal are obtained based on the three-dimensional coordinates in the robot coordinate system, and the position and posture of the reference hole (5) are determined.

2. A method for measuring the position and posture of a reference hole based on a monocular two-line laser as claimed in claim 1, characterized in that: The method specifically includes: S1. Use a camera to capture an image and use Zhang Zhengyou calibration method to calibrate the camera and obtain internal parameters and external parameters, perform light plane calibration and obtain light plane equations, and perform hand-eye calibration and obtain a hand-eye transformation matrix; S2, correcting the image distortion according to internal parameters; S3, performing Hough circle detection after the distortion correction, then binarizing the image and extracting the laser center line; S4, according to the laser center line detected and extracted by the Hough circle, four feature points and two sets of two-dimensional coordinates of the laser center line are obtained; S5. Obtain the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system according to the two-dimensional coordinates of the four feature points and the two groups of laser center lines and the light plane equation; convert the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the camera coordinate system into the three-dimensional coordinates of the four feature points and the two groups of laser center lines in the robot base coordinate system according to the hand-eye transformation matrix; S6. Perform circle fitting according to the three-dimensional coordinates of the four feature points in the robot base coordinate system to obtain the three-dimensional coordinates of the center of the circle; perform curve fitting according to the three-dimensional coordinates of the two sets of laser center lines in the robot base coordinate system and calculate the intersection of the two curves and the tangent vector at the intersection, thereby calculating the normal direction of the center of the circle; S7. Determine the position and orientation of the reference hole (5) according to the coordinates of the circle center and the normal direction of the circle center.

3. The method according to claim 2, characterized in that The camera image capture in S1 specifically includes: S111, drag the robot so that the intersection of the two groups of laser center lines is in the reference hole (5) and the reference hole (5) is in the camera field of view, and take an image of the reference hole (5).

4. The method according to claim 2, characterized in that The light plane equation obtained in S1 specifically includes: S121, projecting two sets of laser center lines onto a checkerboard calibration plate, and using a camera to capture images of the checkerboard calibration plate with the laser center lines from different angles; S122, extracting the coordinates of the intersection of the laser center line and the checkerboard calibration plate, and the coordinates of the marking points on the checkerboard calibration plate from the acquired image; S123, establishing a laser line model according to the intersection point coordinates, and establishing a calibration plate model according to the marking point coordinates; S124. Solve the light plane equation according to the laser line model and the calibration plate model.

5. The method according to claim 2, characterized in that: The hand-eye transformation matrix obtained in S1 specifically includes: S131, fix the position of the checkerboard calibration plate relative to the robot coordinate system, and control the robot to carry a camera to shoot the checkerboard calibration plate, obtain the posture relationship of the checkerboard calibration plate relative to the camera, the posture relationship of the robot end relative to the robot base coordinate system, and the posture relationship of the robot base coordinate system relative to the calibration plate; S132. Calculate a hand-eye transformation matrix according to the posture relationship.

6. The method according to claim 2, characterized in that The Hough circle detection in S3 specifically includes: S311, using Hough circle detection to detect the reference hole (5) in the image, and finding the two-dimensional coordinates and radius of the center of the reference hole (5).

7. The method according to claim 3, characterized in that The laser centerline extraction in S3 specifically includes: S321, set a grayscale threshold, perform binarization on the image, assign a value of 255 to pixels greater than the grayscale threshold, and assign a value of 0 to pixels less than the grayscale threshold; S322, extracting the laser center line of the binarized image, using the Steger method for extracting the center of the light stripe to extract the pixel points of the laser center line, and obtaining the two-dimensional coordinates of the laser center line.

8. The method according to claim 7, characterized in that The S4 specifically includes: S41. Obtain the two-dimensional coordinates and radius of the circle center according to the Hough circle detection, obtain the two-dimensional coordinates of the laser center line according to the extracted laser center line, divide the laser center line into two groups according to the circle center and the radius and obtain four feature points.

9. The method according to claim 8, characterized in that The four feature points obtained in S41 specifically include: S411, calculating the two-dimensional coordinate distance from the extracted laser center line to the center of the reference hole (5), the distance being recorded as D; S412, based on the distance D being equal to the radius of the reference hole (5), four pixel points on the contour of the reference hole (5) are selected from the two-dimensional coordinates of the extracted laser center line, that is, four feature points are obtained.

10. The method according to claim 9, characterized in that The center normal direction in S6 specifically includes: S61, fitting the parametric equations of the two curves using a quadratic polynomial according to the three-dimensional coordinates of the centers of the two sets of laser lines; S62. Calculate the intersection of the two curves and calculate the tangent vector at the intersection according to the parametric equation. Perform a cross product operation on the tangent vector to obtain the normal at the intersection and obtain the normal to the center of the circle.

Images