A single-vision-based threaded hole space positioning and multi-hole alignment method

Abstract

The application discloses a thread hole space positioning and multi-hole alignment method based on monocular vision and belongs to the technical field of visual detection. The method comprises the following steps: S1, constructing a space positioning system based on monocular vision; S2, calibrating the system to establish a conversion relationship between a camera coordinate system and an upper cover through-hole coordinate system; S3, collecting a single thread hole image and positioning a center; S4, using a ray set and a three-dimensional center point set of the thread hole to model pose solving into registration between multiple points and multiple lines to obtain a pose relationship between the upper cover and the lower cover; and S5, aligning the holes of the upper cover through hole and the thread hole. The multi-hole alignment problem is modeled into a registration problem between multiple points and multiple lines. Compared with the manual alignment mode, the total angular deviation is controlled within 0.8 degrees, and the total position deviation is within 0.8 mm.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual detection, and particularly relates to a thread hole space positioning and multi-hole alignment method based on monocular vision. BACKGROUND

[0002] Under the background of continuous innovation of intelligent manufacturing technology, the aerospace field gradually adopts industrial manipulators to complete tasks such as target grabbing, pin hole part assembly, and blade part repair. In actual application scenarios, industrial manipulators often rely on preset programs and lack adaptability to environmental changes, affecting their efficiency and precision in complex industrial environments.

[0003] To improve their intelligent assembly capabilities, integrating visual sensor technology into the manipulator system is key, which will enhance their perception and adaptability to the environment, enabling more efficient and precise automated operations. Research on vision-aided manipulator assembly technology has reached a fairly mature level at home and abroad. Existing technologies successfully integrate visual systems into manipulator systems, enabling the manipulator to automatically identify workpieces, accurately position, and efficiently complete assembly tasks through precise perception of the surrounding environment by the visual system.

[0004] Among them, the vision-based manipulator space positioning system has cooperative and non-cooperative schemes when performing positioning tasks. The cooperative scheme uses a cooperative target, and the visual system can obtain the accurate pose of the target and feed it back to the manipulator system for subsequent tasks. Existing technologies propose a manipulator multi-station alignment technology with far and near distance guidance, which establishes a task table for manipulator multi-station alignment on the basis of system calibration, and then completes three-stage automatic alignment of the manipulator end tool and the target with the assistance of the cooperative target. However, in actual engineering applications, due to the limitations of narrow space and complex working conditions, the cooperative target often cannot be used. The non-cooperative scheme uses the characteristics of the target itself to realize positioning measurement. In order to capture random targets in space, existing technologies propose a measurement and planning method for capturing non-cooperative targets by a space robot based on stereo vision. The target is identified and measured through stereo vision, and the feature points are reconstructed in 3D through the least squares method, so as to measure the position and attitude of the target, and then plan the path and control strategy of the space robot, to achieve the capture of the non-cooperative target.

[0005] At present, the visual-based peg-hole part assembly positioning technology is mainly applied to the peg-hole. In order to solve the problem of large deviation between pegs and holes before assembly, a method for self-alignment of pegs and holes in the robot assembly process is discussed in the prior art, which is based on force / torque measurement. In the prior art, the hole shaft assembly is performed by force control guidance when the visual system cannot be used. Threaded holes are more difficult to handle than through holes, and through holes are easier to position. The research on threaded hole image positioning is relatively less at home and abroad. In the prior art, in order to solve the problem of force impact generated by tightening the bolt when switching from free space to constrained space, an elliptical arc fitting method and a three-point method are used to estimate the initial position and direction of the threaded hole. In the prior art, in order to solve the problem of internal thread detection limitation of mechanical products, an internal thread defect detection method based on deep learning is studied, two methods for obtaining M8 nut internal surface images are introduced, and convolutional neural networks (CNNs) are applied to identify defects in captured images. In summary, the visual-based peg-hole part assembly positioning is relatively less in hole-hole alignment research, and most of the positioning methods combine visual and force control with other technologies, which increases the complexity of the system. In addition, the visual-based threaded hole image processing is difficult, and most of the threaded hole image processing combines deep learning, and the research on spatial positioning of threaded holes is also relatively less. Unlike the common hole shaft alignment, the holes-holes alignment problem cannot be effectively assisted by force control, and the space of the cabin section is limited, so the holes-holes positioning cannot be performed by visual cooperation target.

[0006] For the holes-holes alignment problem of through holes and threaded holes in a narrow cabin section, the present application only uses the cooperation of the visual system and the mechanical arm to study a threaded hole accurate spatial positioning and multi-hole alignment technology between the threaded hole and the through hole. SUMMARY

[0007] For the problems mentioned in the background art, the present application proposes a threaded hole spatial positioning and multi-hole alignment method based on monocular vision, which models the multi-hole alignment problem as a point-line registration problem of multiple points and multiple lines. Compared with the manual alignment method, the total angular deviation is controlled within 0.8°, and the total position deviation is within 0.8mm.

[0008] Technical scheme: In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0009] A threaded hole spatial positioning and multi-hole alignment method based on monocular vision, comprising the following steps:

[0010] S1: constructing a monocular vision-based spatial positioning system;

[0011] S2: calibrating the system to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system;

[0012] S3: performing single-thread hole image acquisition and center positioning;

[0013] S4: using the ray set and the three-dimensional center point set of the threaded hole to model the pose solving as a registration between multiple points and multiple lines to obtain the pose relationship between the upper and lower covers;

[0014] S5: aligning the holes of the upper and lower cover through-holes and the threaded holes.

[0015] As preferred, in S1, the specific content of constructing a monocular vision-based spatial positioning system is:

[0016] The system includes a mechanical arm, a camera, a ring light source, a flange, an upper cover, a lower cover, and a hand-eye calibration board; the camera and the ring light source constitute the vision system of the system;

[0017] The coordinate system of the system is defined as follows: the mechanical arm base coordinate system {b}, the camera coordinate system {c}, the mechanical arm end coordinate system {r}, the upper cover through-hole coordinate system {s}, and the lower cover threaded hole coordinate system {w}.

[0018] As preferred, in S2, the specific content of calibrating the system to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system is:

[0019] The system calibration includes camera calibration and hand-eye calibration;

[0020] Camera calibration: Zhang Zhengyou plane calibration method is used for calibration to ensure that the camera can accurately map the three-dimensional space to the two-dimensional image, establish and solve the geometric model parameters of the camera, including the intrinsic, extrinsic, and distortion parameters;

[0021] Hand-eye calibration: a hand-eye calibration board is set, two types of array holes with different sizes are set on the calibration board, the camera captures the array hole image, and the array hole center coordinates are extracted by processing the image;

[0022] Given the spatial three-dimensional coordinates of the array holes, the array hole image center pixel coordinates, and the camera model parameters, the conversion relationship from the mechanical arm end coordinate system to the camera coordinate system is solved by using the iterative method including a rotation matrix and a translation vector

[0023] As preferred, the specific content of camera calibration is:

[0024] A calibration board with known size and uniform dot distribution is selected, and a camera to be calibrated is used to capture multiple clear images from different angles and positions, ensuring that the calibration board is clear and complete in the field of view; using Zhang Zhengyou plane calibration method, the coordinates of the center of the dot on the image are first detected by using a corner detection algorithm specially designed for dot pattern; the calibration board plane is set as the XY plane of the world coordinate system, a certain specific dot is selected as the origin of the world coordinate system, and the world coordinate system is established in units of dot spacing.

[0025] Solve the homography matrix, calculate the initial value of the camera intrinsic parameter according to the homography matrix, and then calculate the extrinsic parameter; use a nonlinear optimization algorithm to optimize the intrinsic parameter and the extrinsic parameter; calculate the distortion parameters of the camera, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

[0026] As preferred, in S3, the specific content of single thread hole image acquisition and center positioning is:

[0027] S31: Single thread hole image acquisition: control the robot arm to move the camera above the lower cover thread hole, sequentially capture the thread hole image and record the robot arm pose;

[0028] S32: Center positioning: based on the elliptical contour extraction algorithm of adjacent arc segment combination, the thread hole image is processed, the strong interference image noise near the thread hole is removed, and the center coordinates of the thread hole image are obtained through robust edge fitting for spatial positioning.

[0029] As preferred, in S32, the specific content of center positioning: based on the elliptical contour extraction algorithm of adjacent arc segment combination, the thread hole image is processed, the strong interference image noise near the thread hole is removed, and the center coordinates of the thread hole image are obtained through robust edge fitting for spatial positioning is:

[0030] S321: Perform conventional preprocessing on the collected thread hole image, including Gaussian filtering, median filtering and Canny edge processing;

[0031] S322: Extract thread hole arc segments based on edge extraction;

[0032] S323: Remove too short arc segments using adaptive threshold, and construct adjacency relationship through adjacency matrix to complete edge segmentation and arc segment combination;

[0033] S324: Fit the elliptical image through least squares method to obtain the elliptical center coordinate information, and verify the fitting result.

[0034] As preferred, in S4, the specific content of using the set of rays and the set of three-dimensional center points of the thread hole to model the pose solving as the registration between multiple points and multiple lines to obtain the pose relationship between the upper cover and the lower cover is:

[0035] S41: According to the visual imaging geometry, the equation of the ray passing through the center of the ellipse in the camera coordinate system is constructed;

[0036] First, under the ideal pinhole imaging model, a single threaded hole imaging is modeled; sequentially shooting a single threaded hole target, a set of rays can be obtained;

[0037] S42: Using the set of rays and the set of three-dimensional center points of the threaded hole, the alignment problem is converted into a registration optimization solution of multiple rays passing through multiple points, and the relationship between the camera coordinate system and the threaded hole coordinate system of the lower cover is obtained by the iterative geometric solution in point-line registration.

[0038] As preferred, in S42, using the set of rays and the set of three-dimensional center points of the threaded hole, the alignment problem is converted into a registration optimization solution of multiple rays passing through multiple points, and the relationship between the camera coordinate system and the threaded hole coordinate system of the lower cover is obtained by the iterative geometric solution in point-line registration.

[0039] In the point-line registration process of multiple points and multiple lines, the multiple points are defined by the known three-dimensional space points, and the multiple rays are defined by the starting points and direction vectors. The specific calculation formula is as follows:

[0040]

[0041] Where P is a matrix composed of the direction vectors of the rays, O is a matrix composed of the starting points of the rays, S is a matrix composed of three-dimensional points in the world coordinate system, Z is a diagonal parameter matrix, and ζ1…ζ n for parameterizing a straight line in space, c is a vector representing the position of the ray, is a rotation matrix, is a translation vector, and α is a scaling factor; M is a three-dimensional center point of the threaded hole; is a rotation matrix transpose of; p n T transpose of each direction vector; o n T transpose of each optical center vector; c T transpose of the ray position vector; M n T transpose of the threaded hole three-dimensional center point vector; 1c T is a matrix composed of the ray position vector;

[0042] As preferred, the iterative method is used to solve the point-line registration problem, and the pose relationship of the threaded hole coordinate system of the lower cover relative to the camera coordinate system is calculated contains a rotation matrix and a translation vector

[0043] As preferred, in S5, the specific content of the hole-hole alignment of the upper cover through-hole and the threaded hole is that:

[0044] The pose relationship of the robot end coordinate system relative to the base coordinate system is obtained through the robot TCP calibration According to the above obtained hand-eye calibration result And the pose relationship of the lower cover to the virtual multi-view system The pose relationship of the lower cover threaded hole coordinate system relative to the robot base coordinate system is obtained Including rotation matrix And translation vector The specific formula is:

[0045]

[0046] Among them, And The conversion relationship of the lower cover threaded hole coordinate system to the camera coordinate system; And The conversion relationship of the robot end coordinate system and the base coordinate system;

[0047] Through the pose relationship between the upper cover through-hole coordinate system and the robot end coordinate system, including rotation matrix And translation vector According to The pose relationship of the upper cover through-hole coordinate system relative to the robot base coordinate system is obtained Including rotation matrix And translation vector The specific formula is:

[0048]

[0049] Among them, And The conversion relationship of the robot end coordinate system and the base coordinate system;

[0050] Given the above pose relationship, a new Including rotation matrix And translation vector And adjust the robot end pose accordingly to realize the multi-hole alignment of the upper and lower covers, and the specific formula is:

[0051]

[0052] Among them, And The conversion relationship of the lower cover threaded hole coordinate system and the robot base coordinate system, And The conversion relationship of the robot end coordinate system and the upper cover through-hole coordinate system.

[0053] Advantages: Compared with the prior art, the present application has the following advantages:

[0054] (1) In the robot automatic assembly of the electrical control device on the missile cabin section, the multiple through holes and multiple threaded holes on the upper and lower covers of the control shell need to be accurately positioned and aligned. The holes-holes alignment problem cannot be effectively assisted by force control, and the cabin space limitation also cannot be assisted by visual cooperation target for hole-hole positioning. Therefore, the present application proposes a threaded hole space positioning and multi-hole alignment technology based on monocular vision. The core idea of the technology is to model the multi-hole alignment problem as a multi-point and multi-line registration problem.

[0055] (2) The present application proposes a method of mechanical transfer of hand-eye calibration board based on "two faces and one hole" positioning for hand-eye calibration, to obtain more accurate hand-eye relationship for subsequent multi-hole positioning and alignment.

[0056] (3) The present application adopts an adjacent arc segment combined ellipse extraction algorithm, which can effectively eliminate strong interference image noise near the threaded hole, and obtain the center coordinates of the threaded hole image for space positioning through edge robust fitting. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a working site schematic diagram of the present application;

[0058] Figure 2 is a positioning system structure schematic diagram of the present application;

[0059] Figure 3 is a system technical route flowchart of the present application;

[0060] Figure 4 is a hand-eye calibration schematic diagram of the present application;

[0061] Figure 5 is a threaded hole image processing schematic diagram of the present application, wherein, Fig. (a) is a threaded hole original image; Fig. (b) is a target contour, Fig. (c) is a preprocessed image, Fig. (d) is an arc segment extraction, Fig. (e) is a short arc segment elimination, and Fig. (f) is a result image;

[0062] Figure 6 is a threaded hole imaging modeling diagram of the present application;

[0063] Figure 7 is a virtual rigid connection multi-camera system schematic diagram of the present application;

[0064] Figure 8 is an experimental working scene schematic diagram of the present application;

[0065] Figure 9is a schematic diagram of the upper and lower cover alignment state of each hole of the present application;

[0066] Figure 10 is a schematic diagram of the bolt screwing into each hole state of the present application;

[0067] Figure 11 is a schematic diagram of the pose deviation of the present application, wherein figure (a) is the angular deviation around the x, y, z axis, and figure (b) is the position deviation along the x, y, z axis. DETAILED DESCRIPTION

[0068] The present application will be further illustrated below in combination with specific embodiments, which are implemented on the premise of the technical scheme of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application.

[0069] The single-vision-based threaded hole space positioning and multi-hole alignment method provided by the embodiment includes the following steps:

[0070] S1: constructing a single-vision-based space positioning system;

[0071] The hole alignment problem mentioned in the present application is derived from the robot automatic assembly of the control device on the cabin section of the missile, and it is necessary to accurately position and align the multiple through holes and multiple threaded holes on the upper and lower covers of the controller shell (such as a servo thermocell).

[0072] As shown in Figure 1 , Figure 2 The present embodiment constructs a single-vision-based space positioning system in the form of Eye-in-hand. The system includes a mechanical arm, a camera, a ring light source, a flange, an upper cover, a lower cover, and a high-precision hand-eye calibration board; the camera and the ring light source constitute the vision system of the system; under the laboratory scene, the upper and lower covers are simulation samples; the flange is used to fix the mechanical arm end and the upper cover together; the pose relationship between the camera coordinate system and the mechanical arm end coordinate system is needed in the pose solving process, therefore a hand-eye calibration board with high-precision array holes is constructed in the present embodiment for high-precision hand-eye calibration, and the distance from the camera to the array holes on the hand-eye calibration board is similar to the distance for normal single-hole image acquisition. The upper cover is fixed on the flange at the end of the mechanical arm, and the camera is installed in the middle slot of the upper cover; when performing hand-eye calibration, the hand-eye calibration board is installed below the camera with an adapter plate.

[0073] The coordinate system of the system is defined as follows: the mechanical arm base coordinate system {b}, the camera coordinate system {c}, the mechanical arm end coordinate system {r}, the upper cover through hole coordinate system {s}, and the lower cover threaded hole coordinate system {w}. Refers to the pose relationship of the mechanical arm end coordinate system relative to the base coordinate system; It refers to the pose relationship of the mechanical arm end coordinate system relative to the camera coordinate system, which is defined as the hand-eye relationship in the embodiment. It refers to the pose relationship of the lower cover threaded hole coordinate system relative to the camera coordinate system. It refers to the pose relationship of the upper cover through-hole coordinate system relative to the mechanical arm base coordinate system. It refers to the pose relationship of the lower cover threaded hole coordinate system relative to the mechanical arm base coordinate system.

[0074] S2: Calibrate the system to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system.

[0075] As shown in Figure 3 , it is the specific technical route of the system.

[0076] Calibration is extremely important in the visual positioning system. In the embodiment, it mainly includes camera calibration and high-precision hand-eye calibration. Camera calibration provides accurate camera model parameters, which is the prerequisite for subsequent hand-eye calibration and threaded hole image processing. High-precision hand-eye calibration board is used for hand-eye calibration. Hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system. The final mechanical arm alignment requires obtaining the pose relationship between the camera coordinate system and the mechanical arm end coordinate system.

[0077] Camera calibration ensures that the camera can accurately map the three-dimensional space to the two-dimensional image, establishes and solves the geometric model parameters of the camera, including intrinsic, extrinsic and distortion parameters. In the embodiment, the classic and effective Zhang Zhengyou plane calibration method is used for calibration.

[0078] Camera calibration requires preparation, including selecting a calibration board with known size and uniform circular point distribution, accurately measuring the key information such as the circular point spacing (in the embodiment, the circular point spacing is 10mm), and taking at least 10 clear images from different angles and positions with the camera to be calibrated, ensuring that the calibration board is complete and has no blur and obstruction in the field of view. Using Zhang Zhengyou plane calibration method, first detect the coordinates of the circular point center in the image on the calibration board using a corner detection algorithm specially designed for circular point patterns. The calibration board plane is set as the XY plane of the world coordinate system, and a certain specific circular point (such as the upper left corner circular point) is selected as the origin of the world coordinate system. The corresponding relationship between the world coordinate system and the calibration board is established in units of circular point spacing. Then solve the homography matrix and calculate the initial value of the camera intrinsic parameter, and then calculate the extrinsic parameter. Then use a nonlinear optimization algorithm to optimize the intrinsic and extrinsic parameters. At the same time, calculate the distortion parameters of the camera, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

[0079] Hand-eye calibration is an important basis for the visual spatial positioning technology of threaded holes. In the hand-eye system of the embodiment, the end of the robot arm is the "hand", and the camera is the "eye". The hand-eye calibration is to establish the relationship between the robot arm end coordinate system and the camera coordinate system. The hand-eye calibration diagram is shown in Figure 4 .

[0080] In the embodiment, a high-precision hand-eye calibration board is provided as shown in Figure 4 . The calibration board is provided with two types of array holes of different sizes, nine large array holes with a diameter of 10 mm and four small array holes with a diameter of 8 mm. The positional relationship between the array holes is known, and the accuracy is guaranteed by machining. In order to avoid uneven lighting and poor image of the edge array hole, the small size array hole is arranged in the center area with good lighting condition to improve the accuracy of the ellipse feature extraction. Generally, there is a pin hole on the flange of the end of the robot arm for accurate positioning. In order to ensure the accuracy of positioning, the "two-hole-one-face" positioning method is adopted in the embodiment, and the threaded connection at the end is changed to threaded pin positioning to realize the accurate positioning of the flange of the workpiece at the end. In addition, two pin holes are designed at the connection of the workpiece related to the hand-eye calibration board for accurate positioning between the workpieces, and finally the positional relationship of the array holes on the hand-eye calibration board is transmitted to the end of the robot arm.

[0081] When the system performs hand-eye calibration, the calibration board is installed below the clamp and the assembly accuracy is ensured by the pin hole positioning described above, so the coordinates of the array holes in the robot arm end coordinate system are known. The camera captures the array hole image, processes the image and extracts the array hole center coordinates. Knowing the spatial three-dimensional coordinates of the array holes, the array hole image center pixel coordinates and the camera model parameters, the pose relationship between the robot arm end coordinate system and the camera coordinate system is solved, which can be summarized as the PnP problem (Perspective-n-Point).

[0082] The PnP problem involves determining the pose of a three-dimensional object relative to the camera coordinate system from a set of points in three-dimensional space and their two-dimensional projections on the image. Among the many methods for solving the PnP problem, the iterative method can handle any number of feature points, and through a continuous iterative process, the camera pose estimate is gradually optimized. This method not only has high flexibility, but also can provide high accuracy. For the hand-eye calibration board of the embodiment, the number of feature points is large, and these characteristics of the iterative method are particularly important. Using the iterative method, the conversion relationship from the robot arm end coordinate system to the camera coordinate system is solved which contains a rotation matrix and a translation vector

[0083] Specifically, first perform an initialization estimate. The rotation and translation parameters can be preliminarily determined based on geometric relationships. For example, first roughly determine the translation direction, and set the rotation to the unit matrix. Then construct an error function. According to the perspective projection model, the points in the world coordinate system and the points in the camera coordinate system are related by rotation and translation. Then, according to the relationship between the pixel coordinates in the camera model and the camera coordinate system points, the sum of the projection errors of all spatial points in the world coordinate system is defined as the error function. The error here is the difference between the actual pixel coordinates and the projected pixel coordinates calculated by the current estimated rotation and translation. Then perform an iterative update to obtain the Jacobian matrix by calculating the partial derivatives of the error function with respect to the rotation and translation parameters. Each iteration calculates the Jacobian matrix and error vector based on the current parameter estimate, and updates the parameters through a specific calculation method. When updating the rotation parameters, ensure that they are orthogonal matrices. Finally, determine whether convergence is achieved based on the termination condition. When the parameter update amount is small enough, the iteration ends. For example, for the calculated rotation matrix By calculating the rotation matrix obtained by the previous iteration The Frobenius norm (F-norm) between , defined when When , it is considered that the update amount of the rotation parameter is small enough, where ε1 is a small positive number that is pre-set. The specific value can be determined according to the accuracy requirements of the actual application. In computer vision tasks with high requirements for posture accuracy, the value may be 10 -4 or smaller; for the calculated translation parameters By calculating the translation parameter obtained by the previous iteration The Euclidean norm between When the update amount of the translation parameter is considered small enough, ε2 can be taken as 10 -6 Or smaller. The iteration ends, and the rotation and translation parameters at this time are the final rotation matrix and translation vectors

[0084] S3: Single threaded hole image acquisition and center positioning;

[0085] S31: Single threaded hole image acquisition: Control the robotic arm to move the camera to the top of the threaded hole on the lower cover, take four threaded hole images in sequence, and record the robotic arm's position;

[0086] like Figure 5 As shown in (a) of the figure, the threaded hole of the lower cover is photographed by a camera to obtain a threaded hole image. As can be seen from the figure, the threaded hole image is affected by factors such as lighting conditions, hole edge chamfers, and thread outer teeth. Accurately obtaining the edge contour of the threaded hole image and the coordinate information of the ellipse center is a key step in high-precision spatial positioning of the threaded hole. Under lighting conditions, a bright ring will appear on the thread of the lower cover threaded hole, as shown in the figure.Figure 5 The part pointed by the arrow (b) is the threaded hole edge contour target that needs to be extracted in this embodiment.

[0087] S32: Center positioning: Process the collected threaded hole image and extract the sub-pixel coordinates of the center of the single hole image;

[0088] The threaded hole image is processed by an ellipse contour extraction algorithm based on the combination of adjacent arc segments to remove the strong interference image noise near the threaded hole. The center coordinates of the threaded hole image are obtained for spatial positioning through edge robust fitting.

[0089] S321: performing conventional preprocessing on the acquired threaded hole image, including Gaussian filtering, median filtering, and Canny edge processing;

[0090] First, the threaded hole image is preprocessed conventionally, including Gaussian filtering, median filtering and Canny edge processing. The preprocessed image is as follows: Figure 5 As shown in (c), the edges of the threaded hole image are complex and it is difficult to extract a complete and valid ellipse image.

[0091] S322: Extract threaded hole arc segments based on edges;

[0092] like Figure 5 As shown in (d) in the figure, the threaded hole arc segment is extracted based on the edge; Canny edge detection is first performed on the image, and then the image is thickened to facilitate finding the starting and ending points of the edge contour.

[0093] S323: Using an adaptive threshold to remove short arcs, and constructing adjacency relationships through an adjacency matrix to complete edge segmentation and arc segment combination;

[0094] like Figure 5 As shown in (e) in the figure, edge segmentation is performed based on the concavity and curvature of the elliptical image. After completing the edge sub-arc segmentation, an adaptive threshold is determined, arcs that are too short and have a length less than the threshold are removed, and the remaining sub-arcs are sorted. The adjacency relationship of each sub-arc is constructed through the adjacency matrix, and the arcs are combined and clustered according to the arc search area restriction and edge curvature restriction.

[0095] S324: Fitting the ellipse image using the least squares method to obtain the coordinate information of the ellipse center, and verifying the fitting result;

[0096] The fitting results are verified by four indicators: weight, shape, position and gradient. After completing the arc segment clustering, the candidate elliptical arc segment set can be fitted to the elliptical image by the least squares method.

[0097] In addition, the post-processing method for eliminating the ambiguity of the ellipse image and eliminating the false ellipse caused by noise can effectively improve the recognition accuracy of the threaded hole. Through the above steps, the ellipse and its center two-dimensional coordinate information in the threaded hole image can be effectively extracted, such as Figure 5 (f) shown in.

[0098] S4: using the ray set and the threaded hole three-dimensional center point set, modeling the pose solving as the registration between multiple points and multiple lines to obtain the pose relationship between the upper cover and the lower cover;

[0099] S41: according to the visual imaging geometry, a ray equation passing through the ellipse center in the camera coordinate system is constructed;

[0100] The traditional PnP method is to take a single photo of all features in a visual coordinate system, and the target pose information is calculated by corresponding two-dimensional and three-dimensional information. Due to the limitation of the narrow shell space, the mechanical arm can only be moved to take photos of the threaded holes one by one, which brings the problem that all threaded hole images are obtained in their respective visual coordinate systems.

[0101] First, under the ideal pinhole imaging model, a single threaded hole imaging model is established. The three-dimensional center point M i of the threaded hole is projected onto the camera imaging plane as m i . Figure 6 It can be concluded that there is a ray between the camera optical center, the plane projection point and the three-dimensional point. The ray starts from the optical center o i , and the ray direction vector p i can be obtained by back projection, while the intrinsic matrix A can be obtained by camera calibration, as shown in formula (1).

[0102]

[0103] By sequentially taking photos of single threaded hole targets, a set of rays can be obtained. Wherein, α is the focal length of the camera in the X axis direction, β is the focal length of the camera in the Y axis direction; u 0 and v 0 are the principal point positions, that is, the positions of the image plane center in the pixel coordinate system; p' is the direction vector without normalization processing; u i and v i represent the threaded hole image center coordinates; p i represents the normalized direction vector; x, y, z are the component values of the direction vector in back projection calculation.

[0104] S42: according to the virtual multi-view geometry of the moving pose of the machine arm, the relationship between the camera coordinate system and the threaded hole coordinate system of the lower cover is obtained;

[0105] The center ray of each thread hole image obtained by the visual sequential shooting is represented in the camera coordinate system, and a virtual rigidly connected multi-camera system is established; the three-dimensional coordinates of the thread hole center are obtained according to the lower cover digital model, and then the hole alignment problem is modeled into a multi-point and multi-line registration problem, and then the pose of the thread hole coordinate system of the lower cover to the virtual multi-camera coordinate system is obtained through the iterative geometric solution in the point-line registration, that is If the point-line registration cannot be converged, that is, the tolerance Tol' of the iterative calculation cannot be lower than the set value Tol, the thread hole picture needs to be re-shot and processed.

[0106] In this embodiment, the camera pose for shooting the No. 1 thread hole is taken as the reference, the conversion relationship between the monocular camera of different poses and the reference camera position is obtained through the known hand-eye relationship between the monocular vision and the end of the mechanical arm and the conversion relationship between the end of the mechanical arm coordinate system and the base coordinate system, and the camera coordinate system of the remaining poses is converted to the reference camera coordinate system, and the formula is as follows:

[0107]

[0108] Among them, and is the conversion relationship between the end of the mechanical arm coordinate system and the base coordinate system in the camera pose for shooting the No. 2 thread hole. and is the conversion relationship between the end of the mechanical arm coordinate system and the base coordinate system in the camera pose for shooting the No. 2 thread hole. and is the conversion relationship between the end of the mechanical arm coordinate system and the base coordinate system in the camera pose for shooting the No. 2 thread hole. and is the conversion relationship between the camera coordinate system and the end of the mechanical arm coordinate system. is the transpose of . is the transpose of .

[0109] By using and , the camera coordinate system for shooting the No. 2 thread hole can be converted to the reference camera coordinate system, and the camera coordinate system for shooting the remaining thread holes can be converted to the reference camera coordinate system by using similar formulas, and the camera coordinate systems of different poses can be connected, that is, the ray set is also rigidly connected, so that a virtual rigidly connected multi-camera system can be constructed, as shown in Figure 7 .

[0110] In order to obtain the pose relationship between the camera system and the lower cover, the pose solving problem can be modeled as a registration problem between multiple points and multiple lines, i.e. a point-line registration problem, by using the set of rays and the set of three-dimensional center points of the threaded holes.

[0111] The point-line registration of multiple points and multiple lines is to extend the traditional PnP problem to NPnP (Non-Perspective-n-Point), and to extend the perspective camera to a non-perspective camera, i.e. a camera model in which the projection rays do not intersect at a single point. This includes all non-central devices and multi-camera systems, in which the cameras are rigidly connected, so the lines are also a set of rigidly connected rays. But the overall set of rays does not intersect at a point, i.e. it is not required that these rays form a bundle.

[0112] In the point-line registration process of multiple points and multiple lines, the multiple points are defined by known three-dimensional space points, and the multiple rays are defined by starting points and direction vectors. The goal is to find the rotation matrix, translation vector and possible scaling factor of the three-dimensional space point coordinate system to the rigidly connected multi-camera system coordinate system, so that these points and rays match as much as possible after transformation. The formula is as follows:

[0113]

[0114] In the formula, P is a matrix composed of the direction vectors of the rays, O is a matrix composed of the starting points of the rays, S is a matrix composed of three-dimensional points in the world coordinate system, Z is a diagonal parameter matrix, and ζ1…ζ n For parameterizing a straight line in space, c is a vector representing the position of the ray, is a rotation matrix, is a translation vector, and α is a scaling factor. M is a three-dimensional center point of the threaded hole; is a rotation matrix transpose of the rotation matrix; p n T transpose of each direction vector; o n T transpose of each optical center vector; c T transpose of the ray position vector; M n T transpose of the three-dimensional center point vector of the threaded hole; 1c T is a matrix composed of the ray position vectors.

[0115] There are two solutions to solve the point-line registration problem. One is a direct closed-form solution, which requires at least six sets of point-line and minimizes algebraic error. The other is an iterative solution, which starts from an information-free initialization, obtains a minimized geometric error in 3D space, calculates a tolerance value Tol', converges when it is lower than a set tolerance value Tol, and can use a minimum of four sets of point-line. The four threaded holes of the embodiment can provide a minimum of four sets of point-line, so the iterative method is used to solve the problem, and the pose relationship of the lower cover threaded hole coordinate system relative to the camera coordinate system can be calculated including a rotation matrix and a translation vector

[0116] The specific implementation is as follows. First, initialize the diagonal matrix Z, then calculate the rotation matrix, the purpose of which is to align the directions of the 3D points and lines. Then determine a scale factor α for adjusting the size of the 3D points to match the length of the line segments. Next, calculate a translation vector to move the 3D points along the rotated line direction to best match the position of the line segment. After the rotation matrix, translation vector, and scale factor are known, update the parameters of each line segment to determine the optimal position of the 3D points on each line. Repeat the above steps until the changes in the line segment parameters are very small, and obtain a stable rotation matrix and translation vector.

[0117] S5: Hole-hole alignment of the through holes and threaded holes of the upper and lower covers;

[0118] With the help of the hand-eye relationship, the coordinate systems of the upper and lower covers are converted to the end of the mechanical arm to obtain the pose deviation, thereby guiding the mechanical arm to achieve multi-hole alignment.

[0119] Pose solving:

[0120] The ultimate goal of the embodiment is to obtain the pose relationship between the camera coordinate system and the upper cover through hole coordinate system and the lower cover threaded hole coordinate system through monocular vision guidance, and to achieve accurate alignment between the four through holes of the upper cover and the four threaded holes of the lower cover. Using the already obtained pose relationship between the coordinate systems, the relationship established by vision and the upper and lower covers is converted to the base coordinate system of the mechanical arm, and the mechanical arm connected with the upper cover is controlled to realize hole-hole alignment with the lower cover.

[0121] The pose relationship of the end of the mechanical arm coordinate system relative to the base coordinate system can be obtained through TCP calibration of the mechanical arm According to the hand-eye calibration results obtained above and the pose relationship of the lower cover to the virtual multi-view system the pose relationship of the lower cover threaded hole coordinate system relative to the base coordinate system of the mechanical arm (including a rotation matrix and a translation vector ) can be obtained.

[0122]

[0123] wherein, and is the transformation relationship from the upper cover through-hole coordinate system to the camera coordinate system; and is the transformation relationship between the mechanical arm end coordinate system and the base coordinate system.

[0124] It should be noted that the high-precision machining and pin hole positioning of the present embodiment can guarantee the pose relationship (including rotation matrix and translation vector ) between the upper cover through-hole coordinate system and the mechanical arm end coordinate system, and then the pose relationship (including rotation matrix and translation vector ) of the upper cover through-hole coordinate system relative to the mechanical arm base coordinate system can be obtained.

[0125]

[0126] wherein, and is the transformation relationship between the mechanical arm end coordinate system and the base coordinate system.

[0127] Since the present embodiment is realized by controlling the mechanical arm to achieve alignment, the above-mentioned pose relationship is known, and a new (including rotation matrix and translation vector ) can be solved and the mechanical arm end pose can be adjusted accordingly, so that the holes-holes multi-hole alignment of the upper and lower covers can be realized.

[0128]

[0129] wherein, and is the transformation relationship between the lower cover threaded hole coordinate system and the mechanical arm base coordinate system, and is the transformation relationship between the mechanical arm end coordinate system and the upper cover through-hole coordinate system.

[0130] S6: Multi-hole alignment experimental verification.

[0131] In order to verify the effectiveness of the threaded hole space positioning and alignment method based on vision in the present text, a working scene as shown in Figure 8 is built.

[0132] The system takes UR10 robot as the actuator to guide the vision system to collect images and holes-holes alignment. The monocular camera uses MER2-502-79U3M camera of Daheng series, with a resolution of 2448(H)×2048(V), and is used with a 16mm lens. The inner diameter of the four threaded holes of the lower cover is M7, and the diameter of the through hole of the upper cover is Φ9.

[0133] Before the robot guides the monocular camera to collect images of the threaded holes, the system has completed camera calibration and hand-eye calibration, and obtained the corresponding pose relationship.

[0134] 1. Alignment process and result experiment verification:

[0135] First, the robot guides the monocular camera to take pictures of No. 1 to No. 4 threaded holes in turn, and processes the images to obtain the center coordinates of the threaded holes during the shooting process. After shooting all the threaded holes, the pose relationship of the lower cover relative to the robot base coordinate system can be calculated within a few seconds. Then, the pose of the robot end is adjusted to make the through hole of the upper cover and the threaded hole of the lower cover achieve holes-holes alignment.

[0136] When the upper and lower covers are positioned and aligned, the system state is as shown in Figure 9 In the actual assembly process, the subsequent bolts need to be screwed in to realize the fastening of the upper and lower covers. In this embodiment, the bolts are screwed in manually. In order to facilitate observation, a gap is left between the two surfaces of the upper and lower covers during alignment, and the state of the holes after screwing in is as shown in Figure 10 .

[0137] 2. Repetitive experiment verification:

[0138] Since there is a margin between the through hole and the threaded hole of the upper and lower covers, there are multiple groups of poses that can satisfy the alignment of the holes of the upper and lower covers. Therefore, the effectiveness of this embodiment is further verified by comparing with the manual alignment method.

[0139] In this experiment, the lower cover is placed in three different positions, and three groups of desired poses of the robot end are obtained by manual alignment. The requirement of manual alignment is that all the bolts can be smoothly screwed in, and there is no obvious deviation between the center of the through hole and the threaded hole. By using the method of this embodiment, the alignment experiment of the above (alignment process and result experiment) is independently carried out at three different positions of the lower cover, and compared with the desired pose of the manual alignment. The results are shown in Table 1. Among them, Δα, Δβ and Δγ are the angular deviations around X, Y and Z axes respectively, and ΔX, ΔY and ΔZ are the positional deviations along X, Y and Z axes respectively.

[0140] Table 1 Position deviation of repetitive experiment

[0141] Groups Δα / (°) Δβ / (°) Δγ / (°) ΔX / mm ΔY / mm ΔZ / mm 1 -0.389 -0.427 -0.183 -0.403 0.041 0.482 2 -0.482 0.275 -0.475 0.453 0.122 0.027 3 0.159 0.413 0.028 0.420 -0.516 0.383

[0142] When the lower cover is in position 1, the total angular deviation is 0.606°, and the total position deviation is 0.629 mm. When the lower cover is in position 2, the total angular deviation is 0.730°, and the total position deviation is 0.470 mm. When the lower cover is in position 3, the total angular deviation is 0.443°, and the total position deviation is 0.768 mm. It can be seen from the above that Figure 11 It can be seen that the angular deviation around each axis during alignment is less than 0.5°, and the position deviation along each axis is about 0.5 mm.

[0143] It should be further pointed out that in the actual assembly process, whether the bolt can be screwed into the already aligned threaded hole is the basic standard, and in the above experiment, the four threaded holes of the lower cover can be screwed into the bolt after alignment.

[0144] The present application aims at the problem that multiple through holes and threaded holes of upper and lower workpieces need to be simultaneously aligned in the environment of automatic assembly of non-cooperative target robots, and builds a spatial positioning system based on monocular vision. The hole-hole alignment precision requirement of Φ9 through hole and M7 threaded hole can be achieved, and the final bolt can be successfully screwed in. The system realizes hole-hole alignment in a pure visual manner, and has the advantages of simple structure, high flexibility and strong adaptability, and has important value in actual engineering applications.

[0145] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled persons in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision, characterized by: The following steps are involved: S1: Build a spatial positioning system based on monocular vision; S2: Calibrate the system and establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system; S3: Single threaded hole image acquisition and center positioning; S31: Single threaded hole image acquisition: Control the robotic arm to move the camera to the top of the threaded hole of the lower cover, take images of the threaded holes in sequence, and record the robotic arm's position; S32: Center positioning: The threaded hole image is processed based on the ellipse contour extraction algorithm based on the combination of adjacent arc segments to remove the strong interference image noise near the threaded hole. The center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning. S4: Using the ray set and the 3D center point set of the threaded hole, the pose solution is modeled as a registration between multiple points and multiple lines, i.e., the NPnP problem, to obtain the pose relationship between the upper and lower covers; S41: Based on the visual imaging geometry, construct the ray equation passing through the center of the ellipse in the camera coordinate system; First, a single threaded hole imaging model is built under the ideal pinhole imaging model. By sequentially photographing the single threaded hole target, a ray set can be obtained. S42: Using the ray set and the 3D center point set of the threaded holes, the alignment problem is transformed into a registration optimization solution involving multiple rays passing through multiple points. The relationship between the camera coordinate system and the lower cover threaded hole coordinate system is then obtained through the iterative geometric solution method in point-line registration. S5: Align the through holes of the upper and lower covers with the threaded holes; The position relationship between the end coordinate system of the robot arm and the base coordinate system is obtained by calibrating the robot arm TCP Then based on the hand-eye calibration results And the pose relationship between the lower cover and the virtual multi-eye system Obtain the position relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base Contains the rotation matrix and translation vectors The specific formula is: in, and It is the conversion relationship between the threaded hole coordinate system of the lower cover and the camera coordinate system; and It is the transformation relationship between the robot arm end coordinate system and the base coordinate system; The pose relationship between the upper cover through-hole coordinate system and the robot end coordinate system, including the rotation matrix and translation vectors Based on Obtain the pose relationship between the upper cover through-hole coordinate system and the robot arm base coordinate system Contains the rotation matrix and translation vectors The specific formula is: in, and The transformation relationship between the robot arm end coordinate system and the base coordinate system; Knowing the above posture relationship, solve the new Contains the rotation matrix and translation vectors The end position of the robotic arm is adjusted accordingly to achieve multi-hole alignment of the upper and lower covers. The specific formula is: in, and It is the conversion relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base. and It is the conversion relationship between the robot arm end coordinate system and the upper cover through-hole coordinate system.

2. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S1, the specific contents of building a spatial positioning system based on monocular vision are: The system includes a robotic arm, a camera, a ring light source, a flange, an upper cover, a lower cover and a hand-eye calibration plate; the camera and the ring light source constitute the visual system of the system; The coordinate systems of the system are defined as follows: robot base coordinate system {b}, camera coordinate system {c}, robot end coordinate system {r}, upper cover through hole coordinate system {s}, lower cover threaded hole coordinate system {w}.

3. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S2, the system is calibrated to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system. The specific contents are as follows: System calibration includes camera calibration and hand-eye calibration; Camera calibration: The Zhang Zhengyou plane calibration method is used to ensure that the camera can accurately map the 3D space to the 2D image. The camera's geometric model parameters, including intrinsic parameters, extrinsic parameters, and distortion parameters, are established and solved. Hand-eye calibration: Set up a hand-eye calibration board, set up two types of array holes of different sizes on the calibration board, capture the array hole image with a camera, process the image and extract the center coordinates of the array hole; Given the spatial 3D coordinates of the array hole, the pixel coordinates of the center of the array hole image, and the camera model parameters, the transformation relationship from the manipulator end coordinate system to the camera coordinate system is obtained using an iterative method. Contains the rotation matrix and translation vectors 4. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 3, characterized in that: The specific contents of camera calibration are: Select a calibration plate with known size and evenly distributed dots. Use the camera to be calibrated to take multiple clear images from different angles and positions to ensure that the calibration plate is clear and complete in the field of view. Using Zhang Zhengyou's plane calibration method, we first use a corner detection algorithm specifically for dot patterns to detect the coordinates of the center of the dots on the calibration plate in the image. The calibration plate plane is set as the XY plane of the world coordinate system. A specific dot is selected as the origin of the world coordinate system, and the dot spacing is used as the unit length to establish the correspondence between the world coordinate system and the calibration plate. Solve the homography matrix and use it to calculate the initial values ​​of the camera's intrinsic parameters, and then calculate the extrinsic parameters; use a nonlinear optimization algorithm to optimize the intrinsic and extrinsic parameters; simultaneously calculate the camera's distortion parameters, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

5. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S32, center positioning: the threaded hole image is processed based on the ellipse contour extraction algorithm of the adjacent arc segment combination, the strong interference image noise near the threaded hole is eliminated, and the center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning. The specific content is: S321: performing conventional preprocessing on the acquired threaded hole image, including Gaussian filtering, median filtering, and Canny edge processing; S322: Extract threaded hole arc segments based on edges; S323: Using an adaptive threshold to remove short arcs, and constructing adjacency relationships through an adjacency matrix to complete edge segmentation and arc segment combination; S324: Fitting the ellipse image by the least square method, obtaining the coordinate information of the ellipse center, and verifying the fitting result.

6. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S42, the alignment problem is transformed into a registration optimization solution in which multiple rays pass through multiple points using the ray set and the 3D center point set of the threaded holes. The relationship between the camera coordinate system and the lower cover threaded hole coordinate system is obtained by iterative geometric solution in point-line registration. The specific content is: In the process of multi-point and multi-line point-line registration, the multi-points are defined by known 3D points, and the multi-rays are defined by starting points and direction vectors. The specific calculation formula is as follows: Among them, P is the matrix composed of the direction vector of the ray, O is the matrix composed of the starting point of the ray, S is the matrix composed of the three-dimensional points in the world coordinate system, and Z is the diagonal parameter matrix with the elements ζ1…ζ on the diagonal n For a line in parameterized space, c is a vector representing the position of the ray, is the rotation matrix, is the translation vector, α is the scaling factor; M is the 3D center point of the threaded hole; is the rotation matrix The transpose of p n T is the transpose of the vector in each direction; o n T is the transpose of each optical center vector; c T is the transpose of the ray position vector; M n T is the transpose of the three-dimensional center point vector of the threaded hole; 1c T is a matrix consisting of ray position vectors.

7. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 6, characterized in that: Use the iterative method to solve the point-line registration problem and calculate the pose relationship between the lower cover thread hole coordinate system and the camera coordinate system Contains the rotation matrix and translation vectors

Images