Threaded hole space positioning and multi-hole alignment method based on monocular vision

Abstract

The invention discloses a threaded hole space positioning and multi-hole alignment method based on monocular vision, and belongs to the technical field of visual inspection. The method comprises the following steps: S1, constructing a spatial positioning system based on monocular vision; s2, calibrating the system, and establishing a conversion relation between a camera coordinate system and an upper cover through hole coordinate system; s3, carrying out single threaded hole image acquisition and center positioning; s4, by means of the ray set and the threaded hole three-dimensional center point set, the pose solution is modeled into registration between multiple points and multiple lines, and the pose relation between the upper cover and the lower cover is obtained; and S5, aligning the through holes of the upper and lower covers with the threaded holes. According to the method, a multi-hole alignment problem is modeled into a multi-point and multi-line registration problem, and compared with a manual alignment mode, the total alignment angle deviation is controlled to be within 0.8 degree, and the total position deviation is controlled to be within 0.8 mm.

Description

Technical Field

[0001] The invention belongs to the technical field of visual detection, and in particular relates to a threaded hole spatial positioning and multi-hole alignment method based on monocular vision. Background Art

[0002] In the context of the continuous innovation of intelligent manufacturing technology, the aerospace field has gradually adopted industrial robotic arms to complete tasks such as target grasping, pin hole parts assembly and blade parts repair. In actual application scenarios, industrial robotic arms often rely on preset programs and lack adaptability to environmental changes, affecting their efficiency and accuracy in complex industrial environments.

[0003] In order to improve its intelligent assembly capabilities, integrating visual sensor technology into the robotic arm system is the key, which will enhance its perception and adaptability to the environment and achieve more efficient and accurate automated operations. Research on visual-assisted robotic arm assembly technology at home and abroad has reached a fairly mature level. In the existing technology, the visual system has been successfully integrated into the robotic arm system. Through the accurate perception of the surrounding environment by the visual system, the robotic arm can automatically identify the workpiece, accurately locate and efficiently complete the assembly task.

[0004] Among them, the vision-based manipulator spatial positioning system has cooperative and non-cooperative solutions when performing positioning tasks. In the cooperative solution, the visual system can obtain the accurate position and posture of the target through the cooperative target, and feed it back to the manipulator system for subsequent tasks. In the prior art, a multi-station alignment technology of a manipulator with near and far sight distance guidance is proposed. On the basis of completing the system calibration, a task table is established for the multi-station alignment operation of the manipulator, and then the three-stage automatic alignment of the end tool of the manipulator and the target is completed with the assistance of the cooperative target. However, in actual engineering applications, due to the limitations of small space and complex working conditions, cooperative targets are often not available. The non-cooperative solution uses the characteristics of the target itself to achieve positioning measurement. In order to capture random targets in space, a measurement and planning method for capturing non-cooperative targets by a space robot based on stereo vision is proposed in the prior art. The target is identified and measured through stereo vision, and the feature points are 3D reconstructed through the least squares method, so as to measure the position and posture of the target, and then the path and control strategy of the space robot are planned to achieve the capture of non-cooperative targets.

[0005] At present, the assembly positioning technology of pin-hole parts based on vision is still mainly used in the insertion of pins into holes, namely peg-holes. In order to solve the problem of large deviation between pins and holes before assembly, a method for autonomous alignment of pegs and holes in the robot assembly process is discussed in the prior art, which is based on force / torque measurement. In the prior art, when the visual system may not be available, the hole-shaft assembly is performed by force control guidance. Threaded holes are more difficult to handle than through holes, and through holes are easier to locate. There are relatively few studies on threaded hole image positioning at home and abroad. In the prior art, an assembly method based on vision and force information is studied to solve the problem of positioning and tightening threaded fasteners during assembly. In order to solve the problem of force impact generated by tightening bolts when switching from free space to constrained space, an elliptical arc fitting method and a three-point method are used to estimate the initial position and direction of the threaded hole. In the prior art, in order to solve the limitation problem of internal thread detection of mechanical products, a method for internal thread defect detection based on deep learning is studied, two methods for obtaining internal surface images of M8 nuts are introduced, and convolutional neural networks (CNNs) are applied to identify defects in captured images. In summary, there are relatively few studies on hole alignment in vision-based pin hole part assembly positioning, and most of them use a combination of vision and other technologies such as force control for positioning, which increases the complexity of the system; in addition, vision-based threaded hole image processing is difficult, and most of them will be combined with deep learning for threaded hole image processing, and then there are relatively few studies on spatial positioning using threaded holes. Unlike the common hole-axis alignment, which can be aligned by combining vision with force control search, the hole-hole alignment problem cannot be effectively assisted by force control, and due to the space limitations of the cabin, it is impossible to use visual cooperative targets for hole positioning.

[0006] In order to solve the hole alignment problem between through holes and threaded holes in a narrow cabin section, the present invention studies a technology for precise spatial positioning of threaded holes and multi-hole alignment between threaded holes and through holes by only utilizing the collaborative work of a visual system and a robotic arm. Summary of the invention

[0007] In response to the problems mentioned in the background technology, the present invention proposes a threaded hole spatial positioning and multi-hole alignment method based on monocular vision, and models the multi-hole alignment problem as a multi-point and multi-line point-line registration problem. Compared with the manual alignment method, the total alignment angle deviation is controlled within 0.8°, and the total position deviation is within 0.8mm.

[0008] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0009] A method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision comprises the following steps:

[0010] S1: Build a spatial positioning system based on monocular vision;

[0011] S2: calibrate the system and establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system;

[0012] S3: Single threaded hole image acquisition and center positioning;

[0013] S4: Using the ray set and the 3D center point set of the threaded hole, the pose solution is modeled as the registration between multiple points and multiple lines to obtain the pose relationship between the upper and lower covers;

[0014] S5: Align the through holes of the upper and lower covers with the threaded holes.

[0015] Preferably, in S1, the specific contents of constructing a spatial positioning system based on monocular vision are:

[0016] The system includes a robotic arm, a camera, an annular light source, a flange, an upper cover, a lower cover and a hand-eye calibration plate; the camera and the annular light source constitute the visual system of the system;

[0017] The coordinate systems of the system are defined as follows: robot base coordinate system {b}, camera coordinate system {c}, robot end coordinate system {r}, upper cover through hole coordinate system {s}, lower cover threaded hole coordinate system {w}.

[0018] Preferably, in S2, the system is calibrated to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system. Specifically, the content is:

[0019] System calibration includes camera calibration and hand-eye calibration;

[0020] Camera calibration: The Zhang Zhengyou plane calibration method is used to calibrate the camera to ensure that the camera can accurately map the three-dimensional space to the two-dimensional image, and to establish and solve the camera's geometric model parameters, including intrinsic parameters, extrinsic parameters, and distortion parameters;

[0021] Hand-eye calibration: Set up a hand-eye calibration board, set up two types of array holes of different sizes on the calibration board, use the camera to capture the array hole image, process the image and extract the center coordinates of the array hole;

[0022] Given the spatial 3D coordinates of the array holes, the center pixel coordinates of the array hole image, and the camera model parameters, the iterative method is used to find the transformation relationship from the robot end coordinate system to the camera coordinate system. Contains the rotation matrix and translation vectors

[0023] As a preference, the specific contents of camera calibration are:

[0024] Select a calibration plate with known size and uniform dot distribution, and use the camera to be calibrated to take multiple clear images from different angles and positions to ensure that the calibration plate is clear and complete in the field of view; use Zhang Zhengyou's plane calibration method, first use the corner point detection algorithm specifically for dot patterns to detect the coordinates of the center of the dots on the calibration plate in the image; set the calibration plate plane as the XY plane of the world coordinate system, select a specific dot as the origin of the world coordinate system, and use the dot spacing as the unit length to establish the correspondence between the world coordinate system and the calibration plate;

[0025] Solve the homography matrix, and use it to calculate the initial value of the camera's intrinsic parameters, and then calculate the extrinsic parameters; use a nonlinear optimization algorithm to optimize the intrinsic and extrinsic parameters; at the same time, calculate the camera's distortion parameters, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

[0026] Preferably, in S3, the specific contents of performing single threaded hole image acquisition and center positioning are:

[0027] S31: Single threaded hole image acquisition: Control the robotic arm to move the camera to the top of the threaded hole of the lower cover, take images of the threaded holes in sequence and record the position of the robotic arm;

[0028] S32: Center positioning: The threaded hole image is processed based on the ellipse contour extraction algorithm based on the combination of adjacent arc segments, the strong interference image noise near the threaded hole is eliminated, and the center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning.

[0029] Preferably, in S32, center positioning: the threaded hole image is processed based on an ellipse contour extraction algorithm based on a combination of adjacent arc segments, strong interfering image noise near the threaded hole is removed, and the center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning. The specific contents are:

[0030] S321: performing conventional preprocessing on the acquired threaded hole image, including Gaussian filtering, median filtering and Canny edge processing;

[0031] S322: extracting threaded hole arc segments based on edges;

[0032] S323: using an adaptive threshold to remove short arcs, and constructing adjacency relationships through an adjacency matrix to complete edge segmentation and arc combination;

[0033] S324: Fitting the ellipse image by the least square method, obtaining the coordinate information of the center of the ellipse, and verifying the fitting result.

[0034] Preferably, in S4, the pose solution is modeled as a registration between multiple points and multiple lines using a set of rays and a set of three-dimensional center points of the threaded holes, and the specific content of obtaining the pose relationship between the upper and lower covers is:

[0035] S41: Based on the visual imaging geometry, construct the ray equation passing through the center of the ellipse in the camera coordinate system;

[0036] First, under the ideal pinhole imaging model, a single threaded hole imaging model is built; by photographing the single threaded hole target in sequence, a ray set can be obtained;

[0037] S42: Using the ray set and the 3D center point set of the threaded hole, the alignment problem is transformed into a registration optimization solution of multiple rays passing through multiple points, and then the relationship between the camera coordinate system and the coordinate system of the threaded hole of the lower cover is obtained by the iterative geometric solution method in the point-line registration.

[0038] Preferably, in S42, the alignment problem is converted into a registration optimization solution of multiple rays passing through multiple points by using a set of rays and a set of three-dimensional center points of the threaded holes, and then the relationship between the camera coordinate system and the coordinate system of the threaded holes of the lower cover is obtained by an iterative geometric solution method in point-line registration. The specific content is:

[0039] In the process of point-line registration of multiple points and multiple lines, multiple points are defined by known three-dimensional space points, and multiple rays are defined by starting points and direction vectors. The specific calculation formula is as follows:

[0040]

[0041] Among them, P is the matrix composed of the direction vector of the ray, O is the matrix composed of the starting point of the ray, S is the matrix composed of the three-dimensional points in the world coordinate system, and Z is the diagonal parameter matrix, with the elements ζ1…ζ on the diagonal n For a line in parameterized space, c is a vector representing the position of the ray, is the rotation matrix, is the translation vector, α is the scaling factor; M is the three-dimensional center point of the threaded hole; is the rotation matrix The transpose of n T is the transpose of the vector in each direction; o n T is the transpose of each optical center vector; c T is the transpose of the ray position vector; M n T is the transpose of the three-dimensional center point vector of the threaded hole; 1c T is a matrix consisting of ray position vectors;

[0042] As a preferred method, an iterative method is used to solve the point-line registration problem and calculate the pose relationship between the threaded hole coordinate system of the lower cover and the camera coordinate system. Contains the rotation matrix and translation vectors

[0043] Preferably, in S5, the specific content of aligning the through holes of the upper and lower covers with the threaded holes is:

[0044] The position relationship between the robot end coordinate system and the base coordinate system is obtained by calibrating the robot TCP Based on the hand-eye calibration results obtained above And the pose relationship from the cover to the virtual multi-eye system Obtain the position relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base Contains the rotation matrix and translation vectors The specific formula is:

[0045]

[0046] in, and It is the transformation relationship from the threaded hole coordinate system of the lower cover to the camera coordinate system; and It is the transformation relationship between the robot end coordinate system and the base coordinate system;

[0047] The pose relationship between the upper cover through-hole coordinate system and the robot end coordinate system, including the rotation matrix and translation vectors Based on Obtain the position relationship between the upper cover through-hole coordinate system and the robot base coordinate system Contains the rotation matrix and translation vectors The specific formula is:

[0048]

[0049] in, and The transformation relationship between the robot end coordinate system and the base coordinate system;

[0050] Knowing the above posture relationship, solve the new Contains the rotation matrix and translation vectors The end position of the robot arm is adjusted accordingly to achieve multi-hole alignment of the upper and lower covers. The specific formula is:

[0051]

[0052] in, and It is the conversion relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base. and It is the conversion relationship between the robot end coordinate system and the upper cover through-hole coordinate system.

[0053] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0054] (1) In the process of robot automatic assembly of electrical control devices on missile compartments, it is necessary to accurately locate and align multiple through holes and multiple threaded holes on the upper and lower covers of the control housing. The hole-hole alignment problem cannot be effectively assisted by force control, and due to the space limitations of the compartment, it is also impossible to locate the holes with the help of visual cooperative targets. For this reason, the present invention proposes a threaded hole spatial positioning and multi-hole alignment technology based on monocular vision. The core idea of ​​this technology is to model the multi-hole alignment problem as a multi-point and multi-line registration problem.

[0055] (2) The present invention proposes a method of hand-eye calibration based on a mechanical transmission hand-eye calibration plate for "two sides and one hole" positioning to obtain a more accurate hand-eye relationship for subsequent multi-hole positioning and alignment.

[0056] (3) The present invention adopts an ellipse extraction algorithm based on a combination of adjacent arc segments, which can effectively remove the strong interference image noise near the threaded hole, and obtain the center coordinates of the threaded hole image for spatial positioning through edge robust fitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a working site schematic diagram of the present invention;

[0058] Figure 2 It is a schematic diagram of the structure of the positioning system of the present invention;

[0059] Figure 3 It is a flow chart of the system technology route of the present invention;

[0060] Figure 4 is a schematic diagram of hand-eye calibration of the present invention;

[0061] Figure 5 1 is a schematic diagram of processing a threaded hole image of the present invention, wherein FIG. (a) is an original threaded hole image; FIG. (b) is a target contour; FIG. (c) is a preprocessed image; FIG. (d) is arc segment extraction; FIG. (e) is elimination of too short arc segments; and FIG. (f) is a result image;

[0062] Figure 6 It is the threaded hole imaging modeling diagram of the present invention;

[0063] Figure 7 Schematic diagram of a virtual rigid connection multi-camera system of the present invention;

[0064] Figure 8 It is a schematic diagram of the experimental working scene of the present invention;

[0065] Fig. 9It is a schematic diagram of the state where the upper and lower covers of the present invention are aligned with the holes;

[0066] Fig.10 It is a schematic diagram of the bolts of the present invention being screwed into the holes;

[0067] Fig.11 It is a schematic diagram of the posture deviation of the present invention, wherein Figure (a) is the angular deviation around the x, y, and z axes, and Figure (b) is the position deviation along the x, y, and z axes. DETAILED DESCRIPTION

[0068] The present invention is further illustrated below in conjunction with specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0069] The threaded hole spatial positioning and multi-hole alignment method provided in this embodiment based on monocular vision includes the following steps:

[0070] S1: Build a spatial positioning system based on monocular vision;

[0071] The hole alignment problem mentioned in the present invention originates from the automatic robot assembly of the control device on the missile compartment, which requires precise positioning and alignment between multiple through holes and multiple threaded holes on the upper and lower covers of the controller housing (such as a servo thermal battery).

[0072] like Figure 1 , Figure 2 As shown, this embodiment constructs a monocular vision-based spatial positioning system in the form of Eye-in-hand. The system includes a robotic arm, a camera, a ring light source, a flange, an upper cover, a lower cover and a high-precision hand-eye calibration plate; the camera and the ring light source constitute the visual system of the system; in the laboratory scene, the upper and lower covers are both simulated samples; the flange is used to fix the end of the robotic arm to the upper cover; the pose relationship between the camera coordinate system and the coordinate system of the end of the robotic arm is required in the pose solution process, so in this embodiment, a hand-eye calibration plate with high-precision array holes is constructed for high-precision hand-eye calibration, and the distance from the camera to the array holes on the hand-eye calibration plate is similar to the distance for normal acquisition of single-hole images. The upper cover is fixed to the flange at the end of the robotic arm, and the camera is installed in the middle of the upper cover. When performing hand-eye calibration, the hand-eye calibration plate is installed under the camera with an adapter plate.

[0073] The coordinate systems of the system are defined as follows: robot base coordinate system {b}, camera coordinate system {c}, robot end coordinate system {r}, upper cover through hole coordinate system {s}, lower cover threaded hole coordinate system {w}. Refers to the position and posture relationship between the robot end coordinate system and the base coordinate system; It refers to the position relationship between the robot end coordinate system and the camera coordinate system, which is defined as the hand-eye relationship in this embodiment; Refers to the position relationship between the threaded hole coordinate system of the lower cover and the camera coordinate system. Refers to the position relationship between the upper cover through-hole coordinate system and the robot base coordinate system; Refers to the position relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base.

[0074] S2: calibrate the system and establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system;

[0075] like Figure 3 The following is the specific technical route of the system.

[0076] Calibration is extremely important in the visual positioning system, and this embodiment mainly includes camera calibration and high-precision hand-eye calibration. Camera calibration provides accurate camera model parameters, which is the prerequisite for subsequent hand-eye calibration and threaded hole image processing. Hand-eye calibration is performed using a high-precision hand-eye calibration board. The hand-eye calibration is to obtain the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system. The conversion to the final robotic arm alignment requires obtaining the position relationship between the camera coordinate system and the robotic arm end coordinate system.

[0077] Camera calibration ensures that the camera can accurately map the three-dimensional space to the two-dimensional image, and establishes and solves the geometric model parameters of the camera, including intrinsic parameters, extrinsic parameters and distortion parameters. In this embodiment, the classic and effective Zhang Zhengyou plane calibration method is used for calibration.

[0078] Camera calibration requires preparation work, including selecting a calibration plate with known size and uniform dot distribution, accurately measuring key information such as dot spacing (the dot spacing in this embodiment is 10mm), and taking at least 10 clear images from different angles and positions with the camera to be calibrated to ensure that the calibration plate is complete and without blur or occlusion in the field of view. Using Zhang Zhengyou's plane calibration method, first use a corner detection algorithm specifically for dot patterns to detect the coordinates of the center of the dots on the calibration plate in the image. Set the calibration plate plane to the XY plane of the world coordinate system, select a specific dot (such as the dot in the upper left corner) as the origin of the world coordinate system, and use the dot spacing as the unit length to establish the correspondence between the world coordinate system and the calibration plate. Then solve the homography matrix, and calculate the initial value of the camera's intrinsic parameters based on this, and then calculate the external parameters. Then use a nonlinear optimization algorithm to optimize the internal and external parameters. At the same time, calculate the distortion parameters of the camera, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

[0079] Hand-eye calibration is an important foundation for the visual spatial positioning technology of threaded holes. In the Eye-in-hand system of this embodiment, the end of the robot arm is the "hand" and the camera is the "eye". The purpose of hand-eye calibration is to establish the relationship between the coordinate system of the end of the robot arm and the coordinate system of the camera. The schematic diagram of hand-eye calibration is as follows: Figure 4 shown.

[0080] In this embodiment, the following is set Figure 4 The high-precision hand-eye calibration plate shown. Two types of array holes of different sizes are set on the calibration plate, 9 large array holes with a diameter of 10mm and 4 small array holes with a diameter of 8mm. The positional relationship between the array holes is known, and the accuracy is guaranteed by machining. In order to avoid uneven lighting, the array hole image at the edge is not good. For this reason, the small-sized array holes are set in the central area with good lighting conditions to improve the accuracy of elliptical feature extraction. Generally speaking, there is a pin hole on the end flange of the robot arm for precise positioning. In order to ensure the accuracy of positioning, the "two holes and one face" positioning method is adopted in this embodiment, and the threaded connection at the end is changed to threaded pin positioning to achieve precise positioning of the end workpiece flange. In addition, in this embodiment, two pin holes are designed at the connection of the workpieces related to the hand-eye calibration plate for precise positioning between the workpieces, and finally the positional relationship transmission from the end of the robot arm to the array holes on the hand-eye calibration plate is realized.

[0081] When the system performs hand-eye calibration, the calibration plate is installed under the fixture and the above-mentioned pin hole positioning is used to ensure its assembly accuracy, so the coordinates of the array hole in the robot end coordinate system are known. The camera captures the array hole image, processes the image and extracts the center coordinates of the array hole. Given the spatial three-dimensional coordinates of the array hole, the center pixel coordinates of the array hole image and the camera model parameters, the pose relationship between the robot end coordinate system and the camera coordinate system is solved, which can be summarized as a PnP problem (Perspective-n-Point).

[0082] The PnP problem involves determining the pose of a three-dimensional object relative to the camera coordinate system from a set of points in three-dimensional space and their two-dimensional projections on the image. Among the many methods for solving the PnP problem, the iterative method can handle any number of feature points and gradually optimize the camera pose estimation through a continuous iterative process. This method is not only highly flexible, but also can provide high accuracy. For the hand-eye calibration plate of this embodiment, which has a large number of feature points, these characteristics of the iterative method are particularly important. Using the iterative method, the transformation relationship from the robot end coordinate system to the camera coordinate system is obtained. Contains the rotation matrix and translation vectors

[0083] Specifically, an initial estimate is first performed. The rotation and translation parameters can be preliminarily determined based on geometric relationships. For example, the translation direction can be roughly determined first, and the rotation is first set to the unit matrix. Then, an error function is constructed. According to the perspective projection model, the points in the world coordinate system and the points in the camera coordinate system are associated through rotation and translation. Then, according to the relationship between the pixel coordinates in the camera model and the points in the camera coordinate system, the sum of the projection errors of all spatial points in the world coordinate system is defined as the error function. The error here is the difference between the actual pixel coordinates and the projected pixel coordinates calculated by the current estimated rotation and translation. Then, an iterative update is performed to obtain the Jacobian matrix by calculating the partial derivatives of the error function with respect to the rotation and translation parameters. Each iteration calculates the Jacobian matrix and error vector based on the current parameter estimates, and updates the parameters through a specific calculation method. When updating the rotation parameters, it is necessary to ensure that they are orthogonal matrices. Finally, the termination condition is used to determine whether convergence has occurred. When the parameter update amount is small enough, the iteration ends. For example, for the calculated rotation matrix By calculating the rotation matrix obtained by the previous iteration The Frobenius norm (F-norm) between , defined when When , the update amount of the rotation parameter is considered to be small enough, where ε1 is a pre-set small positive number. The specific value can be determined according to the accuracy requirements of the actual application. In computer vision tasks with high requirements for posture accuracy, the value may be 10 -4 or smaller; for the calculated translation parameters By calculating the translation parameter obtained by the previous iteration The Euclidean norm between When , it is considered that the update amount of the translation parameter is small enough, and ε2 can be taken as 10 -6 Or smaller. The iteration ends, and the rotation and translation parameters at this time are the final rotation matrix and translation vectors

[0084] S3: Single threaded hole image acquisition and center positioning;

[0085] S31: Single threaded hole image acquisition: Control the robotic arm to move the camera to the top of the threaded hole of the lower cover, take four threaded hole images in sequence and record the robotic arm position;

[0086] like Figure 5 As shown in (a) in the figure, the threaded hole of the lower cover is photographed by a camera to obtain the threaded hole image. As can be seen from the figure, the threaded hole image is affected by factors such as lighting conditions, hole edge chamfers, and thread outer teeth. Accurately obtaining the edge contour of the threaded hole image and the coordinate information of the ellipse center is a key link in the high-precision spatial positioning of the threaded hole. Under lighting conditions, a bright ring will appear on the thread of the lower cover threaded hole, such as Figure 5 The part pointed by the arrow in (b) is the threaded hole edge contour target that needs to be extracted in this embodiment.

[0087] S32: Center positioning: Process the collected threaded hole image to extract the sub-pixel coordinates of the center of the single hole image;

[0088] The threaded hole image is processed based on the ellipse contour extraction algorithm of adjacent arc segment combination to remove the strong interference image noise near the threaded hole. The center coordinates of the threaded hole image are obtained for spatial positioning through edge robust fitting.

[0089] S321: performing conventional preprocessing on the acquired threaded hole image, including Gaussian filtering, median filtering and Canny edge processing;

[0090] First, the threaded hole image is preprocessed conventionally, including Gaussian filtering, median filtering, and Canny edge processing. The preprocessed image is as follows: Figure 5 As shown in (c), the edges of the threaded hole image are complex and it is difficult to extract a complete and valid ellipse image.

[0091] S322: extracting threaded hole arc segments based on edges;

[0092] like Figure 5 As shown in (d) in the figure, the threaded hole arc segment is extracted based on the edge; the image is first subjected to Canny edge detection, and then the image is thickened to facilitate finding the start and end points of the edge contour.

[0093] S323: using an adaptive threshold to remove short arcs, and constructing adjacency relationships through an adjacency matrix to complete edge segmentation and arc combination;

[0094] like Figure 5 As shown in (e) in the figure, edge segmentation is performed based on the concavity and curvature of the elliptical image. After completing the edge sub-arc segmentation, an adaptive threshold is determined, arcs that are too short and have a length less than the threshold are removed, and the remaining sub-arcs are sorted. The adjacency relationship of each sub-arc is constructed through the adjacency matrix, and the arcs are combined and clustered according to the arc search area restrictions and edge curvature restrictions.

[0095] S324: fitting the ellipse image by the least square method, obtaining the coordinate information of the center of the ellipse, and verifying the fitting result;

[0096] The fitting results are verified by four indicators: weight, shape, position and gradient. After completing the arc segment clustering, the candidate elliptical arc segment set can be fitted to the elliptical image by the least squares method.

[0097] In addition, the post-processing method of eliminating the ambiguity of the ellipse image and removing the false ellipse caused by noise can effectively improve the recognition accuracy of the threaded hole. Through the above steps, the information such as the ellipse and the two-dimensional coordinates of the center of the threaded hole image can be effectively extracted. Figure 5 (f) as shown.

[0098] S4: Using the ray set and the 3D center point set of the threaded hole, the pose solution is modeled as the registration between multiple points and multiple lines to obtain the pose relationship between the upper and lower covers;

[0099] S41: Based on the visual imaging geometry, construct the ray equation passing through the center of the ellipse in the camera coordinate system;

[0100] The traditional PnP method takes a single photo of all features in a visual coordinate system and calculates the target pose information through the correspondence between two-dimensional and three-dimensional information. Due to the limitation of the narrow shell space, the threaded holes can only be photographed one by one by moving the robot arm. The resulting problem is that all threaded hole images are obtained in their respective visual coordinate systems.

[0101] First, a single threaded hole imaging model is built under the ideal pinhole imaging model. The three-dimensional center point M of the threaded hole i The projection on the camera imaging plane is m i ,Depend on Figure 6 It can be concluded that there is a ray between the camera optical center, the plane projection point and the three-dimensional point. i is the starting point, the ray direction vector p i can be obtained by back projection, and the intrinsic parameter matrix A can be obtained by camera calibration, as shown in formula (1).

[0102]

[0103] By shooting the single threaded hole target in sequence, a ray set can be obtained. Where α is the focal length of the camera in the X-axis direction, β is the focal length of the camera in the Y-axis direction; u 0 and v 0 is the principal point position, i.e. the position of the center of the image plane in the pixel coordinate system; p' is the unnormalized direction vector; u i and v i represents the center coordinates of the threaded hole image; p i Represents the normalized direction vector; x, y, and z are the component values ​​of the direction vector in the back-projection calculation.

[0104] S42: construct virtual multi-eye vision geometry according to the moving posture of the robot arm, and obtain the relationship between the camera coordinate system and the coordinate system of the threaded hole of the lower cover;

[0105] By expressing the central rays of each threaded hole image obtained by visual sequential photography in the camera coordinate system, a virtual rigid connection multi-eye system can be established; the three-dimensional coordinates of the center of the threaded hole are obtained according to the digital model of the lower cover, and then the hole alignment problem is modeled as a multi-point and multi-line registration problem, and then the pose of the lower cover threaded hole coordinate system to the virtual multi-eye camera coordinate system is obtained by the iterative geometric solution method in the point-line registration, that is, If the point-line registration cannot converge, that is, the iteratively calculated tolerance Tol' cannot be lower than the set value Tol, it is necessary to re-shoot and process the threaded hole image.

[0106] This embodiment uses the camera posture of the first threaded hole as a reference, and uses the monocular vision and the known hand-eye relationship between the end of the robot arm. The conversion relationship between the coordinate system of the end of the robot and the coordinate system of the base is obtained, and the conversion relationship between the monocular camera of different postures and the reference camera position is obtained. The camera coordinate system of the remaining postures is converted to the reference camera coordinate system. The formula is as follows:

[0107]

[0108] in, and It is the transformation relationship between the robot end coordinate system and the base coordinate system under the reference camera posture; and It is the conversion relationship between the robot end coordinate system and the base coordinate system under the camera posture when shooting the No. 2 threaded hole. and It is the conversion relationship between the robot end coordinate system and the base coordinate system under the camera posture when shooting the No. 2 threaded hole; and It is the transformation relationship between the camera coordinate system and the robot end coordinate system; yes The transpose of yes The transpose of .

[0109] use and The camera coordinate system when shooting the second threaded hole can be converted to the reference camera coordinate system. The camera coordinate system when shooting the remaining threaded holes can be converted to the reference camera coordinate system using a similar formula. The camera coordinate systems of different positions can be connected, that is, the ray set also establishes a rigid connection. In this way, a virtual rigid connection multi-camera system can be constructed, such as Figure 7 shown.

[0110] In order to obtain the pose relationship between the camera system and the lower cover, the pose solution problem can be modeled as a registration problem between multiple points and multiple lines, namely, a point-line registration problem, by using a set of rays and a set of three-dimensional center points of threaded holes.

[0111] Multi-point and multi-line point-line registration generalizes the traditional PnP problem to NPnP (Non-Perspective-n-Point), and generalizes perspective cameras to non-perspective cameras, that is, camera models whose projection rays do not intersect at a single point. This includes all non-central devices and multi-camera systems, where the cameras are rigidly connected, so the lines are also a set of rigidly connected rays. However, the overall set of rays does not intersect at a single point, that is, it is not required that these rays form a beam.

[0112] In the process of point-line registration of multiple points and multiple lines, multiple points are defined by known three-dimensional space points, and multiple rays are defined by starting points and direction vectors. The goal is to find the rotation matrix, translation vector and possible scaling factor of this set of three-dimensional space point coordinate system to the rigidly connected multi-camera system coordinate system, so that these points and rays match as much as possible after transformation. The formula is as follows:

[0113]

[0114] In the formula, P is a matrix consisting of the direction vectors of the rays, O is a matrix consisting of the starting points of the rays, S is a matrix consisting of the three-dimensional points in the world coordinate system, and Z is a diagonal parameter matrix with the elements ζ1…ζ on the diagonal n For a line in parameterized space, c is a vector representing the position of the ray, is the rotation matrix, is the translation vector, α is the scaling factor. M is the 3D center point of the threaded hole; is the rotation matrix The transpose of n T is the transpose of the vector in each direction; o n T is the transpose of each optical center vector; c T is the transpose of the ray position vector; M n T is the transpose of the three-dimensional center point vector of the threaded hole; 1c T is a matrix consisting of ray position vectors.

[0115] There are generally two solutions to solve the point-line registration problem. One is a direct closed-form solution, which requires at least six sets of points and lines and minimizes the algebraic error. The other is an iterative solution, which starts with information-free initialization, obtains the minimum geometric error in 3D space, calculates the tolerance value Tol', converges when it is lower than the set tolerance value Tol, and can use a minimum of four sets of points and lines. The four threaded holes in this embodiment can just provide a minimum of four sets of points and lines. Therefore, this embodiment uses an iterative solution to calculate the pose relationship between the coordinate system of the threaded hole of the lower cover and the coordinate system of the camera. Contains the rotation matrix and translation vectors

[0116] Specifically, the diagonal matrix Z is initialized first, and then the rotation matrix is ​​calculated to align the 3D point with the direction of the line; then a scale factor α is determined to adjust the size of the 3D point to match the length of the line segment; next, a translation vector is calculated to move the 3D point along the rotated line direction to best match the position of the line segment; after the rotation matrix, translation vector and scale factor are known, the parameters of each line segment are updated to determine the optimal position of the 3D point on each line; the above steps are repeated until the change in the line segment parameters is very small, and a stable rotation matrix and translation vector are obtained.

[0117] S5: Align the through holes of the upper and lower covers with the threaded holes;

[0118] With the help of the hand-eye relationship, the coordinate systems of the upper and lower covers are transformed to the end of the robotic arm to obtain the posture deviation, thereby guiding the robotic arm to achieve multi-hole alignment.

[0119] Pose solution:

[0120] The ultimate goal of this embodiment is to obtain the pose relationship between the camera coordinate system and the upper cover through-hole coordinate system and the lower cover threaded hole coordinate system through the guidance of monocular vision, so as to achieve precise alignment between the four through-holes of the upper cover and the four threaded holes of the lower cover. By using the pose relationship between the coordinate systems that have been obtained, the connection established between the vision and the upper and lower covers is converted to the robot base coordinate system, and the robot arm fixed to the upper cover is controlled to achieve hole-hole alignment with the lower cover.

[0121] Through the robot TCP calibration, the position relationship between the robot end coordinate system and the base coordinate system can be obtained. Based on the hand-eye calibration results obtained above And the pose relationship from the cover to the virtual multi-eye system The position relationship of the threaded hole coordinate system of the lower cover relative to the coordinate system of the robot base can be obtained (including rotation matrix and translation vectors ).

[0122]

[0123] in, and It is the transformation relationship from the threaded hole coordinate system of the lower cover to the camera coordinate system; and It is the transformation relationship between the robot end coordinate system and the base coordinate system.

[0124] It should be noted that this embodiment can ensure the position relationship (including the rotation matrix) between the upper cover through-hole coordinate system and the robot end coordinate system through high-precision machining and pin hole positioning. and translation vectors ), then based on The position relationship between the upper cover through-hole coordinate system and the robot base coordinate system can be obtained (including rotation matrix and translation vectors ).

[0125]

[0126] in, and The transformation relationship between the robot end coordinate system and the base coordinate system.

[0127] Since this embodiment realizes alignment by controlling the robot arm, the above-mentioned posture relationship is known, and a new (including rotation matrix and translation vectors ) and adjust the end position of the robot arm accordingly to achieve holes-holes multi-hole alignment of the upper and lower covers.

[0128]

[0129] in, and It is the conversion relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base. and It is the conversion relationship between the robot end coordinate system and the upper cover through-hole coordinate system.

[0130] S6: Experimental verification of multi-hole alignment.

[0131] In order to verify the effectiveness of the vision-based threaded hole spatial positioning and alignment method in this paper, a Figure 8 The working scene shown.

[0132] The system uses the UR10 robot as the actuator to guide the visual system to perform image acquisition and hole-hole positioning alignment. The monocular camera used is the MER2-502-79U3M camera of the Daheng series, with a resolution of 2448 (H) × 2048 (V) and a 16mm lens. The inner diameters of the four threaded holes in the lower cover are all M7, and the diameter of the through hole in the upper cover is Φ9.

[0133] Before the robotic arm guides the monocular camera to collect threaded hole images, the system has completed camera calibration and hand-eye calibration and obtained the corresponding posture relationship.

[0134] 1. Alignment process and result experimental verification:

[0135] First, the robot guides the monocular camera to take pictures of threaded holes 1 to 4 in sequence. During the shooting process, the images are processed to obtain the center coordinates of the threaded hole images. After shooting all the threaded holes, the position relationship of the lower cover relative to the robot base coordinate system can be calculated within a few seconds. Then, the position of the end of the robot arm is adjusted so that the through holes of the upper cover and the threaded holes of the lower cover are aligned holes-holes.

[0136] When the upper and lower covers are aligned, the system status is as follows: Fig. 9 In the actual assembly process, bolts need to be screwed in later to fasten the upper and lower covers. In this embodiment, the bolts are screwed in manually. For easy observation, a gap is left between the two sides of the upper and lower covers during alignment. The screwed-in state of each hole is shown in FIG. Fig.10 shown.

[0137] 2. Repeatability experimental verification:

[0138] Since there is margin in the diameters of the through holes and threaded holes of the upper and lower covers, there are multiple sets of postures that can satisfy the alignment of the holes of the upper and lower covers. Therefore, the effectiveness of the direction of this embodiment is further verified by comparing it with the manual alignment method.

[0139] In this experiment, the lower cover is placed in three different positions, and three sets of desired postures of the end of the robot arm are obtained by manual alignment. The requirement of manual alignment is that all bolts can be screwed in smoothly, and there is no obvious deviation in the center of the through hole and the threaded hole when observed with the naked eye. By the method of this embodiment, the above alignment experiment (alignment process and result experiment) is independently carried out at three different lower cover positions, and compared with the expected posture of manual alignment, and the results are shown in Table 1. Among them, Δα, Δβ and Δγ are the angular deviations around the X, Y, and Z axes, respectively, and ΔX, ΔY and ΔZ are the position deviations along the X, Y, and Z axes, respectively.

[0140] Table 1. Pose deviation of repeated experiments

[0141] Groups Δα / (°) Δβ / (°) Δγ / (°) ΔX / mm ΔY / mm ΔZ / mm 1 -0.389 -0.427 -0.183 -0.403 0.041 0.482 2 -0.482 0.275 -0.475 0.453 0.122 0.027 3 0.159 0.413 0.028 0.420 -0.516 0.383

[0142] When the lower cover is in position 1, the total angle deviation of the alignment is 0.606° and the total position deviation is 0.629mm. When the lower cover is in position 2, the total angle deviation of the alignment is 0.730° and the total position deviation is 0.470mm. When the lower cover is in position 3, the total angle deviation of the alignment is 0.443° and the total position deviation is 0.768mm. Fig.11 It can be seen that the angular deviation around each axis during alignment is less than 0.5°, and the position deviation along each axis is about 0.5 mm.

[0143] It should be further explained that, in the actual assembly process, whether the bolts can be screwed into the aligned threaded holes is the basic standard. In the above experiment, the four threaded holes of the lower cover can all be screwed into with bolts after alignment.

[0144] Aiming at the problem that multiple through holes and threaded holes of upper and lower workpieces need to be aligned simultaneously in the automatic assembly environment of non-cooperative target robots, the present invention has built a spatial positioning system based on monocular vision. The hole alignment accuracy requirements of Φ9 through holes and M7 threaded holes can be achieved, and finally the bolts can be screwed in smoothly. This system realizes hole alignment in a purely visual way, and its structure is simple, flexible, and adaptable, and it has important value in practical engineering applications.

[0145] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision, characterized in that: The following steps are involved: S1: Build a spatial positioning system based on monocular vision; S2: calibrate the system and establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system; S3: Single threaded hole image acquisition and center positioning; S4: Using the ray set and the 3D center point set of the threaded hole, the pose solution is modeled as the registration between multiple points and multiple lines to obtain the pose relationship between the upper and lower covers; S5: Align the through holes of the upper and lower covers with the threaded holes.

2. The method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision according to claim 1, characterized in that: In S1, the specific contents of building a spatial positioning system based on monocular vision are: The system includes a robotic arm, a camera, an annular light source, a flange, an upper cover, a lower cover and a hand-eye calibration plate; the camera and the annular light source constitute the visual system of the system; The coordinate systems of the system are defined as follows: robot base coordinate system {b}, camera coordinate system {c}, robot end coordinate system {r}, upper cover through hole coordinate system {s}, lower cover threaded hole coordinate system {w}.

3. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S2, the system is calibrated to establish the conversion relationship between the camera coordinate system and the upper cover through-hole coordinate system. The specific contents are: System calibration includes camera calibration and hand-eye calibration; Camera calibration: The Zhang Zhengyou plane calibration method is used to calibrate the camera to ensure that the camera can accurately map the three-dimensional space to the two-dimensional image, and to establish and solve the camera's geometric model parameters, including intrinsic parameters, extrinsic parameters, and distortion parameters; Hand-eye calibration: Set up a hand-eye calibration board, set up two types of array holes of different sizes on the calibration board, use the camera to capture the array hole image, process the image and extract the center coordinates of the array hole; Given the spatial 3D coordinates of the array holes, the center pixel coordinates of the array hole image, and the camera model parameters, the iterative method is used to find the transformation relationship from the robot end coordinate system to the camera coordinate system. Contains the rotation matrix and translation vectors 4. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 3, characterized in that: The specific contents of camera calibration are: Select a calibration plate with known size and evenly distributed dots, and use the camera to be calibrated to take multiple clear images from different angles and positions to ensure that the calibration plate is clear and complete in the field of view; The Zhang Zhengyou plane calibration method is adopted. First, the coordinates of the center of the dots on the calibration plate in the image are detected using a corner detection algorithm specifically for dot patterns. The plane of the calibration plate is set as the XY plane of the world coordinate system, a specific dot is selected as the origin of the world coordinate system, and the dot spacing is used as the unit length to establish the corresponding relationship between the world coordinate system and the calibration plate. Solve the homography matrix, and use it to calculate the initial value of the camera's intrinsic parameters, and then calculate the extrinsic parameters; use a nonlinear optimization algorithm to optimize the intrinsic and extrinsic parameters; at the same time, calculate the camera's distortion parameters, including radial distortion and tangential distortion, and perform corresponding distortion compensation.

5. The method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision according to claim 1, characterized in that: In S3, the specific contents of single threaded hole image acquisition and center positioning are as follows: S31: Single threaded hole image acquisition: Control the robotic arm to move the camera to the top of the threaded hole of the lower cover, take images of the threaded holes in sequence and record the position of the robotic arm; S32: Center positioning: The threaded hole image is processed based on the ellipse contour extraction algorithm based on the combination of adjacent arc segments, the strong interference image noise near the threaded hole is eliminated, and the center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning.

6. The method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision according to claim 5, characterized in that: In S32, center positioning: the threaded hole image is processed based on the ellipse contour extraction algorithm of adjacent arc segment combination, the strong interference image noise near the threaded hole is eliminated, and the center coordinates of the threaded hole image are obtained through edge robust fitting for spatial positioning. The specific contents are as follows: S321: performing conventional preprocessing on the acquired threaded hole image, including Gaussian filtering, median filtering and Canny edge processing; S322: extracting threaded hole arc segments based on edges; S323: using an adaptive threshold to remove short arcs, and constructing adjacency relationships through an adjacency matrix to complete edge segmentation and arc combination; S324: Fitting the ellipse image by the least square method, obtaining the coordinate information of the center of the ellipse, and verifying the fitting result.

7. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S4, the pose solution is modeled as the registration between multiple points and multiple lines using the ray set and the 3D center point set of the threaded hole. The specific content of the pose relationship between the upper and lower covers is obtained as follows: S41: Based on the visual imaging geometry, construct the ray equation passing through the center of the ellipse in the camera coordinate system; First, under the ideal pinhole imaging model, a single threaded hole imaging model is built; by photographing the single threaded hole target in sequence, a ray set can be obtained; S42: Using the ray set and the 3D center point set of the threaded hole, the alignment problem is transformed into a registration optimization solution of multiple rays passing through multiple points, and then the relationship between the camera coordinate system and the coordinate system of the threaded hole of the lower cover is obtained by the iterative geometric solution method in the point-line registration.

8. The method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision according to claim 7, characterized in that: In S42, the alignment problem is transformed into a registration optimization solution of multiple rays passing through multiple points by using the ray set and the threaded hole three-dimensional center point set. Then, the relationship between the camera coordinate system and the lower cover threaded hole coordinate system is obtained by the iterative geometric solution method in point-line registration. The specific content is: In the process of point-line registration of multiple points and multiple lines, multiple points are defined by known three-dimensional space points, and multiple rays are defined by starting points and direction vectors. The specific calculation formula is as follows: Among them, P is a matrix composed of the direction vector of the ray, O is a matrix composed of the starting point of the ray, S is a matrix composed of the three-dimensional points in the world coordinate system, and Z is a diagonal parameter matrix with the elements on the diagonal For a line in parameterized space, c is a vector representing the position of the ray, is the rotation matrix, is the translation vector, α is the scaling factor; M is the three-dimensional center point of the threaded hole; is the rotation matrix The transpose of n T is the transpose of the vector in each direction; o n T is the transpose of each optical center vector; c T is the transpose of the ray position vector; M n T is the transpose of the three-dimensional center point vector of the threaded hole; 1c T is a matrix consisting of ray position vectors.

9. The method for spatial positioning of threaded holes and alignment of multiple holes based on monocular vision according to claim 8, characterized in that: The iterative method is used to solve the point-line registration problem and calculate the pose relationship between the threaded hole coordinate system of the lower cover and the camera coordinate system. Contains the rotation matrix and translation vectors 10. The method for threaded hole spatial positioning and multi-hole alignment based on monocular vision according to claim 1, characterized in that: In S5, the specific contents of aligning the through holes of the upper and lower covers with the threaded holes are as follows: The position relationship between the robot end coordinate system and the base coordinate system is obtained by calibrating the robot TCP Based on the hand-eye calibration results obtained above And the pose relationship from the cover to the virtual multi-eye system Obtain the position relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base Contains the rotation matrix and translation vectors The specific formula is: in, and It is the transformation relationship from the threaded hole coordinate system of the lower cover to the camera coordinate system; and It is the transformation relationship between the robot end coordinate system and the base coordinate system; The pose relationship between the upper cover through-hole coordinate system and the robot end coordinate system, including the rotation matrix and translation vectors Based on Obtain the position relationship between the upper cover through-hole coordinate system and the robot base coordinate system Contains the rotation matrix and translation vectors The specific formula is: in, and The transformation relationship between the robot end coordinate system and the base coordinate system; Knowing the above posture relationship, solve the new Contains the rotation matrix and translation vectors The end position of the robot arm is adjusted accordingly to achieve multi-hole alignment of the upper and lower covers. The specific formula is: in, and It is the conversion relationship between the threaded hole coordinate system of the lower cover and the coordinate system of the robot base. and It is the conversion relationship between the robot end coordinate system and the upper cover through-hole coordinate system.

Images