Hole-hole alignment assembly visual guidance method based on multi-embedded visual sensor

Abstract

The application relates to a hole-hole alignment assembly visual guidance method based on a multi-embedded visual sensor, and comprises the following steps: calibrating the embedded visual sensor; after a robot grasps a workpiece, manually placing the embedded visual sensor into a component, and pasting visual targets on assembly holes of a frame; calibrating the embedded visual sensor in the component; calculating assembly feature points on the component according to a calibration result of the embedded camera and a hand-eye calibration result; photographing the frame and the targets by using the embedded visual sensor on the component, and calculating the center coordinates of each target by a target positioning algorithm, so that the center coordinates are used as the assembly feature points of the frame; and fitting the assembly feature points of the component and the frame to obtain the assembly pose. The application can automatically realize the guidance of the assembly action according to the image of the embedded visual sensor and the target image.

Description

TECHNICAL FIELD

[0001] The present application relates to a hole-hole alignment assembly visual guidance method based on multi-embedded visual sensor, and belongs to the field of machine vision assembly guidance. BACKGROUND

[0002] Robots have the advantages of large operation space, high flexibility and strong collaboration ability, and have been widely used in the field of assembly. In the process of robot assembly, the guidance technology based on machine vision is a key technology, and the guidance precision and effect have a decisive influence on the final assembly effect.

[0003] Many large component assemblies are processed on the frame to form threaded holes, and light holes are processed on the components. The assembly of the components is achieved by using bolts to pass through the light holes and lock them into the threaded holes. When using a robot to perform this assembly task, the robot needs to accurately move the component to the corresponding position on the frame to ensure that the holes on the component are completely aligned with the holes on the frame.

[0004] Current visual guidance technology separately captures the component and the frame with a camera before performing the assembly action, and obtains the pose correspondence between the component and the frame through feature point extraction or stereo vision technology. Then, the component is moved to the corresponding position on the frame according to the calculated correspondence. This method is effective when performing some small-size and small-weight assembly tasks, because its field of view is small and the positioning deviation is not large. However, in the assembly process of large components, this method is difficult to achieve good precision. Because it is essentially an open-loop guidance method, the camera capturing process and the execution process of the assembly action are independent of each other, and a small deviation between the two will result in the failure of the final assembly.

[0005] To solve the above problems, in some component assembly tasks, designers use force feedback technology to guide the last segment of the assembly action, that is, to judge whether the assembly is successful by the contact force between the component and the frame. This technology is commonly used in shaft-hole assembly or gear assembly. However, this technology has its limitations: when the two cannot produce multi-dimensional contact force, the force feedback technology cannot determine the assembly state. The hole-hole assembly task can only produce one-dimensional contact force, which is orthogonal to the contact surface, and cannot reflect the alignment of the holes. Some researchers use visual sensors to detect hole-hole alignment, and place the visual sensor outside the component to take a photo of the assembly hole and detect whether the hole is aligned through image recognition. In this method, the assembly hole severely limits the field of view of the visual sensor, which can only see a small area through the assembly hole, resulting in that this method can only determine whether the holes are aligned, but cannot guide the hole-hole assembly process. SUMMARY

[0006] Therefore, the application provides a hole-hole alignment assembly visual guidance method based on a multi-embedded visual sensor to meet the visual guidance requirement of a large component assembly task with hole-hole alignment requirements.

[0007] The application adopts the following technical scheme: a hole-hole alignment assembly visual guidance method based on a multi-embedded visual sensor, comprising the following steps:

[0008] 1) calibrate the embedded visual sensor to obtain the intrinsic and extrinsic parameters of the embedded visual sensor;

[0009] 2) after the robot grasps the component to be assembled, set the embedded visual sensor in the assembly hole of the component, and paste the visual target on the periphery of the assembly hole of the frame used for assembly with the component;

[0010] 3) calibrate the embedded visual sensor after being loaded into the component to obtain the pose relationship between the camera coordinate system of all embedded visual sensors and the robot tool coordinate system;

[0011] 4) obtain the coordinates of the component assembly feature points in the corresponding camera coordinate system according to the calibration results of the embedded visual sensor, and convert the coordinates of all feature points into the tool coordinate system of the robot according to the pose relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3);

[0012] 5) when the robot moves the component to the position to be assembled, use the embedded visual sensor to capture the image of the visual target on the frame, and use the positioning algorithm to process the image to obtain the three-dimensional coordinates of the center point of the visual target in each camera coordinate system; then convert the coordinates of all visual target center points into the robot tool coordinate system using the pose relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3);

[0013] 6) fit the coordinates of the visual target center points obtained in step 5) and the coordinates of the workpiece assembly feature points obtained in step 4) to obtain the final assembly action trajectory, and realize assembly visual guidance.

[0014] The embedded visual sensor comprises a shell, and a camera, a lens and a light source arranged in the shell, the shell is a cylinder, and the camera, the lens and the light source are located at the end face of the cylinder; the parameters of the camera and the lens meet the requirements of the actual scene, so that when the camera captures the image of the frame and the target before the assembly action, the field of view of the camera can cover all areas where the target may appear, and there is only one target in the field of view.

[0015] The step 1) calibrates the embedded camera to obtain intrinsic and extrinsic parameters of the embedded camera, specifically, the intrinsic parameters include focal length, pixel size and distortion coefficient of the camera, and the extrinsic parameters include the pose relationship between the camera coordinate system and the shell of the embedded visual sensor.

[0016] The visual target is a circular ring, the color of the circular ring is black, the periphery is a white static area, the center of the circular ring is a circular hole, the diameter of the circular hole is equal to or greater than the diameter of the assembly hole, the target is pasted on the surface of the frame and is arranged at the periphery of the assembly hole, the inner circle of the circular ring is coincident with or concentric with the assembly hole.

[0017] The hand-eye calibration is performed by selecting a circular dot array calibration board for calibration: a plurality of groups of calibration board images are captured by moving the robot, and the end tool coordinates of the robot at each time of capturing are recorded; the plurality of groups of circular dot array calibration board images and the corresponding coordinates are substituted into a hand-eye model to obtain the pose relationship between the camera coordinate system and the tool coordinate system through a nonlinear optimization algorithm; the hand-eye model is as follows:

[0018] camera_H_cal = camera_H_tool * base_H_tool -1 * base_H_cal

[0019] Wherein, camera_H_cal is a transfer matrix from the camera coordinate system to the calibration board coordinate system, camera_H_tool is a transfer matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 is an inverse matrix of a transfer matrix from the robot base coordinate system to the tool coordinate system, and base_H_cal is a transfer matrix from the base coordinate system to the calibration board coordinate system.

[0020] The center of the hole on the surface of the component to be assembled is the assembly feature point P i , and the coordinates of the assembly feature point in the camera coordinate system can be obtained through the extrinsic part of the calibration parameters of the embedded camera; the center of the visual target on the frame corresponding to P i is defined as the assembly feature point Q i .

[0021] The feature point P i on the surface of the component is matched with the feature point Q i on the surface of the frame, and the component pose represented by the matching result that can make the distance between P i and Q i less than a predetermined value is taken as the final assembly pose.

[0022] The center point of the visual target is obtained by the following steps:

[0023] (1) Mean filtering is performed on the original gray image I to obtain an image Imean;

[0024] (2) image I is subtracted from Imean, and binarization is performed according to a threshold value to obtain a candidate region R;

[0025] (3) a morphological screening is performed on a connected region in the candidate region R according to the features of roundness and area;

[0026] (4) the region R with the highest roundness in the result obtained in step (3) is selected as a final result; m A contour region is extracted, and sub-pixel contour extraction and fitting are performed in the original gray-scale image by using the contour region as a mask;

[0027] (5) a monocular positioning circle algorithm is used to realize positioning of the center coordinates of the circle.

[0028] In step 6), the fitting is realized by the following formula:

[0029]

[0030] Wherein, Px[i], Py[i], Pz[i] are the coordinates of the i th assembly feature point on the component, Qx[i], Qy[i], Qz[i] are the coordinates of the i th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, and minimum represents the minimum value.

[0031] The beneficial effects and advantages of the present application are as follows:

[0032] The present application adopts an embedded visual sensor, which can be arranged in the assembly hole of the component to be assembled to directly capture the image of the combined position of the component and the frame.

[0033] The present application provides a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, which can automatically realize the guidance of the assembly action according to the image of the embedded visual sensor and the target image. Compared with the current common visual guidance technology of respectively capturing the component and the frame by using a camera before performing the assembly action, the method provided by the present application is a closed-loop control method, which can also be adjusted in real time in the case of deviation between the photographing process and the guidance process, realize smooth assembly, and observe the alignment of the assembly hole to ensure the assembly quality. Compared with the method of arranging the visual sensor outside the component, the method provided by the present application can observe a larger field of view, and can realize the guidance of the assembly by combining the target positioning algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a schematic view of the installation position relationship between the component and the frame;

[0035] Figure 2 is a structural schematic view of the embedded visual sensor;

[0036] Figure 3 is a schematic diagram of the position relationship between the embedded visual sensor and the component;

[0037] Figure 4 is a schematic diagram of the visual target used by the present application;

[0038] Figure 5a 、 Figure 5b 、 Figure 5c 、 Figure 5d is a calibration image of different angles collected by the embedded visual sensor of the present application;

[0039] Figure 6 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in conjunction with examples.

[0041] The embodiment of the present application discloses a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, which is used to complete the assembly task of components and frames. The task description is as follows Figure 1 , the component to be assembled has multiple light holes, and the corresponding positions on the frame have threaded holes. The assembly requirement is to use a robot to hold and move the component, and then manually insert and lock the stud after tightly fitting the component to the corresponding position on the frame. In this assembly process, because the positions of the component and the frame are not fixed, the assembly cannot be completed by teaching, and the visual guidance method is needed to control the movement of the end mechanism to realize the fitting of the component and the frame, and to ensure that the holes on the component and the frame are completely aligned, and the bolt can be smoothly inserted.

[0042] The following rules are used to define the component assembly target in the method:

[0043] (1) the side of the component close to the frame is defined as the inner side of the component, and the centers of each assembly hole on the inner side surface of the component are sequentially defined as assembly feature points Pi;

[0044] (2) the side of the frame close to the component is defined as the outer side of the frame, and the centers of each assembly hole on the outer side surface of the frame are sequentially defined as assembly feature point set Qi;

[0045] (3) the assembly qualified standard is that Pi and Qi form a one-to-one correspondence relationship, the distance between all corresponding points meets the assembly requirement, and the total distance sum is minimum.

[0046] As shown in Figure 6 , the method includes the following steps:

[0047] (1) Calibrate the embedded visual sensor. The embedded visual sensor is composed of a camera, a lens, a light source, and a shell. The whole is cylindrical, and the diameter is slightly smaller than the diameter of the mounting hole. The camera, lens, and light source are located on the end face of the cylinder. The camera and lens are assembled together and located in the center. The light source is evenly distributed around the camera. The light source is turned on when the camera takes pictures. The structure is shown in Figure 2 .

[0048] The purpose of calibrating the embedded visual sensor is to obtain the intrinsic and extrinsic parameters of the camera. The intrinsic parameters include the focal length, pixel size, and distortion coefficient of the camera. The extrinsic parameters mainly obtain the pose relationship between the camera coordinate system and the shell, especially the distance parameter D between the camera optical center and the end face of the shell.

[0049] (2) After the robot grasps the component, lift the component and lock the robot. Manually install multiple embedded visual sensors into the mounting hole of the component and fix them. When installing the embedded visual sensor, place it in the mounting hole of the component, with the camera end facing the frame, and make the end face of the camera end of the embedded visual sensor as flat as possible with the surface of the component to increase the field of view of the camera. However, it cannot protrude from the component to avoid collision and damage during assembly. The installation form is shown in Figure 3 .

[0050] In this embodiment, the component assembly feature point Pi is obtained by reasoning the installation position of the embedded camera. The feature point is located on the z-axis of the camera coordinate system, and the z-coordinate value is the parameter D obtained from the calibration extrinsic parameters. That is, the coordinate of the feature point corresponding to the mounting hole in the camera coordinate system of the embedded visual sensor installed in the mounting hole is (0, 0, D).

[0051] (3) Manually paste the visual target on the mounting hole of the frame. The target is composed of three parts: a white static area, a black ring, and a circular hole. The structure is shown in Figure 4 . The key part is the outer edge of the black ring, which is accurate and known. The width of the ring can be appropriate. The diameter of the circular hole is equal to or slightly larger than the diameter of the mounting hole on the frame.

[0052] When pasting the target, if the diameter of the circular hole of the target is consistent with the diameter of the mounting hole, the circular hole of the target needs to be tightly coincided with the mounting hole. If the diameter of the circular hole of the target is larger than the diameter of the mounting hole, the circular hole of the target needs to be concentric with the mounting hole.

[0053] In this method, the center of the circular ring of the pasted target is taken as the assembly landmark point Qi, which is used for assembly guidance.

[0054] (4) Hand-eye calibration for the embedded visual sensor mounted on the component, specifically, eye-in-hand hand-eye calibration. Eye-in-hand means that the visual sensor is mounted on the end of the robot and moves with the robot, and hand-eye calibration means obtaining the conversion relationship between the camera coordinate system and the robot tool coordinate system. Through the results of hand-eye calibration, the point coordinates in the camera coordinate system can be converted to the robot tool coordinate system.

[0055] Specifically, the present application selects a circular dot array calibration board for hand-eye calibration. The world coordinates of the circular dots on the circular dot array calibration board have been measured in advance using high-precision equipment. The specific calibration process is as follows: place the circular dot array calibration board at a certain position within the robot working range, operate the robot to move and make the visual sensor capture images of the calibration board from different poses. The calibration image is shown in Figures 5a to 5d The world coordinates of the circular dot array calibration board and the tool coordinates of the robot at the time of each shot are brought into the hand-eye calibration model, and the relationship between the camera coordinate system and the robot tool coordinate system is calculated by nonlinear optimization. The camera model of "eye-in-hand" is as follows:

[0056] camera_H_cal=camera_H_tool*base_H_tool -1 *base_H_cal

[0057] Wherein, camera_H_cal is the transfer matrix from the camera coordinate system to the calibration board coordinate system, camera_H_tool is the transfer matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 is the inverse matrix of the transfer matrix from the robot base coordinate system to the tool coordinate system, and base_H_cal is the transfer matrix from the base coordinate system to the calibration board coordinate system. The camera_H_tool therein is the transfer matrix between the camera and the tool coordinate system that we are interested in.

[0058] After obtaining the transfer matrix between the camera coordinate system and the robot tool coordinate system of all embedded visual sensors, the transfer matrix between all cameras is further obtained. The following formula is used when calculating the coordinate system relationship between multiple cameras:

[0059]

[0060] Wherein, camera_H_camera ni represents the transfer matrix from camera n to camera i.

[0061] (5) After the robot moves the component to the vicinity of the assembly position, the embedded visual sensor takes a picture of the frame. At this time, it is necessary to ensure that each visual sensor contains only one target in its field of view. Then, the image is processed according to the following steps to calculate the three-dimensional coordinates of the center point of the visual target.

[0062] 1) Mean filtering is performed on the original gray image I to obtain image Imean;

[0063] 2) The image I is subtracted from Imean, and binarization is performed with a threshold of 100 to obtain the candidate region R;

[0064] 3) The connected regions in the candidate region R are morphologically filtered based on the circularity and area characteristics;

[0065] 4) The contour region of the most suitable region Rm obtained in step 3 is extracted, and the sub-pixel contour extraction and fitting are performed in the original gray image using the contour region as a mask;

[0066] 5) The monocular positioning circle algorithm is used to realize the positioning of the center coordinates. The monocular positioning circle algorithm has ambiguity because the plane formed by cutting the elliptical cone in space to form a circle with a radius R usually has two. In this algorithm, because the optical axis of the camera and the plane of the circular hole on the frame are constantly approaching orthogonal as the distance between the workpiece to be assembled and the frame approaches, the two center coordinates obtained are usually very close, and taking the average of the coordinates can meet the assembly accuracy requirements without using other complex methods to eliminate ambiguity;

[0067] (6) According to the transfer matrix between the cameras obtained in step 4, the coordinates of the assembly feature points S1 obtained in step 2 are transferred to the robot tool coordinate system, and the center coordinates of the target obtained in step 5, i.e., the assembly feature point set S2, are transferred to the robot tool coordinate system.

[0068] (7) In the robot tool coordinate system, S1 and S2 are fitted to obtain the pose with the minimum overall distance as the final assembly pose. The solving process is as follows:

[0069]

[0070] where Px[i], Py[i], Pz[i] are the coordinates of the i-th assembly feature point on the component, Qx[i], Qy[i], Qz[i] are the coordinates of the i-th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, and minimum represents the minimum value.

Claims

1. A hole-hole alignment assembly vision guidance method based on multi-in-line vision sensor, characterized in that, The method comprises the following steps: 1) calibrating the embedded visual sensor to obtain the intrinsic and extrinsic parameters of the embedded visual sensor; 2) after the robot grasps the component to be assembled, the embedded visual sensor is arranged in the assembly hole of the component, and a visual target is attached to the periphery of the assembly hole of the frame for assembly with the component; 3) hand-eye calibration is performed on the embedded visual sensor after the component is assembled, and the pose relationship between the camera coordinate system of all embedded visual sensors and the robot tool coordinate system is obtained; 4) the coordinates of the assembly feature points of the component in the corresponding camera coordinate system are obtained according to the calibration results of the embedded visual sensor, and the coordinates of all feature points are converted into the tool coordinate system of the robot according to the pose relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3); 5) when the robot moves the component to the assembly position, the embedded visual sensor is used to capture images of the visual target on the frame, and a positioning algorithm is used to process the images to obtain the three-dimensional coordinates of the center points of the visual targets in the camera coordinate system; then the coordinates of all the center points of the visual targets are converted into the robot tool coordinate system using the pose relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3); 6) fitting is performed on the coordinates of the center points of the visual targets obtained in step 5) and the coordinates of the workpiece assembly feature points obtained in step 4) to obtain the final assembly motion trajectory, thereby realizing assembly visual guidance.

2. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method according to claim 1, wherein, The embedded visual sensor comprises a shell, and a camera, a lens and a light source arranged in the shell, the shell is a cylinder, and the camera, the lens and the light source are located on the end face of the cylinder; the parameters of the camera and the lens meet the requirements of the actual scene, so that when the camera is used to capture images of the frame and the target before the assembly motion, the field of view of the camera can cover all possible areas of the target, and there is only one target in the field of view.

3. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method according to claim 2, wherein, The embedded camera is calibrated to obtain the intrinsic and extrinsic parameters of the embedded camera, specifically: the intrinsic parameters include the focal length, the pixel size and the distortion coefficient of the camera, and the extrinsic parameters include the pose relationship between the camera coordinate system and the shell of the embedded visual sensor.

4. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method of claim 1, wherein, The visual target is a circular ring; the color of the circular ring is black, the periphery is a white static area, the center of the circular ring is a circular hole, the diameter of the circular hole is equal to or greater than the diameter of the assembly hole, the target is attached to the surface of the frame and arranged at the periphery of the assembly hole, and the inner circle of the circular ring is coincident with or concentric with the assembly hole.

5. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method according to claim 1, wherein, The hand-eye calibration is performed by selecting a circular dot array calibration board: a plurality of calibration board images are captured by moving the robot, and the end tool coordinates of the robot at each time of capturing are recorded; the plurality of circular dot array calibration board images and the corresponding coordinates are substituted into a hand-eye model to obtain the pose relationship between the camera coordinate system and the tool coordinate system by a nonlinear optimization algorithm; the hand-eye model is as follows: ； where camera_H_cal is the transformation matrix from the camera coordinate system to the calibration board coordinate system, camera_H_tool is the transformation matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 is the inverse matrix of the transformation matrix from the robot base coordinate system to the tool coordinate system, and base_H_cal is the transformation matrix from the base coordinate system to the calibration board coordinate system.

6. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method of claim 1, wherein, The center of the hole of the surface of the component to be assembled is the assembly feature point P i And the coordinates of the assembly feature point in the camera coordinate system can be obtained through the extrinsic part of the calibration parameters of the embedded camera; the coordinates of the corresponding visual target on the frame are defined as the assembly feature point Q i The center of the hole of the surface of the component to be assembled is the assembly feature point P i .

7. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method according to claim 6, wherein, Feature point P on the surface of the component i Feature point Q on the frame surface i Perform a match, and will enable P i With Q i The component pose represented by the matching result where the distance is less than the predetermined value is taken as the final assembly pose.

8. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method of claim 1, wherein, The center point of the visual target is obtained by the following steps: (1) mean filtering is performed on the original gray image I to obtain an image Imean; (2) the image I is subtracted from Imean, and binarization is performed according to a threshold value to obtain a candidate region R; (3) the connected regions in the candidate region R are morphologically screened according to the circularity and area characteristics; (4) the region R with the highest circularity in the result obtained in step (3) m The contour region is extracted, and the sub-pixel contour extraction and fitting are performed in the original gray image by using the contour region as a mask. (5) a monocular positioning circle algorithm is used to realize the positioning of the center coordinates.

9. The multi-in-line vision sensor based hole-hole alignment assembly vision guidance method of claim 1, wherein, In step 6) the fitting is achieved by the following formula: ； wherein, is the coordinate of the i-th assembly feature point on the component, is the coordinate of the i-th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, denotes the minimum value.

Images