Hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors

Abstract

The invention relates to a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors. The hole-hole alignment assembly visual guidance method comprises the following steps: calibrating the embedded visual sensors; after the robot grabs the workpiece, the embedded visual sensor is manually placed in the component, and the visual target is attached to the assembly hole of the frame; hand-eye calibration is carried out on the embedded visual sensor installed in the component; calculating assembly feature points on the component according to the calibration result of the embedded camera and the hand-eye calibration result; using an embedded visual sensor on the component to photograph the frame and the targets, calculating the circle center coordinate of each target through a target positioning algorithm, and taking the circle center coordinate as an assembly feature point of the frame; fitting is carried out according to the assembly feature points of the component and the frame, and the assembly pose is obtained. According to the image of the embedded visual sensor and the target image, the guiding of the assembly action can be automatically realized.

Description

Technical Field

[0001] The invention relates to a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, belonging to the field of machine vision assembly guidance. Background Art

[0002] Robots have the advantages of large operating space, high flexibility, and strong collaboration ability, and have been widely used in the field of assembly. In the process of robot assembly, guidance technology based on machine vision is a key technology, and the guidance accuracy and effect have a decisive influence on the final assembly effect.

[0003] Many large components are assembled by machining threaded holes on the frame, machining blank holes on the component, and then using bolts to pass through the blank holes and lock them into the threaded holes to achieve component assembly. When using a robot for this kind of assembly task, it is necessary to use the robot to accurately move the component to the corresponding position of the frame to ensure that the holes on the component are completely aligned with the holes on the frame.

[0004] The current visual guidance technology is to use a camera to shoot the components and frames separately before performing the assembly action, obtain the pose correspondence between the components and the frame through feature point extraction or stereo vision and other technologies, and then move the components to the corresponding positions on the frame according to the calculated correspondence. This method is effective for some small-sized and light-weight assembly tasks because its field of view is small and the positioning deviation is not large. However, in the assembly process of large components, this method is difficult to achieve good accuracy. Because it is essentially an open-loop guidance method, the camera shooting process and the assembly action execution process are independent of each other, and a slight deviation between the two will lead to the failure of the final assembly.

[0005] In order to solve the above problems, in some component assembly tasks, designers will use force feedback technology to guide the end of the assembly action, that is, to use the force between the component and the frame to determine whether the assembly is successful. This technology is commonly used in shaft-hole assembly or gear assembly. However, this technology has its limitations: when the two cannot generate multi-dimensional contact forces, force feedback technology cannot determine the state of the assembly. The assembly task of hole-hole matching can only generate one-dimensional contact force, and its direction is orthogonal to the contact surface, which cannot reflect the alignment of the holes. Some researchers also use visual sensors to detect hole-hole alignment. They place the visual sensor on the outside of the component, take pictures of the assembly hole, and detect whether the hole is aligned by image recognition. In this method, the assembly hole severely limits the field of view of the visual sensor, and it can only see a small area through the assembly hole, resulting in this method can only judge whether the hole-hole is aligned, but cannot guide the hole-hole assembly process. Summary of the invention

[0006] In view of this, the present invention proposes a hole-hole alignment assembly vision guidance method based on multiple embedded vision sensors, which is used to meet the vision guidance requirements of large-scale component assembly tasks with hole-hole alignment requirements.

[0007] The present invention adopts the following technical solution: a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, comprising the following steps:

[0008] 1) Calibrate the embedded visual sensor to obtain the internal and external parameters of the embedded visual sensor;

[0009] 2) After the robot grabs the component to be assembled, it sets the embedded visual sensor in the assembly hole of the component and affixes the visual target to the periphery of the assembly hole of the frame used for assembling with the component;

[0010] 3) Perform hand-eye calibration on the embedded visual sensor after the component is installed to obtain the position and posture relationship between the camera coordinate system of all embedded visual sensors and the robot tool coordinate system;

[0011] 4) Obtaining the coordinates of the component assembly feature points in the corresponding camera coordinate system according to the calibration results of the embedded visual sensor, and transforming the coordinates of all feature points into the robot's tool coordinate system according to the pose relationship between the camera coordinate system and the robot's tool coordinate system obtained in step 3);

[0012] 5) When the robot moves the component to the position to be assembled, the built-in visual sensor is used to capture images of the visual targets on the frame, and the positioning algorithm is used to process the images to obtain the three-dimensional coordinates of the center points of the visual targets in each camera coordinate system; then, the position relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3) is used to transform the coordinates of the center points of all visual targets into the robot tool coordinate system;

[0013] 6) According to the coordinates of the center point of the visual target obtained in step 5) and the coordinates of the workpiece assembly feature points obtained in step 4), the final assembly action trajectory is obtained to achieve assembly visual guidance.

[0014] The embedded visual sensor includes a shell, and a camera, a lens, and a light source arranged in the shell. The shell is a cylinder, and the camera, the lens, and the light source are located on the end face of the cylinder. The parameters of the camera and the lens meet the requirements of the actual scene, so that when the camera is used to capture the image of the frame and the target before the assembly action, the camera's field of view can cover all areas where the target may appear, and there is only one target in the field of view.

[0015] The step 1) calibrates the embedded camera to obtain the internal and external parameters of the embedded camera, specifically: the internal parameters include the focal length, pixel size, and distortion coefficient of the camera, and the external parameters include the posture relationship between the camera coordinate system and the shell of the embedded visual sensor.

[0016] The visual target is in the shape of a ring; the color of the ring is black, and the surrounding area is a white static area. The center of the ring is a circular hole, and the diameter of the circular hole is equal to or greater than the diameter of the assembly hole. The target is pasted on the surface of the frame and is arranged outside the assembly hole. The inner circle of the ring coincides with or is concentric with the assembly hole.

[0017] The hand-eye calibration is performed by using a circular dot matrix calibration plate: multiple sets of calibration plate images are captured by a mobile robot, and the tool coordinates of the robot's end tool are recorded during each capture; multiple sets of circular dot matrix calibration plate images and corresponding coordinates are substituted into the hand-eye model, and the position relationship between the camera coordinate system and the tool coordinate system is obtained by a nonlinear optimization algorithm; the hand-eye model is as follows:

[0018] camera_H_cal=camera_H_tool*base_H_tool -1 *base_H_cal

[0019] Among them, camera_H_cal is the transfer matrix from the camera coordinate system to the calibration plate coordinate system, camera_H_tool is the transfer matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 It is the inverse matrix of the transfer matrix from the robot base coordinate system to the tool coordinate system, and base_H_cal is the transfer matrix from the base coordinate system to the calibration plate coordinate system.

[0020] The center of the hole on the surface of the component to be assembled is the assembly feature point P i , and the coordinates of the assembly feature points in the camera coordinate system can be obtained through the external parameter part of the calibration parameters of the embedded camera; i The center of the visual target on the corresponding frame is defined as the assembly feature point Q i .

[0021] The characteristic point P on the component surface i The characteristic point Q on the frame surface i to match, and will be able to make P i With Q i The component pose represented by the matching result whose distance is less than a predetermined value is taken as the final assembly pose.

[0022] The center point of the visual target is obtained by the following steps:

[0023] (1) Perform mean filtering on the original grayscale image I to obtain image Imean;

[0024] (2) Subtract image I from Imean and binarize it according to the threshold to obtain the candidate region R;

[0025] (3) Perform morphological screening on the connected regions in the candidate region R based on roundness and area;

[0026] (4) For the area R with the highest roundness among the results obtained in step (3), m Extract the contour area, and use the contour area as a mask to perform sub-pixel contour extraction and fitting in the original grayscale image;

[0027] (5) Use the monocular positioning circle algorithm to realize the center coordinate positioning of the circle.

[0028] In step 6), the fitting is achieved by the following formula:

[0029]

[0030] Among them, Px[i], Py[i], Pz[i] are the coordinates of the i-th assembly feature point on the component, Qx[i], Qy[i], Qz[i] are the coordinates of the i-th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, and minimum represents the minimum value.

[0031] The beneficial effects and advantages produced by the present invention are as follows:

[0032] The present invention adopts an embedded visual sensor, which can be arranged in the assembly hole of the component to be assembled, and directly captures the image of the combination position of the component and the frame.

[0033] The present invention proposes a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, which can automatically guide the assembly action according to the image of the embedded visual sensor and the target image. Compared with the current common visual guidance technology of photographing the component and the frame with a camera before performing the assembly action, the method proposed by the present invention is a closed-loop control method. Even if there is a deviation between the photographing process and the guidance process, it can also be adjusted in real time to achieve smooth assembly, and the alignment of the assembly holes can be observed to ensure the assembly quality. Compared with the method of setting the visual sensor on the outside of the component, the method proposed by the present invention can observe a larger field of view, and combined with the target positioning algorithm, it can achieve assembly guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the relationship between the component and the frame installation position;

[0035] Figure 2 It is a schematic diagram of the structure of the embedded visual sensor;

[0036] Figure 3 It is a schematic diagram of the relationship between the embedded visual sensor and the component installation position;

[0037] Figure 4 is a schematic diagram of a visual target used in the present invention;

[0038] Figure 5 a. Figure 5 b. Figure 5 c. Figure 5 d is the calibration images at different angles collected by the embedded visual sensor of the present invention;

[0039] Figure 6 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the embodiments.

[0041] The embodiment of the present invention discloses a hole-hole alignment assembly visual guidance method based on multiple embedded visual sensors, which is used to complete the assembly task of components and frames. Task description: Figure 1 There are multiple light holes on the component to be assembled, and there are threaded holes at the corresponding positions on the frame. The assembly requirement is to use a robot to clamp and move the component, fit the component tightly to the corresponding position on the frame, and then manually insert and lock the studs. In this assembly process, because the position of the component and the frame is not fixed, the assembly cannot be completed by teaching. The visual guidance method is required to control the movement of the end mechanism to achieve the fit between the component and the frame, and ensure that the holes on the component are completely aligned with the holes on the frame so that the bolts can pass through smoothly.

[0042] The following rules are used in this method to define component assembly targets:

[0043] (1) The side of the component close to the frame is defined as the inner side of the component, and the centers of the assembly holes on the inner surface of the component are defined as assembly feature points Pi in turn;

[0044] (2) The side of the frame close to the component is defined as the outer side of the frame, and the centers of the assembly holes on the outer surface of the frame are defined as the assembly feature point set Qi in sequence;

[0045] (3) The qualified assembly standard is: there is a one-to-one correspondence between Pi and Qi, the distances between all corresponding points meet the assembly requirements, and the sum of the total distances is the smallest.

[0046] like Figure 6 As shown, the method comprises the following steps:

[0047] (1) Calibrate the embedded visual sensor. The embedded visual sensor consists of a camera, a lens, a light source, and a shell. The overall shape is cylindrical, and the diameter is slightly smaller than the diameter of the assembly hole. The camera, lens, and light source are located on the end face of the cylinder. The camera and lens are assembled together and centered. Several light sources are evenly distributed around them. The light sources are lit when the camera captures an image. Its structure is as follows: Figure 2 shown.

[0048] The purpose of calibrating the embedded visual sensor is to obtain the camera's internal and external parameters. The internal parameters include the camera's focal length, pixel size, and distortion coefficient, while the external parameters are mainly to obtain the position relationship between the camera coordinate system and the shell, especially the distance parameter D between the camera's optical center and the shell end face.

[0049] (2) After the robot grabs the component, lift the component and lock the robot. Manually install multiple embedded vision sensors into the assembly holes of the component and fix them. When installing the embedded vision sensor, place it in the assembly hole of the component with the end with the camera facing the frame, and make the end face of the embedded vision sensor camera as flush as possible with the surface of the component to increase the camera's field of view, but do not protrude from the component to avoid collision and damage during assembly. Its installation form is as follows: Figure 3 shown.

[0050] In this embodiment, the component assembly feature point Pi is inferred by the installation position of the embedded camera. The feature point is located on the z-axis of the coordinate system in the camera coordinate system, and its z coordinate value is the parameter D obtained by calibrating the external parameter, that is, the coordinates of the feature point corresponding to the assembly hole in the camera coordinate system of the embedded visual sensor installed in the assembly hole are (0, 0, D).

[0051] (3) Manually paste the visual target on the assembly hole of the frame. The target consists of three parts: a white static area, a black ring, and a circular hole. Its structure is as follows: Figure 4 The key is the outer edge of the black ring, whose size is accurate and known. The width of the ring is appropriate. The diameter of the hole is equal to or slightly larger than the diameter of the mounting hole on the frame.

[0052] When pasting the target, if the diameter of the target hole is the same as that of the assembly hole, the target hole and the assembly hole need to be closely overlapped; if the diameter of the target hole is larger than the diameter of the assembly hole, the target hole and the assembly hole need to be concentric;

[0053] In this method, the center of the circular ring of the pasted target is used as the assembly mark point Qi for assembly guidance.

[0054] (4) Perform hand-eye calibration on the embedded vision sensor installed on the component. Specifically, perform eye-on-hand calibration. Eye-on-hand means that the vision sensor is installed on the end of the robot and moves with the robot. Hand-eye calibration means obtaining the conversion relationship between the camera coordinate system and the robot tool coordinate system. The result of hand-eye calibration can be used to convert the point coordinates in the camera coordinate system to the robot tool coordinate system.

[0055] Specifically, the present invention uses a dot array calibration plate for hand-eye calibration. The world coordinates of the dots on the dot array calibration plate have been measured in advance using high-precision equipment. The specific calibration process is: place the dot array calibration plate at a certain position within the robot's working range, operate the robot to move, and enable the visual sensor to capture images of the calibration plate from different positions. The calibration image is as follows: Figure 5 a~ Figure 5 d. The world coordinates of the dot array calibration plate and the tool coordinates of the robot at each shooting are brought into the hand-eye calibration model, and the relationship between the camera coordinate system and the robot tool coordinate system is calculated through nonlinear optimization. The camera model of "eye on hand" is as follows:

[0056] camera_H_cal=camera_H_tool*base_H_tool -1 *base_H_cal

[0057] Among them, camera_H_cal is the transfer matrix from the camera coordinate system to the calibration plate coordinate system, camera_H_tool is the transfer matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 is the inverse matrix of the transfer matrix from the robot base coordinate system to the tool coordinate system, base_H_cal is the transfer matrix from the base coordinate system to the calibration plate coordinate system. Among them, camera_H_tool is the transfer matrix between the camera and tool coordinate systems that we are concerned about.

[0058] After obtaining the transfer matrix between the camera coordinate system of all embedded vision sensors and the robot tool coordinate system, the transfer matrix between all cameras is further obtained. The following formula is used to calculate the coordinate system relationship between multiple cameras:

[0059]

[0060] Among them, camera_H_camera ni represents the transfer matrix from camera n to camera i.

[0061] (5) After the robot moves the component to the vicinity of the assembly position, the built-in visual sensor takes a picture of the frame. When taking pictures, it is necessary to ensure that each visual sensor's field of view contains only one target. After that, the obtained image is processed according to the following steps to calculate the three-dimensional coordinates of the center point of the visual target.

[0062] 1) Perform mean filtering on the original grayscale image I to obtain image Imean;

[0063] 2) Subtract image I from Imean and binarize it with a threshold of 100 to obtain the candidate region R;

[0064] 3) Perform morphological screening on the connected regions in the candidate region R based on roundness and area;

[0065] 4) extracting the contour area of ​​the area Rm that best meets the conditions in the results obtained in step 3, and using the contour area as a mask to perform sub-pixel contour extraction and fitting in the original grayscale image;

[0066] 5) Use the monocular positioning circle algorithm to realize the center coordinate positioning. The result of the monocular positioning circle algorithm is ambiguous because there are usually two planes with a radius of R formed after cutting the elliptical cone in space. In this algorithm, as the distance between the workpiece to be assembled and the frame continues to approach, the optical axis of the camera and the plane where the circular hole on the frame is located continue to approach orthogonality, so the two center coordinates obtained are often very close, and the average of their coordinates can meet the assembly accuracy requirements, without the need to use other complex methods to eliminate ambiguity;

[0067] (6) According to the inter-camera transfer matrix obtained in step 4, the coordinates of the assembly feature point S1 obtained in step 2 are transferred to the robot tool coordinate system, and the target center coordinates obtained in step 5, i.e., the assembly feature point set S2, are transferred to the robot tool coordinate system.

[0068] (7) In the robot tool coordinate system, S1 and S2 are fitted to obtain the pose with the smallest overall distance as the final assembly pose. The solution process is carried out as follows:

[0069]

[0070] Among them, Px[i], Py[i], Pz[i] are the coordinates of the i-th assembly feature point on the component, Qx[i], Qy[i], Qz[i] are the coordinates of the i-th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, and minimum represents the minimum value.

Claims

1. Vision-guided method for hole-hole alignment assembly based on multiple embedded vision sensors, It is characterized in that The following steps are involved: 1) Calibrate the embedded visual sensor to obtain the internal and external parameters of the embedded visual sensor; 2) After the robot grabs the component to be assembled, it sets the embedded visual sensor in the assembly hole of the component and affixes the visual target to the periphery of the assembly hole of the frame used for assembling with the component; 3) Perform hand-eye calibration on the embedded visual sensor after the component is installed to obtain the position and posture relationship between the camera coordinate system of all embedded visual sensors and the robot tool coordinate system; 4) Obtaining the coordinates of the component assembly feature points in the corresponding camera coordinate system according to the calibration results of the embedded visual sensor, and transforming the coordinates of all feature points into the robot's tool coordinate system according to the pose relationship between the camera coordinate system and the robot's tool coordinate system obtained in step 3); 5) When the robot moves the component to the position to be assembled, the built-in visual sensor is used to capture images of the visual targets on the frame, and the positioning algorithm is used to process the images to obtain the three-dimensional coordinates of the center points of the visual targets in each camera coordinate system; then, the position relationship between the camera coordinate system and the robot tool coordinate system obtained in step 3) is used to transform the coordinates of the center points of all visual targets into the robot tool coordinate system; 6) According to the coordinates of the center point of the visual target obtained in step 5) and the coordinates of the workpiece assembly feature points obtained in step 4), the final assembly action trajectory is obtained to achieve assembly visual guidance.

2. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The embedded visual sensor includes a shell, and a camera, a lens, and a light source arranged in the shell. The shell is a cylinder, and the camera, the lens, and the light source are located on the end face of the cylinder. The parameters of the camera and the lens meet the requirements of the actual scene, so that when the camera is used to capture the image of the frame and the target before the assembly action, the camera's field of view can cover all areas where the target may appear, and there is only one target in the field of view.

3. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The step 1) calibrates the embedded camera to obtain the internal and external parameters of the embedded camera, specifically: the internal parameters include the focal length, pixel size, and distortion coefficient of the camera, and the external parameters include the posture relationship between the camera coordinate system and the shell of the embedded visual sensor.

4. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The visual target is in the shape of a ring; the color of the ring is black, and the surrounding area is a white static area. The center of the ring is a circular hole, and the diameter of the circular hole is equal to or greater than the diameter of the assembly hole. The target is pasted on the surface of the frame and is arranged outside the assembly hole. The inner circle of the ring coincides with or is concentric with the assembly hole.

5. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The hand-eye calibration is performed by using a circular dot matrix calibration plate: multiple sets of calibration plate images are captured by a mobile robot, and the tool coordinates of the robot's end tool are recorded during each capture; multiple sets of circular dot matrix calibration plate images and corresponding coordinates are substituted into the hand-eye model, and the position relationship between the camera coordinate system and the tool coordinate system is obtained by a nonlinear optimization algorithm; the hand-eye model is as follows: camera_H_cal=camera_H_tool*base_H_tool -1 *base_H_cal Among them, camera_H_cal is the transfer matrix from the camera coordinate system to the calibration plate coordinate system, camera_H_tool is the transfer matrix from the camera coordinate system to the tool coordinate system, base_H_tool -1 It is the inverse matrix of the transfer matrix from the robot base coordinate system to the tool coordinate system, and base_H_cal is the transfer matrix from the base coordinate system to the calibration plate coordinate system.

6. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The center of the hole on the surface of the component to be assembled is the assembly feature point P i , and the coordinates of the assembly feature points in the camera coordinate system can be obtained through the external parameter part of the calibration parameters of the embedded camera; i The center of the visual target on the corresponding frame is defined as the assembly feature point Q i .

7. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The characteristic point P on the component surface i The characteristic point Q on the frame surface i to match, and will be able to make P i With Q i The component pose represented by the matching result whose distance is less than a predetermined value is taken as the final assembly pose.

8. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that The center point of the visual target is obtained by the following steps: (1) Perform mean filtering on the original grayscale image I to obtain image Imean; (2) Subtract image I from Imean and binarize it according to the threshold to obtain the candidate region R; (3) Perform morphological screening on the connected regions in the candidate region R based on roundness and area; (4) For the area R with the highest roundness among the results obtained in step (3), m Extract the contour area, and use the contour area as a mask to perform sub-pixel contour extraction and fitting in the original grayscale image; (5) Use the monocular positioning circle algorithm to realize the center coordinate positioning of the circle.

9. The hole-to-hole alignment assembly vision guidance method based on multiple embedded vision sensors according to claim 1, It is characterized in that In step 6), the fitting is achieved by the following formula: Among them, Px[i], Py[i], Pz[i] are the coordinates of the i-th assembly feature point on the component, Qx[i], Qy[i], Qz[i] are the coordinates of the i-th assembly feature point on the frame; HomMat represents the transfer matrix corresponding to the assembly action in the current tool coordinate system, and minimum represents the minimum value.

Images