Demonstration-free robot grabbing and releasing point position teaching-free method and equipment based on vision

Abstract

The invention provides a robot grabbing and releasing point position teaching-free method and device based on vision, and the method comprises the steps: carrying out the vision internal correction based on a calibration plate and a camera visual field range, obtaining a vision correction pose C1 and a first pose deviation Offset, and building a mapping relation between a pixel coordinate system and a robot tool coordinate system; and based on the vision correction pose C1, the first pose deviation Offset and the mapping relation between the pixel coordinate system and the robot tool coordinate system, dragging a robot arm to carry out vision teaching to obtain a teaching pose of the plate position. According to the method, calculation of robot grabbing and releasing point positions can be automatically completed, the workload of operators is reduced, and the accuracy and reliability of point position teaching are improved.

Description

Technical Field

[0001] The invention relates to a vision-based robot grasping and releasing point teaching-free method and equipment. Background Art

[0002] Robots are widely used in high-precision tasks in automated production, such as material handling, assembly, and quality inspection. These operations have extremely high requirements for accuracy and stability, especially when handling tiny parts or complex work paths. Robots need to be taught accurately to ensure that they can perform tasks accurately.

[0003] Currently, most robot systems rely on manual teaching to complete the teaching process. To ensure that the robot can correctly identify objects in the working environment and complete tasks. The manual teaching process has the following significant disadvantages:

[0004] (1) Time-consuming and labor-intensive: The manual teaching process is cumbersome and time-consuming, especially when there are multiple workstations or the position of the calibration plate is adjusted frequently. Operators need to participate in the teaching process for a long time, which increases the workload and reduces production efficiency.

[0005] (2) Unstable accuracy: Since manual teaching relies on the operator’s experience and manual operation, it is easily affected by subjective factors such as fatigue, distraction, or differences in personal operating habits. This instability leads to low teaching accuracy, especially when dealing with precision tasks, and it is difficult to ensure that each teaching can achieve the required accuracy.

[0006] (3) High risk of error: During manual teaching, the slightest mistake may lead to deviations, resulting in errors in the robot path or grasping position. Such errors may be further amplified in subsequent tasks, especially in tasks requiring high precision, which may affect production quality and even cause equipment damage. Summary of the invention

[0007] The object of the present invention is to provide a vision-based robot grasping and releasing point teaching-free method and device.

[0008] In order to solve the above problems, the present invention provides a vision-based robot grasping and releasing point-free teaching method, comprising:

[0009] Placing a calibration plate at a plate position to be clamped by a gripper of a robot arm, wherein the calibration plate includes at least two circles; and the robot arm is equipped with a camera and a gripper;

[0010] Drag the robot arm until the camera can detect the shooting range of all circles on the calibration plate in the photos taken by the camera, which is used as the camera's field of view;

[0011] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain a visual calibration posture;

[0012] The robot arm is dragged for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, the posture deviation is obtained, and it is iterated based on the posture deviation to correct the current posture to obtain the teaching posture of the board position.

[0013] When the robot arm is a four-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0014] The calibration plate is placed at the plate position that needs to be clamped by the gripper of the robot arm, wherein a large circle and a small circle are arranged on the calibration plate at intervals;

[0015] Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including:

[0016] Drag the robot arm until the large and small circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view.

[0017] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration posture, including:

[0018] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established;

[0019] Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including:

[0020] Based on the visual calibration posture C1, the first posture deviation Offset and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position. Further, in the above method, the camera's intrinsic and extrinsic parameter matrices are calculated by the calibration algorithm, and the pixel coordinates of the centers of the large and small circles in the first image obtained, and the distance between the camera and the calibration plate are used to establish the mapping relationship between the pixel coordinate system and the robot tool coordinate system, including:

[0021] Perform camera intrinsic calibration: Calculate the camera's intrinsic parameter matrix K through the intrinsic calibration algorithm, including focal length (fx, fy), principal point coordinates (cx, cy) and distortion parameters;

[0022] Perform camera extrinsic calibration: Use the extrinsic calibration algorithm to calculate the extrinsic matrix of the camera relative to the robot tool coordinate system using the three-dimensional coordinates of the corresponding point and the projection of the point in the pixel coordinate system, including: rotation matrix R and translation vector T;

[0023] Based on the results of camera intrinsic calibration and camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established.

[0024] Furthermore, in the above method, based on the result of camera intrinsic calibration and the result of camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including:

[0025] According to formula (1), the pixel coordinates are converted to normalized camera coordinates to eliminate the influence of camera distortion.

[0026]

[0027] Among them, K is the camera internal parameter matrix, (u,v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate;

[0028] According to formula (2), and using the known depth D, the normalized camera coordinates are converted to the actual three-dimensional coordinates P in the camera coordinate system camera , where the depth D is the distance between the camera and the calibration plate;

[0029] According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates P of the point in the camera coordinate system are camera , converted into the three-dimensional coordinates P in the robot tool coordinate system robot , wherein the external parameter matrix includes a rotation matrix R and a translation vector T;

[0030] Map pixel coordinates (u,v) to three-dimensional coordinates (X,Y,Z) in the robot tool coordinate system.

[0031] Furthermore, in the above method, the actual three-dimensional coordinate P camera The calculation formula is as follows:

[0032]

[0033] The three-dimensional coordinate P in the robot tool coordinate system robot The calculation formula is as follows:

[0034] P robot =R·P camera +T (3)

[0035] Among them, P robotis the three-dimensional coordinate of the point in the robot tool coordinate system;

[0036] The pixel coordinates (u, v) are mapped to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is expressed as:

[0037]

[0038] The pixel coordinates of the center of the line connecting the centers of the large circle and the small circle on the calibration plate are mapped between the pixel coordinate system and the robot tool coordinate system, and the pose in the robot tool coordinate system is expressed as (X, Y, Z, Rz), where X, Y, and Z are obtained through the mapping relationship, and Rz is the direction angle of the line connecting the two circle centers.

[0039] Furthermore, in the above method, based on the visual calibration posture C1, the first posture deviation Offset, and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position, including:

[0040] Step S40, fixing any position of the robot arm within the camera field of view as a second position, and taking the second position as the current position;

[0041] Step S41, using a camera to capture a second image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent Z-axis teaching;

[0042] Step S42, obtaining the radius of the large circle in the second image through connected domain detection and minimum circumscribed circle algorithm;

[0043] Step S43, calculating the great circle radius deviation between the current great circle radius in the second image and the great circle radius in the first image during visual internal calibration, if the great circle radius deviation value is greater than a preset radius deviation threshold, proceeding to step S44, otherwise proceeding to step S45;

[0044] Step S44, based on the large circle radius deviation, calculate the Z-axis offset, adjust the Z-axis position of the robot arm according to the Z-axis offset, obtain the adjusted third position of the robot arm, take the third position as the current position, and return to step S41;

[0045] Step S45, fixing the current position of the robot arm, using a camera to capture a third image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent XYRz axis teaching;

[0046] Step S46, detecting the midpoint position of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two center points by connected domain detection and minimum circumscribed circle algorithm;

[0047] Step S47, based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two centers are converted from pixel coordinates to a posture in the robot tool coordinate system, recorded as a third posture, and the third posture is compared with the visual calibration posture C1 recorded during the visual internal calibration to obtain a calculated second posture deviation ΔT. If the second posture deviation is greater than a preset posture deviation threshold, step S48 is entered, otherwise step S49 is entered;

[0048] Step S48, combining the current third posture of the robot arm with the second posture deviation ΔT to obtain a new robot posture as the fourth posture, and adjusting the robot arm to the current fourth posture. Even if the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, returning to step S45;

[0049] Step S49, repeating steps S41 to S48 once;

[0050] Step S50, performing posture combination calculation on the current posture of the robot arm after step S49 and the first posture deviation Offset, and obtaining the final posture of the robot arm used for gripping the plate position with the gripper as the teaching posture of the plate position.

[0051] Furthermore, in the above method, in step S44, the Z-axis offset calculation formula is: (distance between the camera and the calibration plate*large circle radius deviation / current large circle radius).

[0052] Furthermore, in the above method, in step S47, the calculation formula of the second posture deviation ΔT is as follows:

[0053]

[0054] Wherein, ΔT: the second posture deviation; T1: the third posture; T2: the visual calibration posture C1 recorded during visual internal calibration.

[0055] Furthermore, in the above method, when the robot arm is a six-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0056] The calibration plate is placed at the plate position that the gripper of the robot arm needs to grip, wherein four circles are arranged at intervals on the calibration plate, and the centers of the four circles are not collinear;

[0057] Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including:

[0058] Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view.

[0059] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration posture, including:

[0060] Fix the robot arm at any position within the camera field of view as the fourth position, use the camera to shoot a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera intrinsic parameters and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution as the visual calibration pose;

[0061] Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including:

[0062] Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera at the fifth position to capture a fifth image of a calibration plate containing four circles, obtain the current posture of the six axes of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

[0063] According to another aspect of the present invention, there is further provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: execute any of the methods described above.

[0064] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0065] Processor; and

[0066] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0067] Compared with the prior art, the vision-based robot grasping and releasing point-free teaching method provided by the present invention realizes automated Z-axis and XYRz-axis teaching by placing a calibration plate with two circles at the target position and using the camera on the robot arm for detection. This automated teaching process simplifies the operation steps, reduces the errors caused by manual intervention, and improves the efficiency and accuracy of the calibration process. The present invention can automatically complete the calculation of the robot's grasping and releasing points, reduce the workload of the operator, and increase the accuracy and reliability of point teaching.

[0068] The present invention can accurately obtain the relative posture relationship between the calibration plate and the camera through precise visual algorithms, including: detection of circular features, coordinate conversion and posture adjustment. Through multiple iterations of Z-axis and XYRz teaching, and using formulas to calculate Z-axis offset and posture offset, the robot's position and posture in space are highly accurate. This high-precision visual teaching method effectively reduces the error in the teaching process and improves the reliability and stability of the robot's operation.

[0069] The present invention adopts the strategy of repeated teaching and error threshold judgment to ensure that each teaching result has high consistency and accuracy. This method is particularly important for robot operations that require high precision and repeatability. It can effectively avoid operational errors caused by inaccurate teaching. The automatic adjustment and feedback mechanism during the teaching process enables the robot to correct deviations autonomously, further increasing the accuracy of the final teaching result. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of a vision-based robot grasping and releasing point-free teaching method according to an embodiment of the present invention;

[0071] Figure 2 It is a flow chart of visual internal calibration in a vision-based robot grasping and releasing point teaching-free method according to an embodiment of the present invention;

[0072] Figure 3 The present invention is a flowchart of visual teaching in a vision-based robot grasping and releasing point-free teaching method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0074] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPU), input / output interfaces, network interfaces and memories.

[0075] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0077] like Figure 1 As shown, the present invention provides a vision-based robot grasping and releasing point-free teaching method, the method comprising:

[0078] Placing a calibration plate at a plate position to be clamped by a gripper of a robot arm, wherein the calibration plate includes at least two circles; and the robot arm is equipped with a camera and a gripper;

[0079] Drag the robot arm until the camera can detect the shooting range of all circles on the calibration plate in the photos taken by the camera, which is used as the camera's field of view;

[0080] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain a visual calibration posture;

[0081] The robot arm is dragged for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, the posture deviation is obtained, and it is iterated based on the posture deviation to correct the current posture to obtain the teaching posture of the board position.

[0082] Preferably, when the robot arm is a four-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0083] Step S1, placing a calibration plate: placing the calibration plate at a plate position to be clamped by the gripper of the robot arm, wherein a large circle and a small circle are arranged at intervals on the calibration plate, and a camera and a gripper are installed on the robot arm;

[0084] Here, a calibration plate with two circles can be placed at the target position, one of the two circles is a circle with a larger radius, and the other is a circle with a smaller radius, and the two circles can be spaced apart on the calibration plate;

[0085] Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including:

[0086] Step S2, dragging the robot arm: dragging the robot arm until the shooting range of the large circle and the small circle on the calibration plate can be detected in the photo taken by the camera as the camera field of view;

[0087] Here, the camera field of view refers to the shooting range in which the large circle and the small circle on the calibration plate can be detected in the photos taken by the camera;

[0088] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including:

[0089] like Figure 2 As shown, step S3, visual internal calibration: based on the calibration plate and the camera field of view, perform visual internal calibration to obtain the visual calibration pose C1 and the first pose deviation Offset, and establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system. If the visual internal calibration has been completed, skip this step;

[0090] Preferably, step S3 comprises:

[0091] Step S31, taking a photo: taking a photo using a camera on the robot arm, fixing the robot arm at any position within the camera field of view as a first position, and taking a first image of the calibration plate containing the large circle and the small circle using the camera at the first position;

[0092] Step S32, detecting the coordinates of the center of the circle in the first image and recording the parameters: using the connected domain detection and the minimum circumscribed circle algorithm to process the captured first image, identifying the pixel coordinates of the centers of the large circle and the small circle in the first image and the radius of the large circle, measuring and recording the distance between the camera and the calibration plate, and recording the following parameters: the radius of the large circle (pixel unit), the distance between the camera and the calibration plate (mm), the midpoint position of the line connecting the centers of the large circle and the small circle (pixel coordinates), and the direction (angle) of the line connecting the two centers;

[0093] Here, the distance between the camera and the calibration plate can be used in the subsequent formula (2);

[0094] The distance between the camera and the calibration plate in the visual internal calibration can be determined according to the position of the plate to be clamped in the visual internal calibration (the position of the calibration plate) and the camera's shooting position;

[0095] Step S33, establishing a mapping relationship between the pixel coordinate system and the robot tool coordinate system: calculating the camera's intrinsic and extrinsic parameter matrices through a calibration algorithm, using the pixel coordinates of the centers of the large circle and the small circle in the acquired first image, and the distance between the camera and the calibration plate, to establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system;

[0096] Here, the mapping relationship is used in the subsequent step 47;

[0097] Preferably, step S33 includes:

[0098] Step S331, perform camera intrinsic calibration: calculate the camera intrinsic parameter matrix K through the intrinsic parameter calibration algorithm, including: focal length (fx, fy), principal point coordinates (cx, cy) and distortion parameters;

[0099] Step S332: perform camera extrinsic calibration: use the extrinsic calibration algorithm to calculate the extrinsic matrix of the camera relative to the robot tool coordinate system using the three-dimensional coordinates of the corresponding point and the projection of the point in the pixel coordinate system, including: rotation matrix R and translation vector T;

[0100] Step S333: Based on the results of the camera intrinsic calibration and the camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established. Preferably, step S333 includes:

[0101] Step S3331: Normalize pixel coordinates: According to formula (1), the pixel coordinates are converted into normalized camera coordinates to eliminate the influence of camera distortion.

[0102]

[0103] Among them, K is the camera internal parameter matrix, (u,v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate.

[0104] Step S3332: Calculate the three-dimensional coordinates in the camera coordinate system: According to formula (2) and using the known depth D, the normalized camera coordinates are converted into the actual three-dimensional coordinates P in the camera coordinate system. camera , where the depth D is the distance between the camera and the calibration plate,

[0105]

[0106] Step S3333: Conversion from the camera coordinate system to the robot tool coordinate system: According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates P of the point in the camera coordinate system are converted to camera , converted into the three-dimensional coordinates P in the robot tool coordinate system robot , where the external parameter matrix includes the rotation matrix R and the translation vector T,

[0107] P robot =R·P camera +T (3)

[0108] Among them, P robot is the 3D coordinate of the point in the robot tool coordinate system.

[0109] Step S3334: Comprehensive mapping formula: Map the pixel coordinates (u, v) to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is expressed as:

[0110]

[0111] The pixel coordinates of the center of the line connecting the centers of the large circle and the small circle on the calibration plate are mapped between the pixel coordinate system and the robot tool coordinate system, and the pose in the robot tool coordinate system is expressed as (X, Y, Z, Rz), where X, Y, and Z are obtained through the mapping relationship, and Rz is the direction angle of the line connecting the two circle centers.

[0112] Step S34, recording the pose of the photographing point: converting the midpoint position of the line connecting the two center positions of the large circle and the small circle obtained in step S32 and the direction angle of the line connecting the two center positions from pixel coordinates to the pose in the robot tool coordinate system, and recording them as the visual calibration pose C1; at the same time, recording the pose in the robot coordinate system of the first position of the current camera, recorded as pose C2;

[0113] Step S35, move to the teaching point and record the posture: move the robot arm to the position of the calibration plate that the gripper needs to clamp, and ensure that the gripper accurately clamps the calibration plate; record the posture of the robot at this time, recorded as posture C3;

[0114] Step S36, calculate the posture deviation: calculate the first posture deviation between the posture C2 and the posture C3 to obtain the first posture deviation Offset.

[0115] Optionally, C1, C2 C3 and the first pose deviation Offset can be a numerical array containing X, Y, Z and Rz, which respectively represent the coordinates of the X-axis, Y-axis, Z-axis and the rotation angle Rz around the Z-axis. The robot of the present invention can be a four-axis robot of X, Y, Z and Rz, and the robot does not have Rx and Ry.

[0116] Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including:

[0117] like Figure 3 As shown, in step S4, based on the visual calibration posture C1 and the first posture deviation Offset, and establishing a mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position.

[0118] Step S40, fixing any position of the robot arm within the camera field of view as a second position, and taking the second position as the current position;

[0119] Step S41, taking a photo: using a camera to take a second image of the calibration plate including the large circle and the small circle at the current position, and starting the subsequent Z-axis teaching;

[0120] Here, the second position in step S41 may be the first position in step S31 within the camera field of view, or may be any other position within the camera field of view except the first position in step S31;

[0121] Step S42, detecting the radius of the great circle in the second image: obtaining the radius of the great circle in the second image by connected domain detection and minimum circumscribed circle algorithm;

[0122] Step S43, calculating the radius deviation: calculating the radius deviation of the great circle between the current great circle radius in the second image and the great circle radius in the first image in the visual internal calibration step S32. If the great circle radius deviation value is greater than a preset radius deviation threshold, proceed to step S44; otherwise, proceed to step S45.

[0123] Here, you can set the radius deviation threshold to 0.01, and then convert the large circle radius deviation into the Z-axis posture deviation in millimeters;

[0124] Step S44, calculating the Z-axis offset and moving the robot arm: based on the large circle radius deviation, the Z-axis offset is calculated, and the Z-axis position of the robot arm is adjusted according to the Z-axis offset to obtain the adjusted third position of the robot arm. After taking the third position as the current position, return to step S41;

[0125] Here, the Z-axis offset calculation formula in step S44 is: (distance between camera and calibration plate*large circle radius deviation / current large circle radius), wherein the distance between camera and calibration plate can be obtained in step S32;

[0126] Step S45, taking a photo: fixing the current position of the robot arm, using a camera to take a third image of the calibration plate containing the large circle and the small circle, and starting the subsequent teaching of the XYRz axis;

[0127] Here, since the robot does not have Rx and Ry, Rx and Ry do not need to be taught; the current position may be the second position or the adjusted third position, or the fourth position adjusted in the subsequent step S48;

[0128] Step S46, detecting the midpoint position of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two centers: the midpoint position of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two centers can be detected by using connected domain detection and a minimum circumscribed circle algorithm;

[0129] Step S47, calculating the posture deviation: based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system established in step 3, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two centers are converted from pixel coordinates to the posture in the robot tool coordinate system, recorded as the third posture, and the third posture is compared with the visual calibration posture C1 recorded during the visual internal calibration to obtain the calculated second posture deviation ΔT. If the second posture deviation is greater than the preset posture deviation threshold, step S48 is entered, otherwise, step S49 is entered;

[0130] Preferably, the calculation formula of the second posture deviation ΔT is as follows:

[0131]

[0132] Wherein, ΔT: the deviation of the second posture; T1: the third posture; T2: the visual calibration posture C1 recorded during visual internal calibration;

[0133] Here, the posture deviation threshold is 0.01, the units of X and Y are millimeters, and the unit of Rz is angle; it can be set that as long as one of X, Y, and Rz is greater than the preset posture deviation threshold, the process proceeds to step S48;

[0134] Step S48, combining the current third posture of the robot arm with the second posture deviation ΔT to obtain a new robot posture as the fourth posture, and adjusting the robot arm to the current fourth posture. Even if the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, returning to step S45;

[0135] Preferably, the current third posture of the robot arm is combined with the second posture deviation ΔT to obtain a new robot posture, including:

[0136] Step S481, separating the translation and rotation parts of the current third posture of the robot arm, and separating the translation and rotation parts of the second posture deviation;

[0137] Step S482, converting the rotation part of the current third posture of the robot arm from the Euler angle into a first rotation matrix, and converting the rotation part of the second posture deviation into a second rotation matrix;

[0138] Step S483, combining the first rotation matrix and the second rotation matrix to obtain a combined rotation matrix;

[0139] Step S484, based on the combined rotation matrix, the translation vector of the current third posture of the rotating robot arm is combined with the translation vector of the second posture deviation to obtain a combined translation vector;

[0140] Step S485, converting the combined rotation matrix back to Euler angles and normalizing them to obtain normalized Euler angles;

[0141] Step S486, based on the normalized Euler angles and the combined translation vector, a final combined posture is obtained as a new robot posture.

[0142] Step S49, repeating steps S41 to S48 once;

[0143] Here, this step is to repeat the Z-axis teaching (step S41 to step S44) and XYRz teaching (step S45 to step S48) once; this step is an iterative step, which can further avoid errors;

[0144] Step S50, calculate the final pick-up and release points: perform posture combination calculation on the current posture of the robot arm after step S49 and the first posture deviation Offset obtained in step S36 during visual internal calibration, and obtain the final posture of the robot arm used for gripping the board with the gripper as the teaching posture of the board position.

[0145] Here, the current posture of the robot arm after step S49 is completed and the first posture deviation are combined to obtain the robot posture for clamping the plate position.

[0146] After obtaining the final posture of the robot arm used for the gripper to grip the board position, that is, the taught posture of the board position, the calibration plate of the board position is subsequently replaced with the test hole plate. The gripper on the robot arm can then accurately grip the test hole plate in the board position based on the taught posture of the board position.

[0147] In one embodiment of the present invention, the present invention can be expanded to a method for grasping and releasing points without teaching of a six-axis robot. When the robot arm is a six-axis robot arm, the calibration plate is placed at the plate position that the gripper of the robot arm needs to grip. The calibration plate includes at least two circles, including:

[0148] The calibration plate is placed at the plate position that needs to be clamped by the gripper of the robot arm, wherein four circles are arranged at intervals on the calibration plate, and the centers of the four circles are not collinear;

[0149] Here, 4 circles are added to the calibration plate, and the world coordinate position (X, Y in the calibration plate coordinate system) of each circle is known; the 4 circles can be set at intervals, the radii of the 4 circles can be different, and the centers of the 4 circles are not collinear;

[0150] Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including:

[0151] Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view.

[0152] Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration posture, including:

[0153] Fix the robot arm at any position within the camera field of view as the fourth position, use the camera to shoot a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera intrinsic parameters and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution as the visual calibration pose;

[0154] Here, when the camera captures the calibration plate, the pixel coordinates (ui, vi) of the centers of these circles are obtained through connected domain detection and the minimum circumscribed circle algorithm;

[0155] Since the 3D coordinates (Xi, Yi, 0) of each feature on the calibration plate in the calibration plate coordinate system are known (Z = 0 plane), and the corresponding center pixel coordinates (ui, vi) have also been obtained, this forms 4 sets of 2D-3D corresponding points, thereby obtaining the complete posture of the plane in space;

[0156] Combining the camera's intrinsic and extrinsic parameters, the complete 6D pose of the calibration plate in the robot tool coordinate system is obtained through PnP solution.

[0157] Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including:

[0158] Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera at the fifth position to capture a fifth image of a calibration plate containing four circles, obtain the current posture of the six axes of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

[0159] Here, the detected posture is compared with the "visual calibration posture" of the initial shooting point to obtain the posture deviation, and then it is iterated to correct the posture.

[0160] According to another aspect of the present invention, there is further provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: execute any of the methods described above.

[0161] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0162] Processor; and

[0163] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0164] In summary, the vision-based robot grasping and releasing point-free teaching method provided by the present invention realizes automated Z-axis and XYRz-axis teaching by placing a calibration plate with two circles at the target position and using the camera on the robot arm for detection. This automated teaching process simplifies the operating steps, reduces the errors caused by manual intervention, and improves the efficiency and accuracy of the calibration process. The present invention can automatically complete the calculation of the robot's grasping and releasing points, reduce the workload of the operator, and increase the accuracy and reliability of point teaching.

[0165] The present invention can accurately obtain the relative posture relationship between the calibration plate and the camera through precise visual algorithms, including: detection of circular features, coordinate conversion and posture adjustment. Through multiple iterations of Z-axis and XYRz teaching, and using formulas to calculate Z-axis offset and posture offset, the robot's position and posture in space are highly accurate. This high-precision visual teaching method effectively reduces the error in the teaching process and improves the reliability and stability of the robot's operation.

[0166] The present invention adopts the strategy of repeated teaching and error threshold judgment to ensure that each teaching result has high consistency and accuracy. This method is particularly important for robot operations that require high precision and repeatability. It can effectively avoid operational errors caused by inaccurate teaching. The automatic adjustment and feedback mechanism during the teaching process enables the robot to correct deviations autonomously, further increasing the accuracy of the final teaching result.

[0167] The detailed contents of the device embodiments of the present invention can be found in the corresponding parts of the method embodiments, which will not be described in detail here.

[0168] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0169] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0170] In addition, a part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.

[0171] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A vision-based robot grasping and releasing point-free teaching method, characterized in that: include: Placing a calibration plate at a plate position to be clamped by a gripper of a robot arm, wherein the calibration plate includes at least two circles; and the robot arm is equipped with a camera and a gripper; Drag the robot arm until the camera can detect the shooting range of all circles on the calibration plate in the photos taken by the camera, which is used as the camera's field of view; Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain a visual calibration posture; Drag the robot arm for visual teaching to get the current position and posture; Based on the current posture and the visual calibration posture, the posture deviation is obtained, and iteration is performed based on the posture deviation to correct the current posture to obtain the teaching posture of the board position.

2. The vision-based robot grasping and releasing point-free teaching method according to claim 1, characterized in that: When the robot arm is a four-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including: The calibration plate is placed at the plate position that needs to be clamped by the gripper of the robot arm, wherein a large circle and a small circle are arranged on the calibration plate at intervals; Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including: Drag the robot arm until the large and small circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view. Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration posture, including: Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established; Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including: Based on the visual calibration posture C1, the first posture deviation Offset and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position.

3. The vision-based robot grasping and releasing point-free teaching method as claimed in claim 2, characterized in that: Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including: Fixing the robot arm at any position within the field of view of the camera as a first position, and using the camera to capture a first image of the calibration plate including the large circle and the small circle at the first position; The first image captured is processed using connected domain detection and a minimum circumscribed circle algorithm to identify the radius of the large circle, the midpoint of the line connecting the centers of the large circle and the small circle, and the direction angle of the line connecting the centers of the two circles, and to measure and record the distance between the camera and the calibration plate; The camera's intrinsic and extrinsic parameter matrices are calculated by a calibration algorithm, and a mapping relationship between a pixel coordinate system and a robot tool coordinate system is established using the pixel coordinates of the centers of the large circle and the small circle in the acquired first image and the distance between the camera and the calibration plate; The midpoint position of the line connecting the two center positions of the large circle and the small circle and the direction angle of the line connecting the two center positions are converted from pixel coordinates to the pose in the robot tool coordinate system and recorded as the visual calibration pose C1; at the same time, the pose in the robot coordinate system of the current camera at the first position is recorded as pose C2; Move the robot arm to the position where the calibration plate needs to be clamped by the gripper, and ensure that the gripper accurately clamps the calibration plate; record the robot's posture at this time, which is recorded as posture C3; Calculate the first pose deviation between pose C2 and pose C3 to obtain the first pose deviation Offset.

4. The vision-based robot grasping and releasing point-free teaching method as claimed in claim 3, characterized in that: The camera's intrinsic and extrinsic matrices are calculated by the calibration algorithm, and the mapping relationship between the pixel coordinate system and the robot tool coordinate system is established using the pixel coordinates of the centers of the large and small circles in the acquired first image and the distance between the camera and the calibration plate, including: Through the intrinsic calibration algorithm, the intrinsic parameter matrix K of the camera is calculated, including: focal length (fx, fy), principal point coordinates (cx, cy) and distortion parameters; Through the external parameter calibration algorithm, the extrinsic parameter matrix of the camera relative to the robot tool coordinate system is calculated using the three-dimensional coordinates of the corresponding point and the projection of the point in the pixel coordinate system, including: rotation matrix R and translation vector T; Based on the results of camera intrinsic calibration and camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established.

5. The vision-based robot grasping and releasing point-free teaching method as claimed in claim 4, characterized in that: Based on the results of the camera intrinsic calibration and the camera extrinsic calibration, the mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including: According to formula (1), the pixel coordinates are converted to normalized camera coordinates to eliminate the influence of camera distortion. Among them, K is the camera internal parameter matrix, (u,v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate; According to formula (2), and using the known depth D, the normalized camera coordinates are converted to the actual three-dimensional coordinates P in the camera coordinate system camera , where the depth D is the distance between the camera and the calibration plate; According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates P of the point in the camera coordinate system are camera , converted into the three-dimensional coordinates P in the robot tool coordinate system robot , wherein the external parameter matrix includes a rotation matrix R and a translation vector T; Map pixel coordinates (u,v) to three-dimensional coordinates (X,Y,Z) in the robot tool coordinate system.

6. The vision-based robot grasping and releasing point-free teaching method according to claim 5, characterized in that: Actual 3D coordinates P camera The calculation formula is as follows: The three-dimensional coordinate P in the robot tool coordinate system robot The calculation formula is as follows: P robot =R·P camera +T (3) Among them, P robot is the three-dimensional coordinate of the point in the robot tool coordinate system; The pixel coordinates (u, v) are mapped to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is expressed as: The pixel coordinates (u, v) of the center of the line connecting the centers of the large circle and the small circle on the calibration plate are mapped between the pixel coordinate system and the robot tool coordinate system, and the pose in the robot tool coordinate system is expressed as (X, Y, Z, Rz), where X, Y, Z are obtained through the mapping relationship, and Rz is the direction angle of the line connecting the two center points.

7. The vision-based robot grasping and releasing point-free teaching method according to claim 5, characterized in that: Based on the visual calibration posture C1, the first posture deviation Offset, and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position, including: Step S40, fixing any position of the robot arm within the camera field of view as a second position, and taking the second position as the current position; Step S41, using a camera to capture a second image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent Z-axis teaching; Step S42, obtaining the radius of the large circle in the second image through connected domain detection and minimum circumscribed circle algorithm; Step S43, calculating the great circle radius deviation between the current great circle radius in the second image and the great circle radius in the first image during visual internal calibration, if the great circle radius deviation value is greater than a preset radius deviation threshold, proceeding to step S44, otherwise proceeding to step S45; Step S44, based on the large circle radius deviation, calculate the Z-axis offset, adjust the Z-axis position of the robot arm according to the Z-axis offset, obtain the adjusted third position of the robot arm, take the third position as the current position, and return to step S41; Step S45, fixing the current position of the robot arm, using a camera to capture a third image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent teaching of the X, Y, and Rz axes; Step S46, detecting the midpoint position of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two center points by connected domain detection and minimum circumscribed circle algorithm; Step S47, based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two centers are converted from pixel coordinates to a posture in the robot tool coordinate system, recorded as a third posture, and the third posture is compared with the visual calibration posture C1 recorded during the visual internal calibration to obtain a calculated second posture deviation ΔT. If the second posture deviation is greater than a preset posture deviation threshold, step S48 is entered, otherwise step S49 is entered; Step S48, combining the current third posture of the robot arm with the second posture deviation ΔT to obtain a new robot posture as the fourth posture, and adjusting the robot arm to the current fourth posture. Even if the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, returning to step S45; Step S49, repeating steps S41 to S48 once; Step S50, performing posture combination calculation on the current posture of the robot arm after step S49 and the first posture deviation Offset, and obtaining the final posture of the robot arm used for gripping the plate position with the gripper as the teaching posture of the plate position.

8. The vision-based robot grasping and releasing point-free teaching method according to claim 7, characterized in that: In step S44, the Z-axis offset calculation formula is: (distance between the camera and the calibration plate*large circle radius deviation / current large circle radius).

9. The vision-based robot grasping and releasing point-free teaching method according to claim 7, characterized in that: In step S47, the calculation formula of the second posture deviation ΔT is as follows: Wherein, ΔT: the second posture deviation; T1: the third posture; T2: the visual calibration posture C1 recorded during visual internal calibration.

10. The vision-based robot grasping and releasing point-free teaching method according to claim 1, characterized in that: When the robot arm is a six-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including: The calibration plate is placed at the plate position that the gripper of the robot arm needs to grip, wherein four circles are arranged at intervals on the calibration plate, and the centers of the four circles are not collinear; Drag the robot arm until the camera can detect all the circular areas on the calibration plate in the photos taken by the camera, which is the camera's field of view, including: Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view. Based on the calibration plate and the camera field of view, visual internal calibration is performed to obtain the visual calibration posture, including: Fix the robot arm at any position within the camera field of view as the fourth position, use the camera to shoot a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera intrinsic parameters and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution as the visual calibration pose; Drag the robot arm for visual teaching to get the current posture; based on the current posture and the visual calibration posture, get the posture deviation, iterate based on the posture deviation to correct the current posture, and get the teaching posture of the board position, including: Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera at the fifth position to capture a fifth image of a calibration plate containing four circles, obtain the current posture of the six axes of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

11. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: perform the method according to any one of claims 1 to 10.

12. A computer device, wherein: include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to: perform a method as claimed in any one of claims 1 to 9.

Images