Vision-based robot pick-and-place point location method and apparatus

Abstract

The application provides a visual-based robot grasping and placing point position teaching-free method and device, which comprises the following steps: based on a calibration board and a camera field of view range, visual internal calibration is performed to obtain a visual calibration pose C1 and a first pose deviation Offset, and a mapping relationship between a pixel coordinate system and a robot tool coordinate system is established; based on the visual calibration pose C1, the first pose deviation Offset and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, a robot arm is dragged for visual teaching to obtain a teaching pose of the board position. The application can automatically complete the calculation of the robot grasping and placing point position, reduce the work burden of an operator, and increase the accuracy and reliability of point position teaching.

Description

Technical Field

[0001] The present invention relates to a vision-based robot grasping and releasing point teaching-free method and device. Background Art

[0002] Robots are widely used in automated production for high-precision tasks such as material handling, assembly, and quality inspection. These operations place extremely high demands on accuracy and stability, especially when handling tiny parts or complex work paths. Robots require precise teaching to ensure they can accurately perform their tasks.

[0003] Currently, most robotic systems rely on manual teaching to ensure that the robot can correctly identify objects in the working environment and complete tasks. This manual teaching process has several significant disadvantages:

[0004] (1) Time-consuming and labor-intensive: The manual teaching process is tedious and time-consuming, especially when there are multiple workstations or the calibration plate position is frequently adjusted. Operators need to participate in the teaching process for a long time, which increases the workload and reduces production efficiency.

[0005] (2) Unstable accuracy: Since manual teaching relies on the operator's experience and manual operation, it is easily affected by subjective factors such as fatigue, distraction, or differences in personal operating habits. This instability leads to low teaching accuracy, especially when handling precision tasks. It is difficult to ensure that each teaching can achieve the required accuracy.

[0006] (3) High risk of error: During manual teaching, even the slightest mistake can lead to deviations, resulting in errors in the robot’s path or gripping position. This error can be further amplified in subsequent tasks, especially in tasks requiring high precision, affecting production quality and even causing equipment damage. Summary of the Invention

[0007] The purpose of the present invention is to provide a vision-based robot grasping and releasing point-free teaching method and equipment.

[0008] To solve the above problems, the present invention provides a vision-based robot grasping and releasing point-free teaching method, comprising:

[0009] Placing a calibration plate at a position where the gripper of a robot arm needs to grip the plate, wherein the calibration plate includes at least two circles; the robot arm is equipped with a camera and a gripper;

[0010] Drag the robot arm until the camera can detect the shooting range of all circles on the calibration plate in the photos taken by the camera, which serves as the camera's field of view;

[0011] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain a visual calibration pose;

[0012] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture and obtain the teaching posture of the board position.

[0013] When the robot arm is a four-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0014] Place the calibration plate at the plate position that the gripper of the robot arm needs to grip, wherein a large circle and a small circle are arranged on the calibration plate at intervals;

[0015] Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including:

[0016] Drag the robot arm until the camera can detect the large and small circles on the calibration plate in the photo, which serves as the camera's field of view.

[0017] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose, including:

[0018] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established;

[0019] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including:

[0020] Based on the visual calibration posture C1, the first posture deviation Offset, and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position. Furthermore, in the above method, the intrinsic and extrinsic parameter matrices of the camera are calculated by the calibration algorithm, and the pixel coordinates of the centers of the large and small circles in the first image obtained, and the distance between the camera and the calibration plate are used to establish the mapping relationship between the pixel coordinate system and the robot tool coordinate system, including:

[0021] Perform camera intrinsic calibration: Calculate the camera's intrinsic parameter matrix K through the intrinsic calibration algorithm, including focal length (fx, fy), principal point coordinates (cx, cy) and distortion parameters;

[0022] Perform camera extrinsic calibration: Using the extrinsic calibration algorithm, the extrinsic matrix of the camera relative to the robot tool coordinate system is calculated using the 3D coordinates of the corresponding point and the projection of the point in the pixel coordinate system, including: rotation matrix R and translation vector T;

[0023] Based on the results of camera intrinsic calibration and camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established.

[0024] Furthermore, in the above method, based on the results of camera intrinsic parameter calibration and camera extrinsic parameter calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including:

[0025] According to formula (1), the pixel coordinates are converted to normalized camera coordinates to eliminate the influence of camera distortion.

[0026]

[0027] Among them, K is the camera internal parameter matrix, (u, v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate;

[0028] According to formula (2), and using the known depth D, the normalized camera coordinates are converted to the actual three-dimensional coordinates P in the camera coordinate system camera , where the depth D is the distance between the camera and the calibration plate;

[0029] According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates P of the point in the camera coordinate system are camera , converted into the three-dimensional coordinate P in the robot tool coordinate system robot , wherein the external parameter matrix includes the rotation matrix R and the translation vector T;

[0030] Map pixel coordinates (u,v) to three-dimensional coordinates (X,Y,Z) in the robot tool coordinate system.

[0031] Furthermore, in the above method, the actual three-dimensional coordinate P camera The calculation formula is as follows:

[0032]

[0033] The three-dimensional coordinate P in the robot tool coordinate system robot The calculation formula is as follows:

[0034] P robot =R·P camera +T (3)

[0035] Among them, P robotis the three-dimensional coordinate of the point in the robot tool coordinate system;

[0036] The pixel coordinates (u, v) are mapped to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is expressed as:

[0037]

[0038] The pixel coordinates of the center of the line connecting the centers of the large circle and the small circle on the calibration plate are mapped between the pixel coordinate system and the robot tool coordinate system, and the pose in the robot tool coordinate system is expressed as (X, Y, Z, Rz), where X, Y, and Z are obtained through the mapping relationship, and Rz is the direction angle of the line connecting the two circle centers.

[0039] Furthermore, in the above method, based on the visual calibration pose C1 and the first pose deviation Offset, and establishing a mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching pose of the plate position, including:

[0040] Step S40, fixing the robot arm at any position within the camera field of view as a second position, and setting the second position as the current position;

[0041] Step S41, using a camera to capture a second image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent Z-axis teaching;

[0042] Step S42, obtaining the radius of the great circle in the second image through connected component detection and minimum circumscribed circle algorithm;

[0043] Step S43, calculating the great circle radius deviation between the current great circle radius in the second image and the great circle radius in the first image during visual internal calibration. If the great circle radius deviation value is greater than a preset radius deviation threshold, proceed to step S44; otherwise, proceed to step S45;

[0044] Step S44: Calculate a Z-axis offset based on the great circle radius deviation, adjust the Z-axis position of the robot arm according to the Z-axis offset, obtain a third position of the robot arm after adjustment, set the third position as the current position, and return to step S41;

[0045] Step S45, fixing the current position of the robot arm, using a camera to capture a third image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent XYRz axis teaching;

[0046] Step S46, detecting the midpoint of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the centers of the two circles by using connected domain detection and a minimum circumscribed circle algorithm;

[0047] Step S47: Based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two circle centers are converted from pixel coordinates to a posture in the robot tool coordinate system, recorded as the third posture. The third posture is compared with the visual calibration posture C1 recorded during the visual internal calibration to calculate a second posture deviation ΔT. If the second posture deviation is greater than the preset posture deviation threshold, the process proceeds to step S48; otherwise, the process proceeds to step S49.

[0048] Step S48: Combine the current third posture of the robot arm with the second posture deviation ΔT to obtain a new robot posture as the fourth posture, and adjust the robot arm to the current fourth posture. After the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, return to step S45;

[0049] Step S49, repeating steps S41 to S48;

[0050] In step S50, the current posture of the robot arm after step S49 is completed is calculated with the first posture deviation Offset to obtain the final posture of the robot arm used for gripping the plate position as the teaching posture of the plate position.

[0051] Furthermore, in the above method, in step S44, the Z-axis offset calculation formula is: (distance between the camera and the calibration plate*great circle radius deviation / current great circle radius).

[0052] Furthermore, in the above method, in step S47, the calculation formula of the second posture deviation ΔT is as follows:

[0053]

[0054] Where, ΔT: the deviation of the second posture; T1: the third posture; T2: the visual calibration posture C1 recorded during visual internal calibration.

[0055] Furthermore, in the above method, when the robot arm is a six-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0056] Place the calibration plate at the plate position that the gripper of the robot arm needs to grip, wherein the calibration plate is provided with four circles at intervals, and the centers of the four circles are not collinear;

[0057] Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including:

[0058] Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view.

[0059] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose, including:

[0060] Fix the robot arm at any position within the camera's field of view as the fourth position, use the camera to capture a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera's intrinsic and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution, which is used as the visual calibration pose;

[0061] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including:

[0062] Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera to capture a fifth image of the calibration plate containing four circles at the fifth position, obtain the current six-axis posture of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

[0063] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor is enabled to execute any one of the methods described above.

[0064] According to another aspect of the present invention, there is further provided a computer device, comprising:

[0065] processor; and

[0066] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0067] Compared to existing technologies, the vision-based robot pick-and-release point teaching-free method provided by this invention achieves automated Z-axis and XYRZ-axis teaching by placing a calibration plate with two circles at the target location and detecting them using a camera on the robot arm. This automated teaching process simplifies the operation steps, reduces errors caused by manual intervention, and improves the efficiency and accuracy of the calibration process. This invention can automatically calculate the robot's pick-and-release points, reducing the operator's workload and increasing the accuracy and reliability of point teaching.

[0068] This invention utilizes precise visual algorithms, including circular feature detection, coordinate transformation, and pose adjustment, to accurately determine the relative pose relationship between the calibration plate and the camera. Multiple iterations of Z-axis and XYRz teaching, combined with formulas for calculating Z-axis and pose offsets, ensure highly accurate positioning and posture of the robot in space. This high-precision visual teaching method effectively reduces errors during the teaching process and improves the reliability and stability of robot operation.

[0069] This invention employs a strategy of repeated teaching and error threshold determination to ensure highly consistent and accurate teaching results. This approach is particularly important for robotic operations requiring high precision and repeatability, effectively avoiding operational errors caused by inaccurate teaching. The automatic adjustment and feedback mechanisms during the teaching process allow the robot to autonomously correct deviations, further increasing the accuracy of the final teaching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flowchart of a vision-based robot grasping and releasing point-free teaching method according to an embodiment of the present invention;

[0071] Figure 2 This is a flowchart of visual internal calibration in a vision-based robot grasping and releasing point teaching-free method according to one embodiment of the present invention;

[0072] Figure 3 The present invention is a flowchart of visual teaching in a vision-based robot grasping and releasing point-free teaching method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described in detail below with reference to the accompanying drawings.

[0074] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0075] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0076] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0077] like Figure 1 As shown, the present invention provides a vision-based robot grasping and releasing point-free teaching method, the method comprising:

[0078] Placing a calibration plate at a position where the gripper of a robot arm needs to grip the plate, wherein the calibration plate includes at least two circles; the robot arm is equipped with a camera and a gripper;

[0079] Drag the robot arm until the camera can detect the shooting range of all circles on the calibration plate in the photos taken by the camera, which serves as the camera's field of view;

[0080] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain a visual calibration pose;

[0081] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture and obtain the teaching posture of the board position.

[0082] Preferably, when the robot arm is a four-axis robot arm, the calibration plate is placed at a plate position that needs to be clamped by the gripper of the robot arm, and the calibration plate includes at least two circles, including:

[0083] Step S1, placing a calibration plate: placing the calibration plate at the plate position to be gripped by the gripper of the robot arm, wherein a large circle and a small circle are spaced apart on the calibration plate, and the robot arm is equipped with a camera and a gripper;

[0084] Here, a calibration plate with two circles can be placed at the target position. One of the two circles has a larger radius, and the other has a smaller radius. The two circles can be spaced apart on the calibration plate.

[0085] Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including:

[0086] Step S2, dragging the robot arm: drag the robot arm until the shooting range of the large circle and the small circle on the calibration plate can be detected in the photo taken by the camera, which is used as the camera field of view;

[0087] Here, the camera field of view refers to the shooting range in which the large circle and the small circle on the calibration plate can be detected in the photos taken by the camera;

[0088] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including:

[0089] like Figure 2 As shown, step S3, visual internal calibration: Based on the calibration plate and the camera field of view, perform visual internal calibration to obtain the visual calibration pose C1 and the first pose deviation Offset, and establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system. If the visual internal calibration has been completed, skip this step;

[0090] Preferably, step S3 includes:

[0091] Step S31, taking a photo: taking a photo using a camera on the robot arm, fixing the robot arm at any position within the camera's field of view as a first position, and taking a first image of the calibration plate containing the large circle and the small circle using the camera at the first position;

[0092] Step S32, detecting the coordinates of the center of the circle in the first image and recording the parameters: using the connected domain detection and minimum circumscribed circle algorithm to process the captured first image, identifying the pixel coordinates of the centers of the large circle and the small circle in the first image and the radius of the large circle, measuring and recording the distance between the camera and the calibration plate, and recording the following parameters: the radius of the large circle (pixel unit), the distance between the camera and the calibration plate (mm), the midpoint position of the line connecting the centers of the large circle and the small circle (pixel coordinates), and the direction (angle) of the line connecting the two centers;

[0093] Here, the distance between the camera and the calibration plate can be used in the subsequent formula (2);

[0094] The distance between the camera and the calibration plate in the visual internal calibration can be determined according to the position of the plate to be clamped in the visual internal calibration (the position of the calibration plate) and the camera's shooting position;

[0095] Step S33, establishing a mapping relationship between the pixel coordinate system and the robot tool coordinate system: using a calibration algorithm to calculate the camera's intrinsic and extrinsic parameter matrices, using the pixel coordinates of the centers of the large and small circles in the acquired first image and the distance between the camera and the calibration plate, establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system;

[0096] Here, the mapping relationship is used in the subsequent step 47;

[0097] Preferably, step S33 includes:

[0098] Step S331: perform camera intrinsic calibration: calculate the camera's intrinsic parameter matrix K using the intrinsic parameter calibration algorithm, including focal length (fx, fy), principal point coordinates (cx, cy), and distortion parameters;

[0099] Step S332: Perform camera extrinsic calibration: Calculate the extrinsic matrix of the camera relative to the robot tool coordinate system using the 3D coordinates of the corresponding point and the projection of the point in the pixel coordinate system through the extrinsic calibration algorithm, including: rotation matrix R and translation vector T;

[0100] Step S333: Based on the results of the camera intrinsic parameter calibration and the camera extrinsic parameter calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established. Preferably, step S333 includes:

[0101] Step S3331: Pixel coordinate normalization: According to formula (1), the pixel coordinates are converted to normalized camera coordinates to eliminate the influence of camera distortion.

[0102]

[0103] Among them, K is the camera internal parameter matrix, (u, v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate.

[0104] Step S3332: Calculate the three-dimensional coordinates in the camera coordinate system: According to formula (2), and using the known depth D, the normalized camera coordinates are converted into the actual three-dimensional coordinates P in the camera coordinate system camera , where the depth D is the distance between the camera and the calibration plate,

[0105]

[0106] Step S3333: Conversion from the camera coordinate system to the robot tool coordinate system: According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates P of the point in the camera coordinate system are converted to camera , converted into the three-dimensional coordinate P in the robot tool coordinate system robot , where the external parameter matrix includes the rotation matrix R and the translation vector T,

[0107] P robot =R·P camera +T (3)

[0108] Among them, P robot is the 3D coordinate of the point in the robot tool coordinate system.

[0109] Step S3334: Comprehensive mapping formula: Map the pixel coordinates (u, v) to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is expressed as:

[0110]

[0111] The pixel coordinates of the center of the line connecting the centers of the large circle and the small circle on the calibration plate are mapped between the pixel coordinate system and the robot tool coordinate system, and the pose in the robot tool coordinate system is expressed as (X, Y, Z, Rz), where X, Y, and Z are obtained through the mapping relationship, and Rz is the direction angle of the line connecting the two circle centers.

[0112] Step S34, recording the pose of the photographing point: Convert the midpoint of the line connecting the center positions of the large circle and the small circle obtained in step S32, and the direction angle of the line connecting the two center positions, from pixel coordinates to a pose in the robot tool coordinate system, and record this as the visual calibration pose C1. Simultaneously, record the pose of the current camera in the first position in the robot coordinate system, which is recorded as pose C2.

[0113] Step S35, move to the teaching point and record the posture: move the robot arm to the position where the calibration plate needs to be clamped by the gripper, ensuring that the gripper accurately clamps the calibration plate; record the posture of the robot at this time, which is recorded as posture C3;

[0114] Step S36, calculate the pose deviation: calculate the first pose deviation between pose C2 and pose C3 to obtain the first pose deviation Offset.

[0115] Optionally, C1, C2, C3 and the first pose deviation Offset can be an array of X, Y, Z and Rz values, which represent the coordinates of the X axis, Y axis, Z axis and the rotation angle Rz around the Z axis respectively. The robot of the present invention can be a four-axis robot with X, Y, Z and Rz, and the robot does not have Rx and Ry.

[0116] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including:

[0117] like Figure 3 As shown, in step S4, based on the visual calibration posture C1 and the first posture deviation Offset, and establishing a mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching posture of the plate position.

[0118] Step S40, fixing the robot arm at any position within the camera field of view as a second position, and setting the second position as the current position;

[0119] Step S41, taking a photo: using a camera to take a second image of the calibration plate containing the large circle and the small circle at the current position, and starting the subsequent Z-axis teaching;

[0120] Here, the second position in step S41 may be the first position in step S31 within the camera field of view, or any other position within the camera field of view except the first position in step S31;

[0121] Step S42, detecting the radius of the great circle in the second image: obtaining the radius of the great circle in the second image by connected component detection and minimum circumscribed circle algorithm;

[0122] Step S43, calculating the radius deviation: calculating the great circle radius deviation between the current great circle radius in the second image and the great circle radius in the first image during the visual internal calibration step S32. If the great circle radius deviation value is greater than a preset radius deviation threshold, proceeding to step S44; otherwise, proceeding to step S45;

[0123] Here, you can set the radius deviation threshold to 0.01. Later, you can convert the great circle radius deviation into the Z-axis pose deviation in millimeters.

[0124] Step S44, calculating the Z-axis offset and moving the robot arm: Based on the great circle radius deviation, the Z-axis offset is calculated, and the Z-axis position of the robot arm is adjusted according to the Z-axis offset to obtain a third position after adjustment of the robot arm. The third position is set as the current position, and the process returns to step S41;

[0125] Here, the Z-axis offset calculation formula in step S44 is: (distance between camera and calibration plate * great circle radius deviation / current great circle radius), where the distance between camera and calibration plate can be obtained in step S32;

[0126] Step S45, taking a picture: fixing the current position of the robot arm, using a camera to take a third image of the calibration plate containing the large circle and the small circle, and starting subsequent XYRz axis teaching;

[0127] Here, since the robot does not have Rx and Ry, Rx and Ry do not need to be taught; the current position may be the second position or the adjusted third position, or the fourth position adjusted in the subsequent step S48;

[0128] Step S46, detecting the midpoint of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two centers: the midpoint of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the two centers can be detected by using connected domain detection and a minimum circumscribed circle algorithm;

[0129] Step S47, calculating the pose deviation: Based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system established in step 3, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two circle centers are converted from pixel coordinates to a pose in the robot tool coordinate system, recorded as the third pose, and the third pose is compared with the visual calibration pose C1 recorded during the visual internal calibration to obtain a calculated second pose deviation ΔT. If the second pose deviation is greater than the preset pose deviation threshold, the process proceeds to step S48; otherwise, the process proceeds to step S49;

[0130] Preferably, the calculation formula of the second posture deviation ΔT is as follows:

[0131]

[0132] Wherein, ΔT: the deviation of the second posture; T1: the third posture; T2: the visual calibration posture C1 recorded during visual internal calibration;

[0133] Here, the posture deviation threshold is 0.01, the units of X and Y are millimeters, and the unit of Rz is angle. It can be set that as long as any one of X, Y, and Rz is greater than the preset posture deviation threshold, step S48 is entered;

[0134] Step S48: Combine the current third posture of the robot arm with the second posture deviation ΔT to obtain a new robot posture as the fourth posture, and adjust the robot arm to the current fourth posture. After the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, return to step S45;

[0135] Preferably, the current third posture of the robot arm is combined with the second posture deviation ΔT to obtain a new robot posture, including:

[0136] Step S481, separating the translation and rotation parts of the current third posture of the robot arm, and separating the translation and rotation parts of the second posture deviation;

[0137] Step S482: converting the rotation part of the current third posture of the robot arm from Euler angles into a first rotation matrix, and converting the rotation part of the second posture deviation into a second rotation matrix;

[0138] Step S483, combining the first rotation matrix and the second rotation matrix to obtain a combined rotation matrix;

[0139] Step S484: Based on the combined rotation matrix, the translation vector of the current third posture of the robot arm is rotated and combined with the translation vector of the second posture deviation to obtain a combined translation vector;

[0140] Step S485, converting the combined rotation matrix back to Euler angles and normalizing them to obtain normalized Euler angles;

[0141] Step S486: Based on the normalized Euler angles and the combined translation vector, a final combined pose is obtained as a new robot pose.

[0142] Step S49, repeating steps S41 to S48;

[0143] Here, this step is to repeat the Z-axis teaching (steps S41 to S44) and XYRz teaching (steps S45 to S48). This step is an iterative step that can further avoid errors.

[0144] Step S50, calculate the final pick-up and release point: perform posture combination calculation on the current posture of the robot arm after step S49 and the first posture deviation Offset obtained in step S36 during visual internal calibration to obtain the final posture of the robot arm used for gripping the plate position with the gripper, which is used as the teaching posture of the plate position.

[0145] Here, the current posture of the robot arm after step S49 is completed and the first posture deviation are combined to obtain the robot posture for clamping the panel.

[0146] After obtaining the final posture of the robot arm used for the gripper to grasp the board position, that is, the taught posture of the board position, the calibration plate of the board position is subsequently replaced with the test hole plate. The gripper on the robot arm can then accurately grasp the test hole plate in the board position based on the taught posture of the board position.

[0147] In one embodiment of the present invention, the present invention can be expanded to a method for grasping and placing points without teaching of a six-axis robot. When the robot arm is a six-axis robot arm, a calibration plate is placed at a plate position that needs to be grasped by the gripper of the robot arm. The calibration plate includes at least two circles, including:

[0148] Place the calibration plate at the plate position that the gripper of the robot arm needs to grip, wherein four circles are arranged at intervals on the calibration plate, and the centers of the four circles are not collinear;

[0149] Here, 4 circles are added to the calibration plate, and the world coordinate position (X, Y in the calibration plate coordinate system) of each circle is known; the 4 circles can be set at intervals, the radii of the 4 circles can be different, and the centers of the 4 circles are not collinear;

[0150] Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including:

[0151] Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view.

[0152] Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose, including:

[0153] Fix the robot arm at any position within the camera's field of view as the fourth position, use the camera to capture a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera's intrinsic and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution, which is used as the visual calibration pose;

[0154] Here, after the camera captures the calibration plate, the pixel coordinates (ui, vi) of the centers of these circles are obtained through connected domain detection and the minimum circumscribed circle algorithm;

[0155] Since the 3D coordinates (Xi, Yi, 0) of each feature on the calibration plate in the calibration plate coordinate system are known (Z = 0 plane), and the corresponding circle center pixel coordinates (ui, vi) have also been obtained, this forms 4 sets of 2D-3D corresponding points, thereby obtaining the complete posture of the plane in space;

[0156] Combining the camera's intrinsic and extrinsic parameters, the complete 6D pose of the calibration plate in the robot tool coordinate system is obtained through PnP solution.

[0157] Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including:

[0158] Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera to capture a fifth image of the calibration plate containing four circles at the fifth position, obtain the current six-axis posture of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

[0159] Here, the detected posture is compared with the "visual calibration posture" of the initial shooting point to obtain the posture deviation, and the posture is corrected by iteration.

[0160] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor is enabled to execute any one of the methods described above.

[0161] According to another aspect of the present invention, there is further provided a computer device, comprising:

[0162] processor; and

[0163] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0164] In summary, the vision-based robot pick-and-release point teaching-free method provided by the present invention achieves automated Z-axis and XYRZ-axis teaching by placing a calibration plate with two circles at the target location and detecting them using a camera on the robot arm. This automated teaching process simplifies the operation steps, reduces errors caused by manual intervention, and improves the efficiency and accuracy of the calibration process. The present invention can automatically calculate the robot's pick-and-release points, reducing the operator's workload and increasing the accuracy and reliability of point teaching.

[0165] This invention utilizes precise visual algorithms, including circular feature detection, coordinate transformation, and pose adjustment, to accurately determine the relative pose relationship between the calibration plate and the camera. Multiple iterations of Z-axis and XYRz teaching, combined with formulas for calculating Z-axis and pose offsets, ensure highly accurate positioning and posture of the robot in space. This high-precision visual teaching method effectively reduces errors during the teaching process and improves the reliability and stability of robot operation.

[0166] This invention employs a strategy of repeated teaching and error threshold determination to ensure highly consistent and accurate teaching results. This approach is particularly important for robotic operations requiring high precision and repeatability, effectively avoiding operational errors caused by inaccurate teaching. The automatic adjustment and feedback mechanisms during the teaching process allow the robot to autonomously correct deviations, further increasing the accuracy of the final teaching results.

[0167] The detailed contents of the various device embodiments of the present invention can be found in the corresponding parts of the various method embodiments, which will not be repeated here.

[0168] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0169] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0170] In addition, a portion of the present invention may be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instructions for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-carrying medium, and / or stored in a working memory of a computer device that operates according to the program instructions. Here, according to one embodiment of the present invention, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to operate based on the aforementioned methods and / or technical solutions according to multiple embodiments of the present invention.

[0171] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalents of the claims be encompassed within the present invention. Any figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A vision-based robot grasping and releasing point teaching-free method, characterized in that: The robot arm is a four-axis robot arm; a calibration plate is placed at a plate position to be gripped by the gripper of the robot arm, wherein a large circle and a small circle are spaced apart on the calibration plate; the robot arm is equipped with a camera and a gripper; Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including: Drag the robot arm until the camera can detect the large and small circles on the calibration plate in the photo, which serves as the camera's field of view. Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose, including: Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established; Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including: Based on the visual calibration pose C1, the first pose deviation Offset, and the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the robot arm is dragged for visual teaching to obtain the teaching pose of the plate position; Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose C1 and the first pose deviation Offset, and a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including: Fixing the robot arm at any position within the field of view of the camera as a first position, and using the camera to capture a first image of the calibration plate containing the large circle and the small circle at the first position; Processing the captured first image using connected domain detection and a minimum circumscribed circle algorithm to identify the radius of the large circle, the midpoint of the line connecting the centers of the large circle and the small circle, and the azimuth of the line connecting the centers of the two circles, and measuring and recording the distance between the camera and the calibration plate; The camera's intrinsic and extrinsic parameter matrices are calculated using a calibration algorithm. The pixel coordinates of the centers of the large and small circles in the acquired first image, as well as the distance between the camera and the calibration plate, are used to establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system. Convert the midpoint of the line connecting the centers of the large and small circles and the orientation angle of the line connecting the centers of the two circles from pixel coordinates to a pose in the robot tool coordinate system and record it as the calibration pose C1. At the same time, record the pose of the current camera in the first position in the robot coordinate system and record it as pose C2. Move the robot arm to the position where the calibration plate is to be gripped by the gripper, ensuring that the gripper accurately grips the calibration plate. Record the robot's posture at this point, which is recorded as posture C3. Calculate the first pose deviation between pose C2 and pose C3 to obtain the first pose deviation Offset.

2. The vision-based robot grasping and releasing point-free teaching method according to claim 1, characterized in that: The camera's intrinsic and extrinsic parameter matrices are calculated using a calibration algorithm. The pixel coordinates of the centers of the large and small circles in the acquired first image, as well as the distance between the camera and the calibration plate, are used to establish a mapping relationship between the pixel coordinate system and the robot tool coordinate system, including: Calculate the camera's intrinsic parameter matrix through the intrinsic parameter calibration algorithm , including: focal length (fx, fy), principal point coordinates (cx, cy) and distortion parameters; Through the extrinsic calibration algorithm, the extrinsic parameter matrix of the camera relative to the robot tool coordinate system is calculated using the three-dimensional coordinates of the corresponding point and the projection of the point in the pixel coordinate system, including: rotation matrix R and translation vector T; Based on the results of camera intrinsic calibration and camera extrinsic calibration, a mapping relationship between the pixel coordinate system and the robot tool coordinate system is established.

3. The vision-based robot grasping and releasing point-free teaching method according to claim 2, characterized in that: Based on the results of camera intrinsic calibration and camera extrinsic calibration, the mapping relationship between the pixel coordinate system and the robot tool coordinate system is established, including: According to formula (1), the pixel coordinates are converted to normalized camera coordinates to eliminate the influence of camera distortion. （1）； in, is the camera internal parameter matrix, (u,v) is the pixel coordinate, (x n ,y n ) is the normalized camera coordinate; According to formula (2), and using the known depth D, the normalized camera coordinates are converted to the actual three-dimensional coordinates in the camera coordinate system , where the depth D is the distance between the camera and the calibration plate; According to formula (3), and using the external parameter matrix, the actual three-dimensional coordinates of the point in the camera coordinate system are , converted into three-dimensional coordinates in the robot tool coordinate system , wherein the external parameter matrix includes the rotation matrix R and the translation vector T; Map pixel coordinates (u,v) to three-dimensional coordinates (X,Y,Z) in the robot tool coordinate system.

4. The vision-based robot grasping and releasing point-free teaching method according to claim 3, characterized in that: Actual 3D coordinates The calculation formula is as follows: （2）； 3D coordinates in the robot tool coordinate system The calculation formula is as follows: （3）； in, is the three-dimensional coordinate of the point in the robot tool coordinate system; The pixel coordinates (u, v) are mapped to the three-dimensional coordinates (X, Y, Z) in the robot tool coordinate system. The formula is: （4）； The pixel coordinates (u, v) of the center of the line connecting the centers of the large and small circles on the calibration plate are mapped to the robot tool coordinate system. The resulting pose in the robot tool coordinate system is represented as (X, Y, Z, Rz), where X, Y, and Z are obtained through the mapping relationship, and Rz is the orientation angle of the line connecting the two circle centers.

5. The vision-based robot grasping and releasing point-free teaching method according to claim 2, characterized in that: Based on the visual calibration pose C1 and the first pose deviation Offset, and establishing the mapping relationship between the pixel coordinate system and the robot tool coordinate system, drag the robot arm for visual teaching to obtain the teaching pose of the board position, including: Step S40, fixing the robot arm at any position within the camera field of view as a second position, and setting the second position as the current position; Step S41, using a camera to capture a second image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent Z-axis teaching; Step S42, obtaining the radius of the great circle in the second image through connected component detection and minimum circumscribed circle algorithm; Step S43, calculating the great circle radius deviation between the current great circle radius in the second image and the great circle radius in the first image during visual internal calibration. If the great circle radius deviation value is greater than a preset radius deviation threshold, proceed to step S44; otherwise, proceed to step S45; Step S44: Calculate a Z-axis offset based on the great circle radius deviation, adjust the Z-axis position of the robot arm according to the Z-axis offset, obtain a third position of the robot arm after adjustment, set the third position as the current position, and return to step S41; Step S45, fixing the current position of the robot arm, using a camera to capture a third image of the calibration plate containing the large circle and the small circle at the current position, and starting subsequent teaching of the X, Y, and Rz axes; Step S46, detecting the midpoint of the line connecting the large circle and the small circle in the third image and the direction of the line connecting the centers of the two circles by using connected domain detection and a minimum circumscribed circle algorithm; Step S47: Based on the mapping relationship between the pixel coordinate system and the robot tool coordinate system, the midpoint position of the line connecting the large circle and the small circle in the third image and the direction angle of the line connecting the two circle centers are converted from pixel coordinates to the posture in the robot tool coordinate system, recorded as the third posture, and the third posture is compared with the visual calibration posture C1 recorded during the visual internal calibration to obtain the second posture deviation. If the second posture deviation is greater than the preset posture deviation threshold, the process proceeds to step S48, otherwise the process proceeds to step S49; Step S48: The current third posture of the robot arm is offset from the second posture. Perform posture combination to obtain a new robot posture as the fourth posture, adjust the robot arm to the current fourth posture, and after the robot arm moves to the new robot posture, that is, the fourth position corresponding to the fourth posture as the current position, return to step S45; Step S49, repeating steps S41 to S48 once; In step S50, the current posture of the robot arm after step S49 is completed is calculated with the first posture deviation Offset to obtain the final posture of the robot arm used for gripping the plate position as the teaching posture of the plate position.

6. The vision-based robot grasping and releasing point-free teaching method according to claim 4, characterized in that: In step S44, the Z-axis offset is calculated as follows: (distance between the camera and the calibration plate * large circle radius deviation / current large circle radius).

7. The vision-based robot grasping and releasing point-free teaching method according to claim 5, characterized in that: In step S47, the second posture deviation The calculation formula is as follows: in, : Second pose deviation; : The third position; : Visual calibration posture C1 recorded during visual internal calibration.

8. A vision-based robot grasping and releasing point teaching-free method, characterized in that: The robot arm is a six-axis robot arm. The calibration plate is placed at the plate position that the robot arm's gripper needs to grasp. The calibration plate is provided with four circles at intervals, and the centers of the four circles are not collinear. The robot arm is equipped with a camera and a gripper. Drag the robot arm until the camera captures the image and can detect all the circles on the calibration plate. This is the camera's field of view, including: Drag the robot arm until the four circles on the calibration plate can be detected in the photos taken by the camera, which serves as the camera's field of view. Based on the calibration plate and the camera field of view, a visual internal calibration is performed to obtain the visual calibration pose, including: Fix the robot arm at any position within the camera's field of view as the fourth position, use the camera to capture a fourth image of the calibration plate containing four circles at the fourth position, and obtain the pixel coordinates (ui, vi) of the centers of the four circles in the fourth image through connected domain detection and the minimum circumscribed circle algorithm; based on the pixel coordinates (ui, vi) of the centers of the four circles and the 3-axis coordinates (Xi, Yi, 0) of the centers of the four circles in the calibration plate coordinate system, and in combination with the camera's intrinsic and extrinsic parameters, obtain the complete 6-axis pose of the centers of the four circles in the robot tool coordinate system through PnP solution, which is used as the visual calibration pose; Drag the robot arm for visual teaching to obtain the current posture; based on the current posture and the visual calibration posture, obtain the posture deviation, and iterate based on the posture deviation to correct the current posture to obtain the teaching posture of the board position, including: Fix the robot arm at any position within the camera's field of view as the fifth position, use the camera to capture a fifth image of the calibration plate containing four circles at the fifth position, obtain the current six-axis posture of the centers of the four circles in the robot tool coordinate system based on the fifth image, obtain the posture deviation based on the current posture and the visual calibration posture, iterate based on the posture deviation to correct the current posture, and obtain the teaching posture of the plate position.

9. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

10. A computer device, wherein: include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to: perform the method according to any one of claims 1 to 8.

Images