Camera movement detection and grabbing coordinate compensation method

Abstract

The invention discloses a camera movement detection and grabbing coordinate compensation method, and belongs to the technical field of automatic control and vision. The method does not depend on fixed calibration parameters, so that the purposes of obviously improving the positioning precision and the one-time grabbing success rate are achieved by accurately detecting the change of the position of the camera and mapping the related displacement to an actual grabbing coordinate system for compensation. A visual system is adopted to realize detection and accurate positioning of movement of a camera bound on a movable clamping jaw through a fixed calibration plate and an image algorithm processing mode; the visual system comprises a mechanical arm with six-axis degrees of freedom, a camera, a calibration plate and an industrial personal computer; comprising the following execution steps: step 1) installing a calibration plate; step 2) hand-eye calibration; 3) teaching and reference coordinate recording; 4) detecting the position of the camera; (5) camera position deviation judgment is carried out; and (6) the coordinates of the grabbing points are compensated when the materials are grabbed.

Description

Technical Field

[0001] The present application belongs to the field of automatic control and visual technology, and specifically proposes a compensation method for periodically detecting camera movement during material positioning and grasping and correcting grasping coordinates by accurately measuring the movement amount. The method is particularly suitable for high-precision grasping tasks in which the camera is fixed on a mobile robotic arm. Background Art

[0002] With the rapid development of computer vision technology, workpiece positioning guidance is a common and important application scenario in the field of industrial automation. Usually, in order to ensure the stability of the relative position between the camera and the grasped workpiece during shooting, the camera will be fixed on the gripper of the robotic arm. The camera obtains the coordinate conversion relationship with the mechanical gripper through calibration, and obtains the workpiece grasping coordinates through teaching. In the process of grasping materials or workpieces, how to achieve high-precision positioning is the key link to the success of the operation. However, in the actual movement of the robotic arm, factors such as mechanical vibration, temperature changes, and equipment aging may cause the camera position to change, which will directly affect the workpiece grasping accuracy and task success rate. Slight position deviation may lead to grasping failure, while severe displacement may cause the robot to collide with the object, causing equipment damage or material waste.

[0003] The prior art generally only determines whether the camera has changed when performing camera position detection. The positioning and guidance system relies on fixed calibration parameters and cannot detect and compensate for changes in the camera position, resulting in coordinate offset problems in high-precision grasping tasks. For example, the following previously disclosed Chinese patent application, application number CN202110435812.9, is named a camera position detection method, device, and storage medium. The scheme relies on a classification model of a set of image sets and identifiers to determine whether the camera has changed based on the number of images containing identifiers in the image set. Another example is application number CN202011563381.6, which is named a label production method, label, and detection method for thermal camera position detection. Its scheme only detects whether the four edges of the image are visible to determine whether the camera position is correct. It can only determine that the camera position has changed abnormally, and it is impossible to know the specific offset of the camera posture and compensate for changes in the camera position. It is impossible to solve the coordinate offset problem in high-precision grasping tasks, and recalibrating the camera will increase the manual debugging time.

[0004] Therefore, developing a grasping coordinate compensation method in camera movement detection and positioning grasping tasks is crucial to improving the accuracy of material positioning in industrial automation and ensuring equipment safety.

[0005] In view of this, this patent application is proposed. Summary of the invention

[0006] The present application proposes a camera motion detection and grasping coordinate compensation method, which aims to solve the problems existing in the above-mentioned prior art without relying on fixed calibration parameters, thereby achieving the purpose of significantly improving positioning accuracy and one-time grasping success rate by accurately detecting changes in camera position and mapping related displacements to the actual grasping coordinate system for compensation.

[0007] In order to achieve the above design objectives, the camera motion detection and grasping coordinate compensation method adopts a visual system to detect and accurately locate the movement of the camera bound to the mobile gripper through a fixed calibration plate and image algorithm processing; the visual system includes a robotic arm with six-axis freedom, a camera, a calibration plate and an industrial computer; and includes the following execution steps: step 1) calibration plate installation;

[0008] The calibration plate is fixedly installed in the area that the robot arm can reach, and the calibration plate will not shift during use;

[0009] Step 2) hand-eye calibration;

[0010] Use the "eyes on hands" calibration method to obtain the calibration matrix through nine-point calibration and rotation calibration

[0011] To convert the camera coordinate system to the world coordinate system of the robot;

[0012] Step 3) Teaching and recording of reference coordinates;

[0013] Adjust the position of the robot arm so that the gripper of the robot arm is aligned with the workpiece and record the physical point (x base ,y base ,r base ); where r represents the angle; the robot moves to the workpiece photography position and records the physical point (x cam ,y cam ,r cam ); The robot moves to the calibration plate photography position and records the calibration plate photography physical point (x cal ,y cal ,r cal );

[0014] Step 4) Camera position detection;

[0015] During the material grabbing process, the positioning grabbing system regularly operates the robot arm to the calibration plate to take pictures of physical points and re-collect the image of the calibration plate; extracts feature points through image processing algorithms, obtains image coordinates, and converts them into physical coordinates (x run_cal ,y run_cal ,r run_cal ); then the expression of the camera's posture change ΔT is,

[0016]

[0017] The expression of the camera's rotation change Δr is: Δr = r run_cal -r cal ;

[0018] Step 5) Determine the camera position offset;

[0019] Set a slight offset threshold Threshold 1 and a significant offset threshold Threshold 2, where Threshold 2> Threshold 1;

[0020] If ΔT and Δr are both less than Threshold1, it is considered that the camera has basically not moved and no gripping point coordinate compensation is required; if ΔT and Δr are both greater than Threshold1 and less than Threshold2, it is considered that the camera has moved slightly and gripping point coordinate compensation is required; if ΔT and Δr are both greater than Threshold2, it means that the camera has moved significantly and the camera needs to be reinstalled and calibrated;

[0021] Step 6) Compensation of the grabbing point coordinates when grabbing materials;

[0022] During the material grabbing process, the robot arm is moved to the workpiece position to take a photo, and the initial grabbing point P of the workpiece during operation is calculated by calibrating the matrix H0. old =(x old ,y old ,r old );

[0023] Repeat step 4) to obtain the posture change and rotation change again; the coordinates of the workpiece grasping point after compensation are determined according to the following expression:

[0024] Furthermore, in the step 1), the calibration plate adopts a checkerboard structure, which is composed of alternating black and white squares to form a calibration pattern.

[0025] Furthermore, in the step 2), a translation operation is first performed to complete the nine-point calibration, and then a rotation operation is performed to complete the rotation calibration to obtain the position change of the camera; wherein the translation operation includes a Y-axis translation after every three X-axis translations, and each movement distance is 5 to 15 mm, so that the material feature point is at the center point of the image, which is the fifth point of the nine-point calibration; after the translation operation, a rotation calibration is performed at the fifth point, and each rotation angle is 5° to 10°.

[0026] In summary, this application has the following advantages and beneficial effects:

[0027] 1. This application can accurately update the actual coordinates of the grasping point through periodic camera posture detection and dynamic coordinate compensation. Even if the camera position is slightly offset, the accuracy and success rate of positioning and workpiece grasping can still be guaranteed.

[0028] 2. This application not only issues an alarm when the deviation exceeds the threshold, but also avoids grasping errors and equipment damage through dynamic coordinate compensation, thereby significantly improving the safe operation of the equipment, maintaining production continuity and improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present application will now be further described with reference to the following drawings;

[0030] Figure 1 A schematic diagram of a visual system for the camera motion detection and grabbing coordinate compensation method described in this application;

[0031] Figure 2 It is a flow chart of the camera motion detection and grabbing coordinate compensation method;

[0032] Figure 3 is a schematic diagram of the calibration plate;

[0033] Figure 4 Calibrate path map for hand-eye; DETAILED DESCRIPTION

[0034] Embodiment 1, as Figure 1 As shown, in order to realize the camera movement detection and grasping coordinate compensation method, the following visual system is used to detect and accurately locate the movement of the camera bound to the mobile gripper through a fixed calibration plate and image algorithm processing, and the subsequent grasping coordinate compensation is performed based on the positioning guidance.

[0035] Specifically, the visual system includes the following equipment:

[0036] The robot arm adopts a six-degree-of-freedom industrial robot arm, which consists of a base, joints and end effectors, and can achieve flexible movement and grasping and releasing operations. The robot arm communicates with the industrial computer to receive control instructions and execute the corresponding grasping and releasing actions of the workpiece;

[0037] The industrial camera, installed on the robotic arm, can shoot the calibration plate at a fixed angle and obtain the position information of the object through image registration technology. Through the connection with the industrial computer, the captured image data is transmitted to the industrial computer for processing;

[0038] The calibration plate is a special graphic plate fixed on the workbench with unique and obvious identification features;

[0039] The industrial computer, as the core control unit of the system, is responsible for receiving the image data transmitted by the camera and performing subsequent processing and calculation; the corresponding 2D positioning program is installed on the industrial computer, including the single-point grasping module, the camera movement detection module, etc.

[0040] like Figures 2 to 4 As shown, the camera motion detection and grabbing coordinate compensation method includes the following execution steps:

[0041] Step 1) Calibration board installation;

[0042] The calibration plate should be fixedly installed in the area that the robot can reach to ensure that it does not shift during use; the calibration plate should be placed within the working range of the robot and fixed in position to provide a stable reference base;

[0043] like Figure 3 As shown, the calibration plate adopts a checkerboard type, which is composed of alternating black and white squares, so as to provide multiple feature points with higher contrast;

[0044] Step 2) hand-eye calibration;

[0045] Use the "eyes on hands" calibration method to obtain the calibration matrix through nine-point calibration and rotation calibration

[0046] Used to convert the camera coordinate system to the robot arm coordinate system (i.e. the world coordinate system);

[0047] Specifically, according to Figure 4 When operating the robot arm according to the path shown, a translation operation is first performed to complete the nine-point calibration, and then a rotation operation is performed to complete the rotation calibration to obtain the position change of the camera;

[0048] The translation operation includes a Y-axis translation after every three X-axis translations, and each movement distance is 10 mm, so that the material feature point is at the center of the image, which is the fifth point of the nine-point calibration. After the translation operation, the rotation calibration is performed at the fifth point, and each rotation angle is 5°, so that it is more convenient to convert the relative position with other points and facilitate the calculation of the conversion relationship.

[0049] Step 3) Teaching and recording of reference coordinates;

[0050] Adjust the position of the robot arm so that the gripper of the robot arm is aligned with the workpiece and record the physical point (x base ,y base ,r base ), where r represents the angle;

[0051] The robot arm moves to the workpiece photography position and records the physical point (x cam ,ycam ,r cam );

[0052] The robot moves to the calibration plate photography position and records the physical point (x cal ,y cal ,r cal );

[0053] During the above recording process, the distance between the workpiece and the calibration plate taken by the camera should be consistent to ensure the accuracy of the grab point coordinate compensation, and the photo position should be in the center of the image between the workpiece and the calibration plate;

[0054] Step 4) Camera position detection;

[0055] During the material grabbing process, the positioning grabbing system regularly (such as every day or every week) operates the robot arm to take pictures of physical points on the calibration plate and re-collects the image of the calibration plate;

[0056] The feature points are extracted by image processing algorithm, and the image coordinates are obtained and converted into physical coordinates (x run_cal ,y run_cal ,r run_cal ), the physical coordinates are the physical points of the calibration plate photographed in actual operation;

[0057] Then the expression of the camera's posture change ΔT is:

[0058]

[0059] The expression of the camera's rotation change Δr is: Δr = r run_cal -r cal ;

[0060] Step 5) Determine the camera position offset;

[0061] Two offset thresholds are defined, including a slight offset threshold Threshold 1 and a significant offset threshold Threshold 2, and Threshold 2> Threshold 1;

[0062] The offset threshold can be adjusted according to the specific application scenario: in situations where high accuracy is required, such as in, the threshold can be set to ±0.1mm and ±0.3mm; in general industrial applications, it can be set to ±0.3mm and ±1mm; if the environmental conditions are more stringent, or the image interference is large, or the visual positioning accuracy requirement is lower, it can be appropriately relaxed to ±1mm and ±3mm;

[0063] If ΔT and Δr are both less than Threshold1, it is considered that the camera has basically not moved and there is no need to compensate the grab point coordinates;

[0064] If ΔT and Δr are both greater than Threshold1 and less than Threshold2, it is considered that the camera has moved slightly and the grab point coordinates need to be compensated;

[0065] If both ΔT and Δr are greater than Threshold2, it means that the camera has moved significantly and may be unstable or fall off. The camera needs to be reinstalled and calibrated. At the same time, the system triggers an alarm mechanism and displays a warning message on the industrial computer to prompt the operator to calibrate the camera.

[0066] Step 6) Compensation of the grabbing point coordinates when grabbing materials;

[0067] During the material grabbing process, the robot arm is moved to the workpiece position to take a photo, and the initial grabbing point P of the workpiece during operation is calculated by calibrating the matrix H0. old =(x old ,y old ,r old );

[0068] Due to the change in camera posture, the origin of the camera coordinate system is displaced relative to the origin of the robot coordinate system, and the axis of the camera coordinate system is rotated relative to the axis of the robot coordinate system;

[0069] In order to ensure that the machine can accurately grasp the workpiece, the coordinates of the workpiece grasping point need to be adjusted according to the rotation change of the camera, and step 4) is repeated to obtain the posture change and rotation change again;

[0070] The coordinates of the workpiece gripping point after compensation are determined according to the following expression:

[0071]

[0072] By integrating periodic camera posture detection, posture offset judgment and dynamic coordinate compensation technology, this application can not only improve the accuracy of material positioning in industrial automation, but also enhance the safety of equipment. In particular, when the camera moves significantly, it can be discovered and measures can be taken in time to avoid possible operational errors and equipment damage.

[0073] The above contents and the embodiments given in the accompanying drawings are only preferred solutions for achieving the purpose of the present invention. Those skilled in the art can be inspired by this and directly deduce other alternative structures that conform to the design concept of the present invention. Other structural features obtained thereby should also belong to the scope of the solution described in the present invention.

Claims

1. A camera motion detection and grabbing coordinate compensation method, characterized in that: The visual system is used to detect and accurately locate the movement of the camera bound to the mobile gripper through a fixed calibration plate and image algorithm processing; the visual system includes a robotic arm with six-axis freedom, a camera, a calibration plate and an industrial computer; The following execution steps are included: Step 1) Calibration board installation; The calibration plate is fixedly installed in the area that the robot arm can reach, and the calibration plate will not shift during use; Step 2) hand-eye calibration; Use the "eyes on hands" calibration method to obtain the calibration matrix through nine-point calibration and rotation calibration To convert the camera coordinate system to the world coordinate system of the robot; Step 3) Teaching and benchmark coordinate recording; Adjust the position of the robot arm so that the gripper of the robot arm is aligned with the workpiece and record the physical point (x base ,y base ,r base ); where r represents the angle; the robot moves to the workpiece photography position and records the physical point (x cam ,y cam ,r cam ); The robot moves to the calibration plate photography position and records the calibration plate photography physical point (x cal ,y cal ,r cal ); Step 4) Camera position detection; During the material grabbing process, the positioning grabbing system regularly operates the robot arm to the calibration plate to take pictures of physical points and re-collect the image of the calibration plate; extracts feature points through image processing algorithms, obtains image coordinates, and converts them into physical coordinates (x run_cal ,y run_cal ,r run_cal ); then the expression of the camera's posture change ΔT is, The expression of the camera's rotation change Δr is: Δr = r run_cal -r cal ; Step 5) Camera position offset determination; Set a slight offset threshold Threshold 1 and a significant offset threshold Threshold 2, where Threshold 2> Threshold 1; If ΔT and Δr are both less than Threshold1, it is considered that the camera has basically not moved and no gripping point coordinate compensation is required; if ΔT and Δr are both greater than Threshold1 and less than Threshold2, it is considered that the camera has moved slightly and gripping point coordinate compensation is required; if ΔT and Δr are both greater than Threshold2, it means that the camera has moved significantly and the camera needs to be reinstalled and calibrated; Step 6) Compensation of the grabbing point coordinates when grabbing materials; During the material grabbing process, the robot arm is moved to the workpiece position to take a photo, and the initial grabbing point P of the workpiece during operation is calculated by calibrating the matrix H0. old =(x old ,y old ,r old ); Repeat step 4) to obtain the posture change and rotation change again; the coordinates of the workpiece grasping point after compensation are determined according to the following expression:

2. The camera motion detection and grabbing coordinate compensation method according to claim 1, characterized in that: In the step 1), the calibration plate adopts a checkerboard structure, which is composed of alternating black and white squares to form a calibration pattern.

3. The camera motion detection and grabbing coordinate compensation method according to claim 1, characterized in that: In the step 2), a translation operation is first performed to complete the nine-point calibration, and then a rotation operation is performed to complete the rotation calibration to obtain the position change of the camera; wherein the translation operation includes a Y-axis translation after every three X-axis translations, and each movement distance is 5 to 15 mm, so that the material feature point is at the center point of the image, which is the fifth point of the nine-point calibration; after the translation operation, a rotation calibration is performed at the fifth point, and each rotation angle is 5° to 10°.

Images