Dual-camera verification method for material positioning and grasping

Abstract

The dual-camera verification method for material positioning and grasping described in this invention employs a dual-camera system to provide two independent viewpoints and an expanded field of view. This aims to significantly improve positioning accuracy and effectively enhance the accuracy of grasping point calculation through a verification mechanism that averages the two positioning results. The method includes the following steps: Step 1) Dual-camera configuration; Step 2) Feature point extraction; Step 3) Hand-eye calibration; Step 4) Reference point teaching; Step 5) Acquiring an image of the actual material to be grasped; Step 6) Calibration conversion; Step 7) Verification mechanism; Step 8) Grasp point calculation.

Description

Technical Field

[0001] This application proposes a control method that improves accuracy through dual-camera verification during material positioning and grasping. It is applicable to precision operation sites such as new energy battery packs and belongs to the field of automation control and vision technology. Background Technology

[0002] With the rapid development of computer vision technology, 2D cameras have been widely used in various industrial inspection, robot navigation, and automation control fields. For example, in industrial automation and logistics warehousing, 2D cameras are commonly used for cargo identification and positioning to improve operational accuracy and efficiency. Compared to 3D cameras, 2D cameras have advantages such as lower cost and simpler structure.

[0003] In the process of grasping materials or workpieces, achieving high-precision positioning is crucial for the success of the operation. Existing 2D cameras have several shortcomings in terms of field of view and effective verification mechanisms, thus limiting their efficiency and reliability in practical operations. For example, a prior patent application, CN202211097933.8, entitled "A Single-Camera Positioning Guidance Method," limits the field of view of a single camera due to the limited height of the camera relative to the material to be positioned. This prevents the camera from capturing all the details of the material at once, especially when the material is large. This not only affects the comprehensiveness of the positioning but may also lead to blind spots during the grasping process, increasing the uncertainty and risk of the operation. Regarding positioning accuracy, if the field of view of a single camera needs to cover the entire large material, a high-resolution industrial camera must be used. High-precision industrial cameras are expensive, potentially exceeding the cost of using two low-resolution industrial cameras. Furthermore, when using a single camera for positioning, if an error occurs, the system typically lacks an effective verification mechanism. A single camera struggles to detect the material's orientation and posture when identifying incoming materials. This lack of information can lead to problems during the grasping process, such as grasping errors or misalignment. Although vision systems can set certain deviation thresholds to detect anomalies, the system may fail to recognize such errors when the positioning error does not exceed the set threshold, leading to the accumulation of positioning errors and affecting the stability of the entire vision positioning system.

[0004] In view of the above, this patent application is hereby filed. Summary of the Invention

[0005] This application proposes a dual-camera verification method for material positioning and grasping. It addresses the problems of the existing technology by employing a dual-camera system to provide two independent perspectives and expand the field of view. The aim is to significantly improve positioning accuracy and effectively enhance the accuracy of grasping point calculation through a verification mechanism that averages the two positioning results.

[0006] To achieve the above design objectives, the dual-camera verification method for material positioning and grasping includes the following steps:

[0007] Step 1) Dual-camera configuration;

[0008] Two 2D cameras are used to photograph the material to obtain image data;

[0009] Step 2) Feature point extraction;

[0010] Feature points of the material are extracted from images acquired by a 2D camera. The coordinates of the feature points on the image plane are represented as (x, y). img y img );

[0011] Step 3) Hand-eye calibration;

[0012] The "eye on hand" calibration method is adopted to calculate the hand-eye calibration matrix based on feature points and robotic arm coordinates;

[0013] Step 4) Baseline teaching;

[0014] Using the physical coordinates (X) of the robotic arm gripper base Y base ) as a teaching grasp point;

[0015] Step 5) Obtain an image of the actual material to be grabbed;

[0016] In actual production, two 2D cameras are used to capture current images of the material.

[0017] Step 6), Calibration Conversion;

[0018] Using the feature point image coordinates (x) extracted in step 2) img y img The calibration matrix M obtained in step 1) and step 2) is used to transform the physical coordinates of the robotic arm according to the following formula.

[0019]

[0020] Step 7), verification mechanism;

[0021] The calibration transformation results obtained from two 2D cameras were used for cross-verification. The physical coordinates obtained from the first 2D camera were set as follows: The physical coordinates obtained by the second 2D camera are Calculate the positional deviation between the two cameras using the following formula.

[0022]

[0023] Set a predefined tolerance threshold of T, if VXerror >T or VY error If the value is greater than T, it indicates that there is a problem with the direction or posture of the material to be grabbed, triggering an alarm; otherwise, it is considered that the verification has passed.

[0024] Step 8) Calculate the grab points;

[0025] The grab points are calculated using the following formula.

[0026]

[0027] If a safety threshold is pre-set to D, and VX = |X grad -X base |>D or VY=|Y grad -Y base If |>D, then the material to be grasped deviates too much from the reference point, and the grasping point needs to be recalculated starting from step 5); otherwise, the robotic arm performs the grasping action.

[0028] Furthermore, in step 3) of the hand-eye calibration process, the robotic arm is moved and rotated to allow the 2D camera to take pictures of the material from different positions and angles;

[0029] Based on the feature point coordinates extracted in step 2) and the coordinates of the robotic arm, calculate the hand-eye calibration matrices of the two 2D cameras respectively, so as to determine the transformation relationship between the camera coordinate system and the robotic arm coordinate system;

[0030] The transformation matrix M obtained through calibration is expressed as follows:

[0031] Where R is the rotation matrix and t is the translation vector;

[0032] The matrix is ​​used to represent the coordinates (X) in the camera coordinate system. cam Y cam Transform to coordinates in the robot's coordinate system (X) hand Y hand The coordinate transformation formula is as follows:

[0033]

[0034] Furthermore, during the reference point teaching process in step 4), the material is placed on the worktable, the position of the robotic arm is adjusted so that the gripper is aligned with the material, and the physical coordinates (X, Y, F, G) of the robotic arm gripper at this time are recorded. base Y base ) as a teaching grasp point;

[0035] Move the robotic arm to the camera position (X) basecam Y basecam The materials were photographed to serve as teaching photography points.

[0036] Further, in step 2), the feature points are edge points, corner points, or intersection points on the material surface. The coordinates of the feature points are extracted from the images acquired by the 2D camera using Harris corner detection, SIFT, or SURF image processing algorithms from the OPENCV library.

[0037] Further, step 3) of calculating the hand-eye calibration matrix includes: first, translating the robotic arm to capture nine images; determining the transformation relationship between the camera coordinate system and the robotic arm coordinate system using the coordinates of feature points and the robotic arm in these images; and then using a least squares optimization algorithm to calculate the hand-eye calibration matrix.

[0038] Then, by operating the robotic arm to obtain the position data of three points at three different angles, and using the method of determining a circle based on the three points, the rotation center C = (CenterX, CenterY) is calculated to complete the rotation calibration;

[0039] Finally, perform origin normalization, taking the rotation center as the origin. At this point, the hand-eye calibration matrix is:

[0040] In summary, the advantages and beneficial effects of this application are as follows:

[0041] 1. It improves the overall positioning accuracy and system robustness of materials, especially large workpieces, and can capture all the characteristics of materials more comprehensively, thus improving the overall positioning and gripping effect.

[0042] 2. This application employs a dual-camera system and image processing technology to acquire detailed information about the material from different perspectives, thereby improving the positioning accuracy of the material.

[0043] 3. This application adopts a dual-camera configuration to ensure all-round field of view coverage of materials, reduce blind spots that may occur under a single viewpoint, and improve the comprehensiveness of material identification.

[0044] 4. This application uses the mutual verification of a dual-camera system to detect and correct positioning deviations in real time, thereby reducing the risk of grasping failure due to errors.

[0045] 5. This application uses a dual-camera system for mutual verification, which can detect the direction and posture of materials in real time to reduce the risk of grasping errors or improper alignment of the fixture.

[0046] 6. By setting deviation thresholds and alarm mechanisms, this application can promptly detect and correct abnormal situations in the system, thereby enhancing the overall reliability and stability of the system. Attached Figure Description

[0047] The present application will now be further described in conjunction with the following figures;

[0048] Figure 1 A visual system configuration diagram for implementing the verification method described in this application;

[0049] Figure 2 A schematic diagram showing the installation of two 2D cameras;

[0050] Figure 3 This is a flowchart of the dual-camera verification method for material positioning and grasping described in this application;

[0051] Figure 4 This is a schematic diagram of camera displacement to acquire nine images by controlling a robotic arm to move continuously. Detailed Implementation

[0052] Example 1, such as Figure 1 and Figure 2 As shown, to implement a dual-camera verification method for material positioning and grasping, the following vision system is used to capture images using dual cameras and accurately locate the target object (such as materials conveyed on a production line) and calculate the grasping point during subsequent processing. Specifically, the vision system includes the following devices:

[0053] The robotic arm, preferably a six-degree-of-freedom industrial robotic arm, consists of a base, joints, and an end effector; the robotic arm receives control commands and performs corresponding gripping and releasing actions on the workpiece by communicating with the industrial control computer described below.

[0054] A 2D camera is set on each side of the robotic arm. The distance between the two 2D cameras and the two edges of the material to be inspected is equal to that of the two edges, so as to acquire image data symmetrically. The two 2D cameras are respectively connected to the industrial control computer to transmit the captured image data to the industrial control computer for processing.

[0055] The industrial computer, as the core control unit of the system, is responsible for receiving image data transmitted from the camera and performing subsequent processing and calculations. Specifically, corresponding 2D positioning programs, including image registration modules, hand-eye calibration modules, and gripping and releasing point calculation modules, can be installed on the industrial computer.

[0056] like Figure 3 As shown, the dual-camera verification method for material positioning and grasping proposed in this application includes the following steps:

[0057] Step 1) Dual-camera configuration;

[0058] Two 2D cameras are mounted at both ends of the robotic arm that grips the material to ensure that the imaging range covers the left and right edges of the material and that the distance between the two 2D cameras and the two edges of the material to be inspected is equal, that is, the left and right edges of the material are located at the imaging centers of the two 2D cameras respectively.

[0059] Adjust and fix the camera position to ensure stable shooting during the movement of the robotic arm;

[0060] Check and adjust the lighting conditions to ensure that both cameras have similar lighting environments when shooting;

[0061] Two 2D cameras are used to photograph the material to obtain image data;

[0062] Step 2) Feature point extraction;

[0063] Using image processing algorithms, edge detection, corner detection, or line finding algorithms can be selected to extract feature points of materials from images acquired by a 2D camera. The coordinates of the feature points on the image plane are represented as (x, y, x). img y img );

[0064] Feature points include edge points, corner points, intersection points, or other points with significant contrast and characteristics on the material surface. These feature points should be easily identifiable and extracted from the 2D image for subsequent processing. To improve the accuracy and stability of feature point extraction, image processing algorithms from the OpenCV library are used for feature point extraction.

[0065] Harris corner detection locates feature points by detecting corners in an image, and is suitable for detecting corners and intersections of materials;

[0066] SIFT (Scale Invariant Feature Transform) is used to detect key points in an image and calculate the descriptors of these key points. It is suitable for handling various scale and rotational changes on the surface of materials.

[0067] SURF (Accelerated Robust Features) is similar to SIFT, but it has higher computational efficiency and is suitable for feature point extraction in real-time applications.

[0068] The algorithm described above helps to extract feature point coordinates from images acquired by 2D cameras, ensuring that the extracted feature points have high stability and recognizability, and providing accurate data for subsequent hand-eye calibration and grasp point calculation.

[0069] Gamma adjustment is applied to images captured by a 2D camera to improve image brightness and contrast;

[0070] Specifically, the Gamma value is set to 1.5 to improve the visibility of dark areas in the image and enhance the overall image quality;

[0071] The Hough transform algorithm is used for line finding, extracting two mutually perpendicular lines. Their intersection point is taken as a feature point, and the coordinates of the feature point on the image plane are represented as (x, y). img y img );

[0072] Step 3): Hand-eye calibration;

[0073] Adopt the "eye-in-hand" calibration method, and move and rotate the robotic arm to enable the 2D camera to take photos of the material from different positions and angles;

[0074] Calculate the hand-eye calibration matrix based on the feature point coordinates extracted in Step 2) and the coordinates of the robotic arm.

[0075] First, according to the camera displacement diagram as Figure 4 shown, control the robotic arm to translate in a "zigzag" shape to continuously take nine images, and the distance the robotic arm moves each time is 10 mm. Among them, the fifth point is the center point, and its coordinates are defined as (0, 0), and the coordinates of the remaining points are plane coordinates relative to the fifth point; compared with other points, the material feature points taken and obtained at the fifth point can better reflect the spatial relationship between the camera and the robotic arm, and the material feature points are always within the image range when taking pictures at the remaining points;

[0076] Record a set of robotic arm coordinates (U, V), and obtain the feature points through Step 2) to get a set of image coordinates (X, Y); the conversion relationship between the robotic arm coordinate system and the image coordinate system is:

[0077]

[0078] Use the following least squares optimization algorithm formula,

[0079]

[0080] Calculate to obtain the hand-eye calibration matrix

[0081] Then, set the rotation angle of the robotic arm at each point to be 10° each time, and rotate the robotic arm three times at the fifth point to obtain the captured images at three angles of -10°, 0°, and 10° respectively;

[0082] According to the method of determining a circle by three points, calculate the rotation center C = (CenterX, CenterY) in the robotic arm coordinate system through M1 to complete the rotation calibration;

[0083] Finally, perform origin normalization, and regard the rotation center as the origin. At this time, the hand-eye calibration matrix is

[0084] Calculate the hand-eye calibration matrices of the two 2D cameras respectively to determine the conversion relationship between the camera coordinate system and the manipulator coordinate system;

[0085] The conversion matrix M obtained through calibration is expressed as:

[0086] Where R is the rotation matrix and t is the translation vector;

[0087] The matrix is ​​used to represent the coordinates (X) in the camera coordinate system. cam Y cam Transform to coordinates in the robot's coordinate system (x) hand Y hand The coordinate transformation formula is as follows:

[0088]

[0089] Step 4) Baseline teaching;

[0090] The material is placed on the worktable, and the position of the robotic arm is adjusted so that the gripper is aligned with the material. The physical coordinates (X, Y, F, G) of the robotic arm gripper at this time are recorded. base Y base ) as a teaching grasp point;

[0091] Move the robotic arm to the camera position (X) basecam Y basecam The materials were photographed to serve as teaching photography points;

[0092] The above-mentioned photo location (X) basecam Y basecam The position of the robot arm during the continuous translation process in step 2) of the hand-eye calibration procedure is shown below. Figure 4 The fifth point, being located in the middle of the other points, helps to capture more comprehensive and uniform image data from multiple perspectives, thereby reducing errors caused by extreme positions.

[0093] Place the material on the worktable, adjust the gripper position to align with the material, and record the physical coordinates of the robotic arm gripper at this moment as the teaching gripping point (X). base Y base );

[0094] Step 5) Obtain an image of the actual material to be grabbed;

[0095] In actual production, the robotic arm is moved to the teaching and photography point to obtain material images, and two 2D cameras are used to capture the current images of the material.

[0096] Step 6), Calibration Conversion;

[0097] Using the feature point image coordinates (x) extracted in step 2) img y img The calibration matrix M obtained in step 1) and step 2) is used to transform the physical coordinates of the robotic arm according to the following formula.

[0098]

[0099] Step 7), verification mechanism;

[0100] The calibration transformation results obtained from two 2D cameras were used for cross-verification. The physical coordinates obtained from the first 2D camera were set as follows: The physical coordinates obtained by the second 2D camera are Calculate the positional deviation between the two cameras using the following formula.

[0101]

[0102]

[0103] Set a predefined tolerance threshold of T, if VX error >T or VY error If the value is greater than T, it indicates that there is a problem with the direction or posture of the material to be grabbed, triggering an alarm; otherwise, it is considered that the verification has passed.

[0104] Step 8) Calculate the grab points;

[0105] The grab points are calculated using the following formula.

[0106]

[0107] If a safety threshold is pre-set to D, and VX = |X grad -X base |>D or VY=|Y grad -Y base If |>D, then the material to be grasped deviates too much from the reference point, and the grasping point needs to be recalculated starting from step 5); otherwise, the robotic arm performs the grasping action.

[0108] As described above, the embodiments given in conjunction with the accompanying drawings are merely preferred solutions for achieving the objectives of this invention. Those skilled in the art can draw inspiration from this and directly derive other alternative structures that conform to the design concept of this invention. Other structural features derived therefrom should also fall within the scope of the solutions described in this invention.

Claims

1. A dual-camera verification method for material positioning and grasping, characterized in that: Includes the following steps, Step 1) Dual-camera configuration; Two 2D cameras are used to photograph the material to obtain image data; Step 2) Feature point extraction; Feature points of the material are extracted from images acquired by a 2D camera. The coordinates of the feature points on the image plane are represented as (x, y). img y img ); Step 3) Hand-eye calibration; The "eye on hand" calibration method is adopted to calculate the hand-eye calibration matrix based on feature points and robotic arm coordinates; Step 4) Baseline teaching; Using the physical coordinates (X) of the robotic arm gripper base Y base ) as a teaching grasp point; Step 5) Obtain an image of the actual material to be grabbed; In actual production, two 2D cameras are used to capture current images of the material. Step 6), Calibration Conversion; Using the feature point image coordinates (x) extracted in step 2) img y img The calibration matrix M obtained in step 1) and step 2) is used to transform the physical coordinates of the robotic arm according to the following formula. Step 7), verification mechanism; The calibration transformation results obtained from two 2D cameras were used for cross-verification. The physical coordinates obtained from the first 2D camera were set as follows: The physical coordinates obtained by the second 2D camera are Calculate the positional deviation between the two cameras using the following formula. Set a predefined tolerance threshold of T, if VX error >T or VY error If the value is greater than T, it indicates that there is a problem with the direction or posture of the material to be grabbed, triggering an alarm; otherwise, it is considered that the verification has passed. Step 8) Calculate the grab points; The grab points are calculated using the following formula. If a safety threshold is pre-set to D, and VX = |X grad -X base |>D or VY=|Y grad -Y base If |>D, then the material to be grasped deviates too much from the reference point, and the grasping point needs to be recalculated starting from step 5); otherwise, the robotic arm performs the grasping action.

2. The dual-camera verification method for material positioning and grasping according to claim 1, characterized in that: In step 3) of the hand-eye calibration process, the robotic arm is moved and rotated so that the 2D camera can take pictures of the material from different positions and angles; Based on the feature point coordinates extracted in step 2) and the coordinates of the robotic arm, calculate the hand-eye calibration matrices of the two 2D cameras respectively, so as to determine the transformation relationship between the camera coordinate system and the robotic arm coordinate system; The transformation matrix M obtained through calibration is expressed as follows: Where R is the rotation matrix and t is the translation vector; The matrix is ​​used to represent the coordinates (X) in the camera coordinate system. cam Y cam Transform to coordinates in the robot's coordinate system (X) hand Y hand The coordinate transformation formula is as follows:

3. The dual-camera verification method for material positioning and grasping according to claim 1, characterized in that: During step 4) of the reference point teaching process, the material is placed on the worktable, the position of the robotic arm is adjusted so that the gripper is aligned with the material, and the physical coordinates (X, Y, F, Z) of the robotic arm gripper at this time are recorded. base Y base ) as a teaching grasp point; Move the robotic arm to the camera position (X) basecam Y basecam The materials were photographed to serve as teaching photography points.

4. The dual-camera verification method for material positioning and grasping according to claim 1, characterized in that: In step 2), the feature points are the edge points, corner points, or intersection points of the material surface. The coordinates of the feature points are extracted from the images acquired by the 2D camera using Harris corner detection, SIFT, or SURF image processing algorithms from the OPENCV library.

5. The dual-camera verification method for material positioning and grasping according to claim 2, characterized in that: Step 3) of calculating the hand-eye calibration matrix includes: first, taking nine images by translating the robotic arm; determining the transformation relationship between the camera coordinate system and the robotic arm coordinate system using the coordinates of feature points and the robotic arm in these images; and then calculating the hand-eye calibration matrix using a least squares optimization algorithm. Then, by operating the robotic arm to obtain the position data of three points at three different angles, and using the method of determining a circle based on the three points, the rotation center C = (CenterX, CenterY) is calculated to complete the rotation calibration; Finally, perform origin normalization, taking the rotation center as the origin. At this point, the hand-eye calibration matrix is:

Images