Double-camera verification method for material positioning and grabbing

Abstract

According to the double-camera verification method for material positioning and grabbing, a double-camera system is adopted to provide two independent visual angles and expand the visual field range, so that the positioning precision is remarkably improved by averaging a verification mechanism of two positioning results, and the accuracy of grabbing point calculation is effectively improved. Comprising the following steps: step 1), double-camera configuration; step 2), feature point extraction; step 3), performing hand-eye calibration; step 4), reference point demonstration; (5) an image of the actual to-be-grabbed material is obtained; (6) calibration conversion is carried out; step 7), a verification mechanism; and step 8), grabbing point calculation.

Description

Technical Field

[0001] This application proposes a control method that uses dual-camera verification to improve accuracy during material positioning and grasping, which can be applied to precise operation sites such as new energy battery packs, and belongs to the field of automated control and visual technology. Background Art

[0002] With the rapid development of computer vision technology, 2D cameras have become widely used in various industrial inspection, robotic navigation, and automated control fields. For example, in industrial automation and logistics warehousing, conveying and sorting operations often utilize 2D cameras for cargo identification and location detection to improve accuracy and efficiency. Compared to 3D cameras, 2D cameras offer advantages such as low cost and simple structure.

[0003] When grasping materials or workpieces, achieving high-precision positioning is crucial for successful operations. Existing 2D cameras still have numerous deficiencies in their field of view and effective verification mechanisms, limiting their effectiveness and reliability in practical applications. For example, a prior patent application, application number CN202211097933.8, entitled "A Single-Camera Positioning Guidance Method," states that due to the height restriction between the camera and the material being positioned, the field of view of a single camera is limited, making it impossible to capture all details of the material at once, especially with large materials. This not only affects the comprehensiveness of positioning but can also lead to blind spots during the grasping process, increasing operational uncertainty and risk. Regarding positioning accuracy, if a single camera's field of view needs to cover the entire large material, a high-resolution industrial camera is essential. High-precision industrial cameras are expensive, potentially exceeding the cost of using two low-resolution industrial cameras. Furthermore, when using a single camera for positioning, if positioning errors occur, the system often lacks an effective verification mechanism. This makes it difficult for a single camera to detect information such as the material's orientation and posture when identifying incoming materials. This lack of information can lead to problems during the grasping process, such as incorrect grasping or incorrect alignment. Although vision systems can set certain deviation thresholds to detect anomalies, when positioning errors do not exceed the set threshold, the system may not be able to identify such errors, which can lead to the accumulation of positioning errors and affect the stability of the entire vision positioning system.

[0004] In view of this, this patent application is hereby filed. Summary of the Invention

[0005] This application proposes a dual-camera verification method for material positioning and grasping, which aims to solve the problems existing in the above-mentioned existing technologies and adopts a dual-camera system to provide two independent perspectives and an expanded field of view, so as to significantly improve the positioning accuracy and effectively improve the accuracy of grasping point calculation through a verification mechanism that averages the two positioning results.

[0006] To achieve the above design objectives, the dual-camera verification method for material positioning and grasping includes the following steps:

[0007] Step 1) Dual camera configuration;

[0008] Use two 2D cameras to take pictures of the material to obtain image data;

[0009] Step 2), feature point extraction;

[0010] The feature points of the material are extracted from the image captured by the 2D camera. The coordinates of the feature points on the image plane are expressed as (x img ,y img );

[0011] Step 3), hand-eye calibration;

[0012] The "eye-on-hand" calibration method is used to calculate the hand-eye calibration matrix based on the feature points and the robot arm coordinates;

[0013] Step 4) Teaching of reference points;

[0014] The physical coordinates of the robot gripper (X base , Y base ) as the teaching grasping point;

[0015] Step 5) Obtain an image of the actual material to be grasped;

[0016] In the actual production process, two 2D cameras are used to capture the current image of the material;

[0017] Step 6), calibration conversion;

[0018] Using the feature point image coordinates (x img ,y img ) and the calibration matrix M obtained in step 3), the physical coordinates of the manipulator are converted according to the following formula:

[0019]

[0020] Step 7), verification mechanism;

[0021] Use the calibration conversion results obtained by the two 2D cameras to verify each other, and set the physical coordinates obtained by the first 2D camera to be The physical coordinates obtained by the second 2D camera are The position deviation between the two cameras is calculated according to the following formula:

[0022]

[0023] Set the predefined tolerance threshold to T. If VXerror >T or VY error >T, it means there is a problem with the direction or posture of the material to be grasped, and an alarm is triggered; otherwise, it is considered to have passed the verification;

[0024] Step 8), grasping point calculation;

[0025] The grab point is calculated using the following formula:

[0026]

[0027] The security threshold is set to D in advance. If VX=|X grad -X base |>D or VY=|Y grad -Y base |>D, the deviation of the material to be grasped from the reference point material is too large, and the grasping point needs to be recalculated starting from step 5); otherwise, the robot arm performs the grasping action.

[0028] Furthermore, in the step 3) of hand-eye calibration, the 2D camera is used to take photos of the material from different positions and angles by moving and rotating the robotic arm;

[0029] Calculate the hand-eye calibration matrices of the two 2D cameras based on the feature point coordinates extracted in step 2) and the coordinates of the manipulator to determine the conversion relationship between the camera coordinate system and the manipulator coordinate system;

[0030] The transformation matrix M obtained by calibration is expressed as:

[0031] Where R is the rotation matrix and t is the translation vector;

[0032] The matrix is ​​used to convert the coordinates (X cam , Y cam ) is converted to the coordinates in the robot coordinate system (X hand , Y hand ), the coordinate transformation formula is as follows:

[0033]

[0034] Furthermore, in the step 4) of benchmark teaching, the material is placed on the workbench, the position of the robot arm is adjusted so that the gripper is aligned with the material, and the physical coordinates (X base , Y base ) as the teaching grasping point;

[0035] Move the robotic arm to the photo taking position (X basecam , Y basecam ) to take pictures of the material as teaching photo points.

[0036] Furthermore, in step 2), the feature points are edge points, corner points or intersection points on the surface of the material, and the feature point coordinates are extracted from the image captured by the 2D camera using Harris corner detection, SIFT or SURF image processing algorithm in the OPENCV library.

[0037] Furthermore, the process of calculating the hand-eye calibration matrix in step 3) includes: first, translating the robotic arm to capture nine images, determining the transformation relationship between the camera coordinate system and the robotic arm coordinate system through the coordinates of the feature points in these images and the robotic arm coordinates, and calculating the hand-eye calibration matrix using the least squares optimization algorithm.

[0038] Then, the robot is operated to obtain the position data of three points at three different angles. The rotation center C = (CenterX, CenterY) is calculated based on the three points to complete the rotation calibration.

[0039] Finally, normalize the origin and use the rotation center as the origin. The hand-eye calibration matrix is:

[0040] In summary, this application has the following advantages and beneficial effects:

[0041] 1. Overall, it improves the positioning accuracy and system robustness of materials, especially large workpieces, and can more comprehensively capture all the characteristics of the materials, improving the overall positioning and grasping effects.

[0042] 2. This application uses a dual-camera system and image processing technology to obtain detailed information of materials from different perspectives and improve the positioning accuracy of materials.

[0043] 3. This application adopts a dual-camera configuration to ensure full coverage of the material field of view, reducing the blind spots that may appear under a single perspective and improving the comprehensiveness of material identification.

[0044] 4. This application uses the mutual verification of the dual-camera system to detect and correct positioning deviations in real time to reduce the risk of grasping failure due to errors.

[0045] 5. This application uses a dual-camera system to verify each other and can detect the direction, posture and other information of the material in real time to reduce the risk of grasping errors or incorrect alignment of the fixture.

[0046] 6. By setting deviation thresholds and alarm mechanisms, this application can promptly detect and correct abnormal situations in the system, thereby enhancing the overall reliability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present application will now be further described with reference to the following drawings;

[0048] Figure 1 A diagram showing the structure of a visual system for implementing the verification method described in this application;

[0049] Figure 2 Schematic diagram of the installation of two 2D cameras;

[0050] Figure 3 This is a flow chart of the dual-camera verification method for material positioning and grasping described in this application;

[0051] Figure 4 Schematic diagram of camera displacement for acquiring nine images by controlling the manipulator to translate continuously; DETAILED DESCRIPTION

[0052] Example 1, as Figure 1 and Figure 2 As shown in the figure, to implement a dual-camera verification method for material positioning and grasping, the following vision system is used to capture images with dual cameras and accurately locate the target object (such as materials conveyed on a production line) and calculate the grasping point during the subsequent processing. Specifically, the vision system includes the following equipment:

[0053] The robotic arm may preferably be an industrial robotic arm with six degrees of freedom, which is composed of a base, joints, and an end effector; the robotic arm receives control instructions and performs corresponding actions of grasping and releasing the workpiece by communicating with the industrial computer described below;

[0054] A 2D camera is set on each side of the robotic arm. The two 2D cameras are equidistant from the two edges of the material to be inspected so as to acquire image data symmetrically. The two 2D cameras are respectively connected to the following industrial computers for communication to transmit the captured image data to the industrial computer for processing.

[0055] The industrial computer, as the core control unit of the system, is responsible for receiving image data transmitted by the camera and performing subsequent processing and calculations. Specifically, the corresponding 2D positioning program can be installed on the industrial computer, including the image registration module, the hand-eye calibration module, and the grasping and releasing point calculation module.

[0056] like Figure 3 As shown, the dual-camera verification method for material positioning and grasping proposed in this application includes the following steps:

[0057] Step 1) Dual camera configuration;

[0058] Two 2D cameras are installed at both ends of the robotic arm that performs material grabbing to ensure that the imaging range covers the left and right edges of the material, and the two 2D cameras are equidistant from the two edges of the material to be inspected, that is, the left and right edges of the material are located at the imaging center of the two 2D cameras respectively;

[0059] Adjust and fix the position of the camera to ensure stable shooting during the movement of the robotic arm;

[0060] Check and adjust lighting conditions to ensure both cameras are shooting in similar lighting environments;

[0061] Use two 2D cameras to take pictures of the material to obtain image data;

[0062] Step 2), feature point extraction;

[0063] Using image processing algorithms, preferably edge detection, corner detection or line search algorithms, the feature points of the material are extracted from the image captured by the 2D camera. The coordinates of the feature points on the image plane are expressed as (x img ,y img );

[0064] Feature points include edge points, corner points, intersection points, or other points with significant contrast and characteristics on the surface of the material. These feature points should be easy to identify and extract from the 2D image to facilitate subsequent processing. To improve the accuracy and stability of feature point extraction, image processing algorithms in the OPENCV library are used for feature point extraction.

[0065] Harris corner detection locates feature points by detecting corners in an image. It is suitable for detecting corners and intersections of materials.

[0066] SIFT (Scale Invariant Feature Transform) is used to detect key points in an image and calculate the descriptors of these key points. It is suitable for processing various scale and rotation changes of the material surface.

[0067] SURF (Speeded Robust Features) is similar to SIFT, but has higher computational efficiency and is suitable for feature point extraction in real-time applications;

[0068] The above algorithm helps to extract feature point coordinates from images captured by 2D cameras, ensuring that the extracted feature points have high stability and recognition, providing accurate data for subsequent hand-eye calibration and grasp point calculation;

[0069] Gamma adjustment is performed on the images captured by the 2D camera to improve the brightness and contrast of the image;

[0070] Specifically, the Gamma value is set to 1.5 to improve the visibility of dark areas in the image and enhance the overall image quality;

[0071] Hough transform algorithm is used to search for straight lines, and two perpendicular straight lines are extracted. The intersection of the two lines is used as the feature point. The coordinates of the feature point on the image plane are expressed as (x img ,y img );

[0072] Step 3), hand-eye calibration;

[0073] Using the "eye-in-hand" calibration method, the 2D camera takes photos of the material from different positions and angles by moving and rotating the robotic arm;

[0074] The hand-eye calibration matrix is ​​calculated based on the feature point coordinates extracted in step 2) and the coordinates of the robotic arm.

[0075] First, follow the Figure 4 The camera displacement diagram shown controls the robotic arm to translate in a "J" shape to capture nine consecutive images, with each movement of 10 mm. The fifth point is the center point, defined as (0, 0). The coordinates of the remaining points are plane coordinates relative to the fifth point. Compared to other points, the material feature point captured and obtained at the fifth point better reflects the spatial relationship between the camera and the robotic arm, and this material feature point remains within the image range when captured at the remaining points.

[0076] Record a set of robot arm coordinates (U, V), and obtain feature points through step 2) to obtain a set of image coordinates (X, Y). The conversion relationship between the robot arm coordinate system and the image coordinate system is:

[0077]

[0078] Use the following least squares optimization algorithm formula,

[0079]

[0080] Calculate the hand-eye calibration matrix

[0081] Then, the robot arm is set to rotate 10° at each point, and the robot arm is rotated three times at the fifth point to obtain images at three angles of -10°, 0°, and 10° respectively;

[0082] According to the three-point circle method, the rotation center C = (CenterX, CenterY) in the manipulator coordinate system is calculated by M1 to complete the rotation calibration;

[0083] Finally, normalize the origin and use the rotation center as the origin. The hand-eye calibration matrix is:

[0084] Calculate the hand-eye calibration matrix of the two 2D cameras respectively to determine the transformation relationship between the camera coordinate system and the manipulator coordinate system;

[0085] The transformation matrix M obtained by calibration is expressed as:

[0086] Where R is the rotation matrix and t is the translation vector;

[0087] The matrix is ​​used to convert the coordinates (X cam , Y cam ) is converted to the coordinates in the robot coordinate system (x hand , Y hand ), the coordinate transformation formula is as follows:

[0088]

[0089] Step 4) Teaching of reference points;

[0090] Place the material on the workbench, adjust the position of the robot arm so that the gripper is aligned with the material, and record the physical coordinates of the robot arm gripper (X base , Y base ) as the teaching grasping point;

[0091] Move the robotic arm to the photo taking position (X basecam , Y basecam ), photograph the material as a teaching photo point;

[0092] The above photo location (X basecam , Y basecam ) is the center position of the robot arm during the continuous translation process in step 2) of the hand-eye calibration process. Figure 4 The fifth point in the image, because it is located in the middle of the other points, helps capture more comprehensive and uniform image data from multiple perspectives, thereby reducing errors caused by extreme positions.

[0093] Place the material on the workbench, adjust the position of the gripper to align with the material, and record the physical coordinates of the gripper at this time as the teaching grasping point (X base , Y base );

[0094] Step 5) Obtain an image of the actual material to be grasped;

[0095] In the actual production process, the robot arm is moved to the teaching photo point to obtain the material image, and two 2D cameras are used to capture the current image of the material respectively;

[0096] Step 6), calibration conversion;

[0097] Using the feature point image coordinates (x img ,y img ) and the calibration matrix M obtained in step 3), the physical coordinates of the manipulator are converted according to the following formula:

[0098]

[0099] Step 7), verification mechanism;

[0100] Use the calibration conversion results obtained by the two 2D cameras to verify each other, and set the physical coordinates obtained by the first 2D camera to be The physical coordinates obtained by the second 2D camera are The position deviation between the two cameras is calculated according to the following formula:

[0101]

[0102]

[0103] Set the predefined tolerance threshold to T. If VX error >T or VY error >T, it means there is a problem with the direction or posture of the material to be grasped, and an alarm is triggered; otherwise, it is considered to have passed the verification;

[0104] Step 8), grasping point calculation;

[0105] The grab point is calculated using the following formula:

[0106]

[0107] The security threshold is set to D in advance. If VX=|X grad -X base |>D or VY=|Y grad -Y base |>D, the deviation of the material to be grasped from the reference point material is too large, and the grasping point needs to be recalculated starting from step 5); otherwise, the robot arm performs the grasping action.

[0108] The embodiments described above, in conjunction with the accompanying drawings, are merely preferred solutions for achieving the objectives of the present invention. Those skilled in the art will readily be able to derive alternative structures consistent with the design concepts of the present invention based on these insights. Other structural features derived from these alternatives are also intended to fall within the scope of the present invention.

Claims

1. A dual-camera verification method for material positioning and grabbing, characterized in that: The following steps are included: Step 1), dual camera configuration; Two 2D cameras are used to take pictures of the material to obtain image data; Step 2), feature point extraction; The feature points of the material are extracted from the image captured by the 2D camera. The coordinates of the feature points on the image plane are expressed as (x img .y img ); Step 3), hand-eye calibration; The "eye on hand" calibration method is used to calculate the hand-eye calibration matrix based on the feature points and the robot arm coordinates; Step 4), benchmark point teaching; The physical coordinates of the robot gripper (X base , Y base ) as the teaching grasping point; Step 5) Obtain the image of the actual material to be grasped; In the actual production process, two 2D cameras are used to capture the current image of the material respectively; Step 6), calibration conversion; Using the feature point image coordinates (x) extracted in step 2 img ,y img ) and the calibration matrix M obtained in step 3), the physical coordinates of the robot are converted according to the following formula: Step 7), verification mechanism; Use the calibration conversion results obtained by the two 2D cameras to verify each other, and set the physical coordinates obtained by the first 2D camera to The physical coordinates obtained by the second 2D camera are The position deviation between the two cameras is calculated according to the following formula: Set the predefined tolerance threshold to T. If VX error >T or VY error >T, it means that there is a problem with the direction and posture of the material to be grasped, and an alarm is triggered; otherwise, it is considered that the verification has passed; Step 8), grasping point calculation; The grab point is calculated using the following formula: The security threshold is set to D in advance. If VX=|X grad -X base |>D or VY=|Y grad -Y base |>D, the deviation of the material to be grasped from the reference point material is too large, and the grasping point needs to be recalculated from step 5); otherwise, the robot arm performs the grasping action.

2. The dual-camera verification method for material positioning and grabbing according to claim 1 is characterized in that: In the step 3) of hand-eye calibration, the robot arm is moved and rotated to allow the 2D camera to take photos of the material from different positions and angles; According to the coordinates of the feature points extracted in step 2) and the coordinates of the manipulator, the hand-eye calibration matrices of the two 2D cameras are calculated respectively to determine the conversion relationship between the camera coordinate system and the manipulator coordinate system; The transformation matrix M obtained by calibration is expressed as: Among them, R is the rotation matrix and t is the translation vector; The matrix is ​​used to convert the coordinates (X cam , Y cam ) is converted to the coordinates in the robot coordinate system (X hand , Y hand ), the coordinate transformation formula is as follows:

3. The dual-camera verification method for material positioning and grabbing according to claim 1 is characterized in that: During the reference point teaching process in step 4), place the material on the workbench, adjust the position of the robot arm so that the gripper is aligned with the material, and record the physical coordinates (X base , Y base ) as the teaching grasping point; Move the robot arm to the photo taking position (X basecam , Y basecam ) to take pictures of the material as teaching photo points.

4. The dual-camera verification method for material positioning and grabbing according to claim 1 is characterized in that: In the step 2), the feature points are edge points, corner points or intersection points on the surface of the material, and the feature point coordinates are extracted from the image captured by the 2D camera using Harris corner point detection, SIFT or SURF image processing algorithm in the OPENCV library.

5. The dual-camera verification method for material positioning and grabbing according to claim 2 is characterized in that: The process of calculating the hand-eye calibration matrix in step 3) includes: first, translating the robotic arm to take nine images, determining the conversion relationship between the camera coordinate system and the robotic arm coordinate system through the feature point coordinates in these images and the robotic arm coordinates, and calculating the hand-eye calibration matrix using the least squares optimization algorithm. Then, the position data of three points are obtained by operating the robot arm at three different angles, and a circle method is determined based on the three points to calculate the rotation center C = (CenterX, CenterY), completing the rotation calibration; Finally, normalize the origin and take the rotation center as the origin. The hand-eye calibration matrix is:

Images