Automatic mechanical arm monocular vision servo system and method

Abstract

The invention provides an automatic mechanical arm monocular vision servo system and method, and the system comprises a hand-eye calibration module, a target flange recognition and positioning module and a marker recognition and positioning module. Wherein the hand-eye calibration module is used for calibrating a conversion matrix between a marker and a coordinate system at the tail end of a mechanical arm, the target flange recognition and positioning module comprises flange recognition and flange pose estimation, and the flange recognition is used for recognizing a target flange and distinguishing a left flange and a right flange according to priori knowledge; the flange pose estimation is used for estimating a transformation matrix between a flange and a camera coordinate system; and the marker identifying and positioning module is used for acquiring the pose of a marker on the mechanical arm in real time so as to obtain the pose of the tail end of the mechanical arm, and the pose of the flange is converted into a coordinate system of the tail end of the mechanical arm through coordinate transformation so as to realize a guiding function.

Description

Technical Field

[0001] The present invention relates to the field of natural gas transportation at an LNG receiving station, and in particular to an automated mechanical arm monocular visual servo system and method. Background Art

[0002] With the continuous construction of LNG receiving stations and the steady improvement of receiving capacity, the market demand for onshore LNG transportation is becoming increasingly urgent. However, the current LNG receiving stations still have deficiencies in automation and intelligence. Loading and unloading operations mainly rely on manual labor, resulting in high labor intensity and certain safety hazards. Therefore, the three major oil companies, pipeline companies and other owners have an increasingly urgent need for automatic control and intelligent management of loading equipment. The market urgently needs an automated loading system that is safe, reliable, engineering-applicable, and can achieve automatic control and intelligent management. Summary of the invention

[0003] The purpose of the present invention is to provide an automated robotic arm monocular visual servo system and method, which can accurately obtain the precise position and posture of the manifold flange of the transport vehicle, thereby realizing automatic docking of the robotic arm and the manifold flange.

[0004] The technical solution to achieve the purpose of the present invention is:

[0005] An automated robotic arm monocular visual servo system, comprising:

[0006] The marker-robotic arm calibration module is used to obtain the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robotic arm;

[0007] Flange identification and positioning module, used to identify the flange and calculate the transformation matrix between the flange and the camera coordinate system;

[0008] The marker recognition and positioning module sets a fixed Aruco tag on the robot arm, uses the Aruco tag decoding method to obtain the transformation matrix between the Aruco tag and the camera coordinate system in real time, and obtains the transformation matrix between the end of the robot arm and the camera coordinate system through coordinate transformation;

[0009] The flange-robotic arm calibration module obtains the transformation matrix between the robotic arm end and the target flange coordinate system based on the transformation matrices of the marker-robotic arm calibration module, the flange recognition and positioning module, and the marker recognition and positioning module.

[0010] Furthermore, obtaining the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm specifically includes:

[0011] Step 1-1, fix the camera and the target flange, set an Aruco tag concentric with the flange on the flange, and obtain the transformation matrix T between the Aruco tag and the camera coordinate system by decoding the Aruco tagAC , that is, the transformation matrix between the flange and camera coordinate systems;

[0012] Step 1-2, place the end of the robot arm close to the flange surface so that the coordinate system of the end of the robot arm coincides with the coordinate system of the flange;

[0013] Step 1-3, by identifying and decoding the aruco tag on the robot arm, the transformation matrix T between the camera and the marker coordinate system on the arm is obtained BC ;

[0014] Step 1-4, calculate the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm based on the two transformation matrices obtained in step 1-1 and step 1-3.

[0015] Furthermore, the transformation matrix between the coordinate system of the arm marker and the coordinate system of the end of the robot arm is

[0016] Furthermore, the flange identification and positioning module includes a rough identification module, a precise identification module and a calculation module, wherein:

[0017] The rough identification and positioning module takes images by a camera, pre-processes the collected images, uses a YOLO neural network model to identify the flange bolt holes, obtains the rough center point coordinates of the flange bolt holes, and uses the PNP algorithm to perform preliminary pose estimation on the flange according to the center point coordinates;

[0018] The precise identification and positioning module uses a camera to take an image, uses a YOLO neural network model to identify the flange, crops the flange image, performs image preprocessing on the cropped region of interest, uses the Canny algorithm to perform edge detection to obtain the flange edge, uses the ellipse detection algorithm AAMED with an edge connection method to perform ellipse fitting on the flange edge, and calculates the flange posture according to the fitted ellipse equation;

[0019] The calculation module determines the real flange posture, that is, the transformation matrix between the flange and the camera coordinate system, based on the posture estimation of the rough identification and positioning module and the precise identification and positioning module.

[0020] Furthermore, the actual flange posture is the flange posture corresponding to the smaller normal vector angle among the postures estimated by the rough identification and positioning module and the precise identification and positioning module.

[0021] Furthermore, the transformation matrix between the robot end and the target flange coordinate system is:

[0022]

[0023] Among them, T bis the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm, T is the transformation matrix between the flange and the camera coordinate system, T a is the transformation matrix between the robot end and the camera coordinate system.

[0024] An automated robotic arm monocular visual servo method, comprising:

[0025] Obtain the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robotic arm through the marker-robotic arm calibration module;

[0026] The flange is identified through the flange identification and positioning module, and the transformation matrix between the flange and the camera coordinate system is calculated;

[0027] The transformation matrix between the end of the robot arm and the camera coordinate system is obtained through the landmark recognition and positioning module;

[0028] The transformation matrix between the robot end and the target flange coordinate system is obtained through the flange-robot calibration module.

[0029] A computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the visual calibration method.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention calibrates the transformation matrix between the marker and the robot arm end coordinate system through the hand-eye calibration module, the target flange recognition and positioning module performs flange recognition and posture estimation, and distinguishes the left and right flanges according to prior knowledge, and estimates the transformation matrix between the flange and the camera coordinate system; the marker recognition and positioning module obtains the posture of the marker on the robot arm in real time to obtain the posture of the robot arm end, and the flange posture is converted to the robot arm end coordinate system through coordinate transformation, and the precise posture of the transport vehicle manifold flange is accurately obtained, so as to realize automatic docking of the robot arm and the manifold flange; the present invention realizes automatic docking and improves safety at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a system schematic diagram of the present invention.

[0032] Figure 2 This is a schematic diagram of the landmark-robotic arm calibration.

[0033] Figure 3 This is an algorithm flow chart of a monocular visual servo system for an automated robotic arm.

[0034] Figure 4 A flowchart for guiding the monocular visual servo system of an automated robotic arm. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further explained and illustrated in detail below based on the accompanying drawings and specific implementation methods.

[0036] Combination Figure 1 , an automated robotic arm monocular visual servo system, the system comprising;

[0037] The marker-robot arm calibration module is used to calibrate the transformation matrix between the end of the robot arm and the marker on the robot arm, so as to convert the flange posture into the robot arm coordinate system and guide the robot arm to dock;

[0038] Flange identification and positioning module, used to identify the flange and calculate the transformation matrix between the flange and the camera coordinate system;

[0039] The marker recognition and positioning module sets a fixed Aruco tag on the robotic arm, uses the Aruco tag decoding method to obtain the transformation matrix between the Aruco tag and the camera coordinate system in real time, and obtains the transformation matrix between the robotic arm end and the camera coordinate system through coordinate transformation.

[0040] The marker-robot arm calibration module of the present invention aims to calibrate the conversion matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm, combined with Figure 2 , the process includes the following steps;

[0041] Step 1: Fix the camera and the target flange, set an Aruco tag on the flange that is concentric with the flange, and obtain the transformation matrix T between the Aruco tag and the camera coordinate system by decoding the Aruco tag. AC That is, the transformation matrix between the flange and camera coordinate systems;

[0042] Step 2: Place the end of the robot arm close to the flange surface so that the coordinate system of the end of the robot arm coincides with the coordinate system of the flange;

[0043] Step 3: By identifying and decoding the Aruco tag on the robotic arm, the transformation matrix T between the camera and the marker coordinate system on the arm is obtained. BC ;

[0044] Step 4: Calculate the transformation matrix T between the arm marker coordinate system and the robot end coordinate system based on the two transformation matrices obtained in step 1 and step 2. b ;

[0045]

[0046] The flange recognition and positioning module of the present invention aims to obtain the position and posture of the flange in the camera coordinate system by processing the flange image, so that when the subsequent guiding robot arm docking is carried out, the flange position and posture can be converted to the robot arm coordinate system in real time, thereby realizing guidance. The flange recognition and positioning module includes a rough recognition module, a precise recognition module and a calculation module, wherein the rough recognition and positioning module takes an image by a camera, pre-processes the collected image, uses a YOLO neural network model to recognize the flange bolt hole, obtains the rough center point coordinates of the flange bolt hole, and uses a PNP algorithm to perform a preliminary position and posture estimation on the flange according to the center point coordinates; the precise recognition and positioning module takes an image by a camera, uses a YOLO neural network model to recognize the flange, crops the flange image, and performs image pre-processing on the cropped area of ​​interest, uses a canny algorithm to perform edge detection to obtain the flange edge, uses an ellipse detection algorithm AAMED of an edge connection method to perform ellipse fitting on the flange edge, and calculates the flange position and posture according to the fitted ellipse equation; the calculation module determines the true flange position and posture based on the position and posture estimation of the rough recognition and positioning module and the precise recognition and positioning module, that is, the transformation matrix between the flange and the camera coordinate system.

[0047] Combined with 3, the specific implementation process of the flange identification and positioning module includes the following steps:

[0048] Step 1: Identify the target flange. When locating the target flange, the flange area can be identified using the deep learning target detection algorithm of the YOLO neural network in machine vision.

[0049] Step 2: Distinguish the target flange. Based on prior knowledge, we can compare the X coordinates of the corner points of the YOLO frame in the image coordinate system to distinguish the left and right flanges.

[0050] Step 3: Select the region of interest, crop the target flange image, and use the YOLO neural network algorithm to identify the target flange bolt holes to obtain the center coordinates of the flange bolt hole YOLO box;

[0051] Step 4: Calculate the rough position T0 of the flange using the PNP algorithm according to the center coordinates of the flange bolt hole YOLO frame;

[0052] Step 5, preprocessing the flange image, using the Canny algorithm to perform edge detection on the flange image to obtain the flange edge;

[0053] Step 6: The ellipse detection algorithm AAMED of the edge connection method performs ellipse fitting on the flange edge;

[0054] Step 7: Calculate the spatial circular pose, i.e., the flange pose, according to the general formula of the ellipse obtained in step 6. Since the ellipse has ambiguity, two sets of accurate flange poses will be obtained.

[0055] Step 8: Eliminate the ambiguity of the ellipse and calculate T0 and The normal vector angle of , the flange posture corresponding to the smaller angle is taken as the true flange posture T;

[0056] The purpose of the marker recognition module of the present invention is to be able to recognize and locate the marker through the camera, obtain the position and posture of the end of the robot arm in real time, thereby obtaining the transformation matrix between the flange and the coordinate system of the end of the robot arm, and realize the guidance function. The specific implementation steps are as follows;

[0057] Step 1: Set a fixed Aruco tag on the robot arm, and use the marker-robot calibration module to calibrate the transformation matrix T between the marker coordinate system on the arm and the robot end coordinate system b ;

[0058] Step 2: The camera takes a picture, identifies and decodes the Aruco tag marker, and obtains the position T of the Aruco tag in the camera coordinate system in real time. a ;

[0059] Step 3: Obtain the position T of the end of the robot arm in the camera coordinate system through coordinate transformation ab ;

[0060]

[0061] Step 4: Obtain the position T of the flange in the robot end coordinate system through coordinate changes c ;

[0062]

[0063] The present invention also includes a flange-mechanical arm calibration module, which converts the target flange posture into the mechanical arm coordinate system through coordinate transformation, and feeds back to the control system through the lower computer to realize the docking function.

[0064] Combination Figure 4 , an automated robotic arm monocular visual servo method, comprising:

[0065] The camera takes a picture of the flange, the flange is identified through the flange recognition and positioning module, and the transformation matrix between the flange and the camera coordinate system is calculated;

[0066] The camera takes a picture of the marker, and obtains the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robotic arm through the marker-robotic arm calibration module; the transformation matrix between the end of the robotic arm and the camera coordinate system is obtained through the marker recognition and positioning module;

[0067] The flange-robot calibration module is used to obtain the transformation matrix between the robot end and the target flange coordinate system, and the position and posture of the flange at the end of the loading robot arm are obtained to complete the guidance.

[0068] A computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the visual calibration method.

[0069] The above are only preferred embodiments of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the invention should be considered to be included in the scope of protection of the invention.

Claims

1. An automated robotic arm monocular visual servo system, characterized in that: include; The marker-robotic arm calibration module is used to obtain the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robotic arm; Flange identification and positioning module, used to identify the flange and calculate the transformation matrix between the flange and the camera coordinate system; The marker recognition and positioning module sets a fixed Aruco tag on the robot arm, uses the Aruco tag decoding method to obtain the transformation matrix between the Aruco tag and the camera coordinate system in real time, and obtains the transformation matrix between the end of the robot arm and the camera coordinate system through coordinate transformation; The flange-robotic arm calibration module obtains the transformation matrix between the robotic arm end and the target flange coordinate system based on the transformation matrices of the marker-robotic arm calibration module, the flange recognition and positioning module, and the marker recognition and positioning module.

2. The automated robotic arm monocular visual servo system according to claim 1, characterized in that: The transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm is obtained specifically including: Step 1-1, fix the camera and the target flange, set an Aruco tag concentric with the flange on the flange, and obtain the transformation matrix T between the Aruco tag and the camera coordinate system by decoding the Aruco tag AC , that is, the transformation matrix between the flange and camera coordinate systems; Step 1-2, place the end of the robot arm close to the flange surface so that the coordinate system of the end of the robot arm coincides with the coordinate system of the flange; Step 1-3, by identifying and decoding the aruco tag on the robot arm, the transformation matrix T between the camera and the marker coordinate system on the arm is obtained BC ; Step 1-4, calculate the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm based on the two transformation matrices obtained in step 1-1 and step 1-3.

3. The automated robotic arm monocular visual servo system according to claim 2, characterized in that: Transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm 4. The automated robotic arm monocular visual servo system according to claim 1, characterized in that: The flange identification and positioning module includes a rough identification module, a precise identification module and a calculation module, wherein: The rough identification and positioning module takes images by a camera, pre-processes the collected images, uses a YOLO neural network model to identify the flange bolt holes, obtains the rough center point coordinates of the flange bolt holes, and uses the PNP algorithm to perform preliminary pose estimation on the flange according to the center point coordinates; The precise identification and positioning module uses a camera to take an image, uses a YOLO neural network model to identify the flange, crops the flange image, performs image preprocessing on the cropped region of interest, uses the Canny algorithm to perform edge detection to obtain the flange edge, uses the ellipse detection algorithm AAMED with an edge connection method to perform ellipse fitting on the flange edge, and calculates the flange posture according to the fitted ellipse equation; The calculation module determines the real flange posture, that is, the transformation matrix between the flange and the camera coordinate system, based on the posture estimation of the rough identification and positioning module and the precise identification and positioning module.

5. The automated robotic arm monocular visual servo system according to claim 4, characterized in that: The actual flange pose is the flange pose corresponding to the smaller normal vector angle among the poses estimated by the coarse identification and positioning module and the precise identification and positioning module.

6. The automated robotic arm monocular visual servo system according to claim 1, characterized in that: The transformation matrix between the end of the manipulator and the target flange coordinate system is: Among them, T b is the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robot arm, T is the transformation matrix between the flange and the camera coordinate system, T a is the transformation matrix between the robot end and the camera coordinate system.

7. An automated robotic arm monocular visual servo method based on the system of any one of claims 1 to 6, characterized in that: include: Obtain the transformation matrix between the coordinate system of the marker on the arm and the coordinate system of the end of the robotic arm through the marker-robotic arm calibration module; The flange is identified through the flange identification and positioning module, and the transformation matrix between the flange and the camera coordinate system is calculated; The transformation matrix between the end of the robot arm and the camera coordinate system is obtained through the landmark recognition and positioning module; The transformation matrix between the robot end and the target flange coordinate system is obtained through the flange-robot calibration module.

8. A computer storage medium, characterized in that The computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the visual calibration method described in claim 7.

Images