Visual-based automatic drill changing method for robot arm

Abstract

The application discloses a visual-based automatic drill changing method for a mechanical arm, acquires a drill rod point cloud from a 3D camera, segments the drill rod point cloud by using a deep learning model, extracts the drill rod point cloud, and then calculates a drill rod pose. Point cloud data containing a holder and a power head is collected by the 3D camera, feature point clouds of the power head and the holder are extracted by segmentation through the deep learning model, and a drill rod installation pose is calculated according to the center point positions of the power head and the holder. The drill rod pose and the drill rod installation pose are converted from a depth camera coordinate system to a mechanical arm base coordinate system through coordinate transformation, a proportional electro-hydraulic control mechanical arm is used to move along a planned trajectory, and a drill taking and drilling action is completed. The application can realize automatic drill searching and automatic drill changing, and is helpful to realize unmanned drilling and automatic drilling of a drilling machine.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, in particular to a vision-based automatic drill changing method for a mechanical arm. BACKGROUND

[0002] The basic structure of a coal mine drilling robot includes a main machine, a power head, a gripper, a drill, an energy device, an electrical system, a remote monitoring device and other auxiliary devices. The connection and disconnection of the drill pipe during the drilling process of the drilling robot is one of the most time-consuming and tedious procedures in the drilling process. Coal mine intelligence is the latest demand for safe mining of coal mines, and the drilling robot is a direct embodiment of this demand, including a drilling machine and a mechanical arm, wherein the drilling machine includes a power head and a gripper, and the drill pipe is placed between the power head and the gripper. The coal mine intelligence requires the drilling robot to have functions such as automatic drill pipe loading and unloading and automatic drilling, but the current mine robot intelligence level is low, the drill pipe replacement precision is not high, and manual assistance is required to complete the replacement, therefore, improving the intelligence level is the direction of the next research. SUMMARY

[0003] The present application aims to provide a vision-based automatic drill changing method for a mechanical arm, which can realize the positioning and orientation of the drill pipe and the automatic replacement of the drill pipe.

[0004] To achieve the above-mentioned purpose, the present application provides a vision-based automatic drill changing method for a mechanical arm, which comprises

[0005] S1, determining the drill pipe pose: obtaining the drill pipe point cloud from the 3D camera, segmenting the drill pipe point cloud through the deep learning model, extracting the drill pipe point cloud; calculating the drill pipe pose;

[0006] S2, determining the drill pipe installation pose: collecting the point cloud data containing the gripper and the power head through the 3D camera, extracting the point cloud of the power head and the gripper through the deep learning model; obtaining the center point positions of the gripper and the power head by using the centroids of the gripper and the power head, calculating the drill pipe installation pose by the center point positions of the power head and the gripper;

[0007] S3, automatic drill changing: determining the drill taking and drilling poses by the drill pipe pose and the drill pipe installation pose; converting the drill pipe pose and the drill pipe installation pose from the depth camera coordinate system to the mechanical arm base coordinate system through coordinate transformation; determining the initial pose of the mechanical arm; moving the mechanical arm along the planned trajectory by using the electro-hydraulic proportional control to complete the drill taking and drilling actions.

[0008] Further, in step S1, the drill pipe pose calculation method is:

[0009] The drill pipe is segmented into the drill pipe male head point cloud and the drill pipe body point cloud using a deep learning model; the centroid of the drill pipe body point cloud is used as the center point of the drill pipe body, and the centroid of the drill pipe male head point cloud is used as the center point of the drill pipe male head; the drill pipe pose is calculated using the center points of the drill pipe body and the drill pipe male head.

[0010] Furthermore, step S3 also includes: using a 3D camera to detect obstacles, calculating the obstacle-free motion space, and using an obstacle avoidance trajectory planning algorithm to plan the motion trajectory of the robotic arm's end effector pose.

[0011] Further, in step S3, multiple 3D cameras are mounted around the robotic arm, and the target pose is converted from the depth camera to the base coordinate system. Specifically, the steps are: obtaining the extrinsic parameter matrix of the depth camera using a calibration plate; measuring the pose matrix of the calibration plate relative to the robotic arm's base coordinate system using a measuring tool; and decomposing the depth camera extrinsic parameter matrix into R... ir T ir The pose matrix of the calibration plate relative to the robot arm's base coordinate system is decomposed into R... w and T w ;P ir P represents the position of a point in the depth camera coordinate system. w Let the coordinates of this point be in the base coordinate system.

[0012] The pose transformation matrix is:

[0013]

[0014] The coordinate transformation formula is:

[0015] P w =R·P ir +T.

[0016] Furthermore, the robotic arm is a six-degree-of-freedom robotic arm. The forward kinematics equations of the robotic arm are obtained through the DH algorithm. After obtaining multiple sets of end-effector poses through the corresponding forward kinematics equations, the inverse kinematics equations are obtained through a deep learning algorithm. The specific steps for solving the inverse kinematics are as follows: determine the structure of the multi-degree-of-freedom robotic arm; establish the forward kinematics equations through the DH method, and obtain the correspondence between joint angles and end-effector poses through the forward kinematics equations; train the structure using deep learning to obtain the inverse kinematics equations of the robotic arm.

[0017] The beneficial effects of the present application are: compared with the prior art, the present application adopts a 3D camera combined with a deep learning technology to identify a drill rod, a power head, a gripper and a drilling machine, point cloud reconstruction is performed by using 3D point cloud completion, the position of the drill rod main body and the position of the male head are identified to realize drill rod positioning and pose, and coordinate transformation is used to convert to the mechanical arm base coordinate system, so that self-searching of the drill rod can be realized; the center of the power head and the gripper is positioned, and the midpoint of the two points is taken as the drill rod installation position, and the direction from the center of the power head to the center of the gripper is taken as the drill rod orientation, so that the determination of the drill rod installation pose is realized, thereby realizing the automation and unmanned of the taking and installing of the drill rod; the 3D point cloud can also be used to detect obstacles, establish a non-collision motion space, plan a drill rod motion trajectory, use electro-hydraulic proportional control to realize motion trajectory control, and multiple 3D cameras can realize multi-directional searching of the drill rod. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A method flow chart of the automatic drill changing of the mechanical arm of the present application;

[0019] Figure 2 A schematic diagram of the automatic drill changing work process of the mechanical arm;

[0020] Figure 3 A position diagram of the drill rod main body and the drill rod male head;

[0021] Figure 4 A training method flow chart of the deep learning;

[0022] Figure 5 A principle diagram of obtaining the drill rod pose;

[0023] Figure 6 A method flow chart of obtaining the drill rod installation pose;

[0024] Figure 7 A flow chart of obtaining the installation pose of the power head and the gripper;

[0025] Figure 8 A coordinate conversion schematic diagram of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0027] Figure 1 A method flow chart of the automatic drill changing of the mechanical arm based on vision of the present application. The method mainly includes three parts, the first part is to obtain complete point clouds of the drill rod, the power head and the gripper; the second part is to obtain the pose of the drill rod and the poses of the power head and the gripper; and the third part is to convert the drill rod, the power head and the gripper from the depth camera coordinates to the base coordinates of the mechanical arm, and then automatically complete the drilling and drill changing operations through the mechanical arm.

[0028] AsFigure 2 As shown, during drilling, the drill pipe is installed between the power head and the chuck. The robotic arm needs to remove the drill pipe from its original position, such as inside the drill pipe box, and then place it between the power head and the chuck. During drill unloading, the robotic arm removes the drill pipe from between the power head and the chuck and then puts it back into its original position, such as inside the drill pipe box.

[0029] like Figure 3 As shown, the drill pipe is divided into the drill pipe body and the drill pipe male end, providing directional data for subsequent determination of the drill pipe's position. The drill pipe is placed in a specific order: when the drill pipe is placed on the drilling rig, its male end is connected to the chuck, and the rear end of the drill pipe body is connected to the power head.

[0030] Figure 4 This is a flowchart of the training method for a deep learning model. The specific method is as follows: First, obtain the complete point cloud of the drill pipe:

[0031] The process involves acquiring point clouds of the drill pipe, segmenting them using a first deep learning model, extracting the drill pipe point cloud from the point cloud acquired by a 3D camera, estimating and reconstructing the entire drill pipe point cloud using a point cloud completion algorithm, segmenting the drill pipe into the drill pipe male end and the drill pipe body using a second deep learning model, and filtering to remove discrete points. Next, point clouds of the power head and gripper are acquired: point cloud data containing the gripper and power head are collected using a 3D camera; filtering removes discrete points; the deep learning model segments and extracts the point clouds of the power head and gripper; and the 3D point cloud completion algorithm reconstructs the complete point clouds of the gripper and power head.

[0032] like Figure 5 As shown, the method for obtaining the drill pipe pose is as follows: using the centroid of the drill pipe body point cloud as the center point of the drill pipe body, and using the centroid of the drill pipe male end point cloud as the center point of the drill pipe male end; the drill pipe pose is calculated using the center points of the drill pipe body and the drill pipe male end. Figure 6 As shown, due to the characteristics of the camera, the point cloud data has denser points closer to the camera and sparser points farther away, and only half a drill rod exists. Therefore, simply calculating the average coordinates when calculating the center coordinates of the drill rod will cause errors. In the point cloud centroid calculation, the 3D point cloud data is first projected into a 2D image on the xoy plane, and the minimum bounding rectangle of the projected point cloud is calculated. The centroid of the drill rod is the centroid (x, y) of the minimum bounding rectangle. To eliminate the influence of individual outliers on the final positioning, we select points (x, y) on the drill rod within the range (xd: x+d, yd: y+d). i y i ), i = 1, 2, ..., n, where n is the number of point clusters. Finally, find the centroid (x) of the projected point cloud. i y i The corresponding z-axis coordinate z i For the coordinates (x) of the point cluster i yi The average value of zi) is taken to obtain the center coordinates D'(x') of the drill pipe surface. d ,y' d ,z' d The formula is as follows:

[0033]

[0034] Similarly, the coordinates P'(x') of the center of the male head can be obtained. p ,y' p ,z' p However, since this coordinate is the center coordinate of the drill pipe's cylindrical surface, in order to obtain the center coordinate of the drill pipe body as D(x)... d ,y d ,z d The following formula is used for calculation:

[0035]

[0036]

[0037]

[0038] In the formula: O is the origin of the coordinate system, D' and P' are the known center points of the drill pipe and male end face, r is the radius of the drill pipe cylinder, and θ is... and The included angle, because the taper angle of the pipe thread is very small, Approximately with the drill pipe axis Parallel. Combining formulas (3) and (4), calculate D(x). d ,y d ,z d )coordinate.

[0039] Similarly, the coordinates of the male head center P(x) can be calculated. P ,y P ,z P ), drill pipe posture express.

[0040] The coordinates of the center points of the power head and the holder can be obtained by calculating only the center point of the surface (Formula 1), thus obtaining the drill pipe mounting pose. The acquisition process is as follows: Figure 7 As shown.

[0041] After determining the positions of the drill pipe, power head, and gripper, the robotic arm's drill-picking and drilling-up positions need to be determined based on the drill pipe's position and installation position. Therefore, coordinate transformation is required to convert the drill pipe's position and installation position from the depth camera coordinate system to the robotic arm's base coordinate system. Then, the initial position of the robotic arm is determined, and electro-hydraulic proportional control is used to move the robotic arm along the planned trajectory to complete the drill-picking and drilling-up actions.Figure 3 As shown, a plurality of 3D cameras are installed around the mechanical arm, the target pose is converted from the depth camera to the base coordinate system, the extrinsic matrix of the depth camera is obtained by using the calibration board, the pose matrix of the calibration board relative to the base coordinate system of the mechanical arm is measured by the measuring tool, the extrinsic matrix of the depth camera is decomposed into R ir , T ir , the pose matrix of the calibration board relative to the base coordinate system of the mechanical arm is decomposed into R w and T w ; P ir is the position of the space point in the depth camera coordinate system, P w is the coordinate of the point in the base coordinate system

[0042] The pose transformation matrix is

[0043]

[0044] The coordinate transformation formula is

[0045] P w =R·P ir +T.

[0046] Figure 8 As shown in the coordinate conversion schematic diagram of the present application, to complete the conversion of the camera coordinate system to the base coordinate system, the pose matrix of the calibration board in the depth camera coordinate system and the pose matrix of the calibration board in the base coordinate system of the mechanical arm need to be obtained respectively. The infrared camera in the depth camera is used to shoot the calibration board to obtain the depth image of the calibration board, the extrinsic matrix of the calibration board in the infrared camera coordinate system is obtained by using the matlab Camera Calibrator module, Rir and Tir are obtained. The pose matrix of the calibration board in the base coordinate system of the mechanical arm is obtained by using the measuring tool, Rw and Tw are obtained, and the pose transformation matrix R and T are obtained.

[0047] The mechanical arm in the embodiment is a six-degree-of-freedom mechanical arm, the forward kinematics equation of the mechanical arm is obtained by D-H algorithm, after a plurality of sets of end poses are obtained by corresponding forward kinematics equations, the inverse kinematics equation is obtained by deep learning algorithm, and the specific steps of inverse kinematics solving are as follows: determining the structure of the multi-degree-of-freedom mechanical arm; establishing the forward kinematics equation by D-H method, and obtaining the corresponding relationship between joint angle and end pose by the forward kinematics equation; training the structure by deep learning to obtain the inverse kinematics equation of the mechanical arm.

[0048] The method can also use 3D cameras to detect obstacles, calculate the obstacle-free motion space, and plan the motion trajectory of the end pose of the mechanical arm by using the obstacle avoidance trajectory planning algorithm.

[0049] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to this, and various changes that can be made within the knowledge of those skilled in the art without departing from the spirit of the present application are within the scope of the claims of the present application.

Claims

1. A vision-based automatic drill changing method for robotic arms, characterized in that, include S1. Determine the drill pipe pose: Obtain the drill pipe point cloud from the 3D camera, segment the drill pipe point cloud using a deep learning model, extract the drill pipe point cloud, and calculate the drill pipe pose; The method for extracting the drill pipe point cloud is as follows: The drill pipe point cloud is acquired, and segmented using a first deep learning model. Drill pipe point clouds are extracted from the point cloud acquired by a 3D camera. A point cloud completion algorithm is used to estimate and reconstruct the entire drill pipe point cloud. A second deep learning model is used to segment the drill pipe into the male end and the main body. Filtering is applied to remove discrete points. Next, the point clouds of the power head and gripper are acquired: point cloud data containing the gripper and power head is collected using a 3D camera; filtering is applied to remove discrete points. A deep learning model is used to segment and extract point clouds of the power head and gripper; a 3D point cloud completion algorithm is used to reconstruct the complete point clouds of the gripper and power head. The method for calculating the drill pipe position is as follows: The drill pipe is segmented into the drill pipe male head point cloud and the drill pipe body point cloud using a deep learning model; the center point of the drill pipe body is obtained using the drill pipe body point cloud, and the center point of the drill pipe male head is obtained using the drill pipe male head point cloud; the drill pipe pose is calculated using the center point of the drill pipe body and the center point of the drill pipe male head. The method for calculating the centroid of a point cloud is as follows: Project the 3D point cloud data onto the xoy plane to form a 2D image, and select points (x, y, y) on the drill rod within the range (xd: x+d, yd: y+d). i y i ), i=1, 2, ..., n, where n is the number of point clusters, find the centroid of the projected point cloud (x i y i The corresponding z-axis coordinate z i For the coordinates (x) of the point cluster i y i , z i Take the average value to obtain the center coordinates of the drill pipe surface. ,in: ， ， （1） The surface center coordinates of the male head are obtained using the above method. ; Calculate the center coordinates of the drill pipe body using the following formula. : (2) （3） （4） In the formula: O is the origin of the coordinate system, , Given the center points of the drill pipe and the male end face, where r is the radius of the drill pipe cylinder, for and The included angle, because the taper angle of the pipe thread is very small, Approximately with the drill pipe axis parallel; The surface center coordinates of the male head are obtained using the above method. Drill pipe posture express; S2. Determine the drill pipe installation pose: Collect point cloud data containing the gripper and power head using a 3D camera, and extract the point clouds of the power head and gripper using a deep learning model; obtain the center point positions of the gripper and power head using the centroids of the gripper and power head, and calculate the drill pipe installation pose using the center point positions of the power head and gripper. S3. Automatic Drill Change: Determine the drill pick-up and drill loading positions based on the drill rod position and drill rod installation position; transform the drill rod position and drill rod installation position from the depth camera coordinate system to the robot arm base coordinate system through coordinate transformation; determine the initial position of the robot arm; and use electro-hydraulic proportional control to move the robot arm along the planned trajectory to complete the drill pick-up and drill loading actions.

2. The vision-based automatic drill changing method for a robotic arm according to claim 1, characterized in that, Step S3 also includes: using a 3D camera to detect obstacles, calculating the obstacle-free motion space, and using an obstacle avoidance trajectory planning algorithm to plan the motion trajectory of the robotic arm's end effector.

3. The vision-based automatic drill changing method for a robotic arm according to claim 2, characterized in that, In step S3, multiple 3D cameras are mounted around the robotic arm, and the target pose is converted from the depth camera to the base coordinate system. Specifically, the steps are: obtaining the extrinsic parameter matrix of the depth camera using a calibration plate; measuring the pose matrix of the calibration plate relative to the robotic arm's base coordinate system using a measuring tool; and decomposing the depth camera extrinsic parameter matrix into... , The pose matrix of the calibration plate relative to the robot arm's base coordinate system is decomposed into... and ; This represents the position of a point in space within the depth camera's coordinate system. Let the coordinates of this point be in the base coordinate system. The pose transformation matrix is: The coordinate transformation formula is: 。 4. The vision-based automatic drill changing method for a robotic arm according to claim 3, characterized in that, The robotic arm is a six-degree-of-freedom robotic arm. The forward kinematics equations of the robotic arm are obtained through the DH algorithm. After obtaining multiple sets of end-effector poses through the corresponding forward kinematics equations, the inverse kinematics equations are obtained through a deep learning algorithm. The specific steps for solving the inverse kinematics are as follows: determine the structure of the multi-degree-of-freedom robotic arm; establish the forward kinematics equations through the DH method; obtain the correspondence between joint angles and end-effector poses through the forward kinematics equations; train the structure using deep learning to obtain the inverse kinematics equations of the robotic arm.

Images