Vision-based Automatic Loading and Unloading Drill Pipe Positioning Method for Coal Mine Tunnel Drilling Robots

Abstract

The present invention discloses a vision-based method for automatically loading and unloading drill rods for a tunnel drilling robot used in coal mines. This invention patent belongs to the field of robot vision. Based on robot vision information, a new method for positioning the position of drill rods loading and unloading based on inherent constraints is proposed. After the system model is established, this method uses the principle of the unchanged inherent position relationship between the cooperative target and the drill rod loading and unloading clamp, and through image processing technology and robot coordinate system conversion, minimizes the error between the expected relative posture and the actual relative posture of the camera and the cooperative target, so as to achieve the purpose of calculating the ideal position of the drill rod loading and unloading clamp. Moreover, during the actual drilling process, the drill rig's own posture changes due to factors such as rotation, lifting, and movement, and there is no need for repeated teaching, and it can still complete precise positioning autonomously.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and in particular to a vision-based automatic loading and unloading drill rod positioning method for a tunnel drilling robot used in a coal mine. Background Art

[0002] Currently, electro-hydraulic controlled automated drilling rigs have achieved remote control, electro-hydraulic control, and automatic loading and unloading of drill rods in underground coal mine gas extraction activities, and have been put into small-scale use in some coal mines. With the promotion and application of automated drilling rigs in coal mines, the importance of drill rod conveying devices has gradually become apparent. Drill rod manipulators, as a common drill rod conveying device, play a vital role in reducing on-site labor intensity and improving safety. However, due to the special operating conditions of underground coal mine drilling rigs and the harsh drilling environment, most industrial robots use offline programming or teaching methods to perform drill rod loading and unloading tasks. When the position of the drill rig changes, the robot cannot adapt to the new situation and will not work properly.

[0003] Automated drill pipe loading and unloading on drilling rigs can effectively improve loading and unloading efficiency, reduce manual labor intensity, and improve drilling safety. Current automated drill pipe loading and unloading systems primarily use three methods for positioning the loading and unloading locations: mechanical positioning, multi-sensor positioning, and visual positioning.

[0004] Mechanical positioning methods rely primarily on mechanical structures and proximity switches to achieve fixed-point access and placement of drill rods. These methods typically utilize a multi-stage transmission mechanism, requiring multiple transfers. These mechanisms are large and complex. While achieving their intended functions, they also suffer from reduced accuracy due to mechanical wear and accumulated errors. Furthermore, these methods lack automation and intelligence.

[0005] Multi-sensor positioning methods use distance sensors, inclination sensors, and other sensors to build models to calculate the drill pipe loading and unloading positions. However, existing positioning methods rely heavily on modeling accuracy, leading to significant accuracy fluctuations and low reliability during subsequent use.

[0006] Vision-based positioning methods offer fast response, high accuracy, and are non-contact. They are highly adaptable to the environment and can effectively address the loss of accuracy caused by accumulated system errors and model changes, while significantly improving the automation and intelligence of the system. While vision-assisted drill rod positioning methods currently exist, they still lack accuracy and environmental adaptability, hindering their practical application in coal mining. Summary of the Invention

[0007] This invention aims to address the technical issues that hinder the practical application of existing vision-assisted drill rod positioning methods in coal mine extraction processes due to their poor system accuracy and environmental adaptability. To address this technical issue, the present invention employs a vision-based approach to automated drill rod loading and unloading on a drilling rig. This approach utilizes a monocular 2D camera and a cooperative target, accurately guiding a robot to the drill rod loading and unloading gripper on the rig through imagery without the need for a high-precision model.

[0008] To achieve the above objectives, the present invention provides a vision-based method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for a coal mine, comprising the following steps:

[0009] S1: Calibrate the camera and robot hand-eye calibration to obtain the hand-eye matrix;

[0010] S2: Teach the camera's photo point and the drill rod placement point, collect an image containing the cooperative target as the desired image, set the relative posture between the camera's photo point and the cooperative target as a fixed constraint based on the robot's end-point posture, hand-eye matrix, and desired image information, and establish a vision-based positioning model for loading and unloading drill rods.

[0011] S3: The robot vision system controls the robot movement to satisfy the fixed constraints between the camera shooting point and the cooperative target;

[0012] S4: Calculate the drill rod placement position based on the robot's posture after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the gripper position to place or grab the drill rod.

[0013] Furthermore, step S1 specifically includes:

[0014] S11: Calibrate the two-dimensional camera. Based on multiple sets of images containing cooperative targets acquired by the camera, establish a mapping relationship between objects in the three-dimensional space and the two-dimensional camera plane, and calculate the camera's intrinsic parameter information.

[0015] S12: Fix the camera at the end of the robot, move it to different positions, and record images to calculate the transformation relationship between the robot end flange and the camera center, that is, the hand-eye matrix E T C .

[0016] Furthermore, step S2 specifically includes:

[0017] S21: Using the robot teaching pendant, move the robot carrying the drill rod to the drilling rig clamping slot and record it as the drill rod placement point;

[0018] S22: Move the robot to the top of the drilling rig and record it as the camera photo point;

[0019] S23: Acquire an image containing the cooperative target as a desired image;

[0020] S24: According to the robot end posture, hand-eye matrix, and expected image information, the relative posture between the camera shooting point and the cooperative target is set as a fixed constraint condition, and a vision-based positioning model for loading and unloading drill rods is established.

[0021] Furthermore, step S3 specifically includes:

[0022] S31: Based on the vision-based positioning model for loading and unloading drill rods, the camera intrinsic parameter information and the P3P algorithm are used to solve the transformation matrix of the cooperative target in the camera coordinate system in the images collected during the robot loading and unloading drill rods. The relative posture transformation matrix between the cooperative target and the initial camera in the initial setting expected image is: That is, the system fixed agreed conditions; according to the relative posture of the relative cooperative target obtained in the collected image and the set expected image, the transformation matrix between the current camera coordinate system and the camera coordinate system under the ideal fixed constraint is obtained according to the transformation relationship between the coordinate systems

[0023]

[0024] in, Represents the transformation matrix of the cooperative target in the current camera coordinate system, Represents the fixed constraint between the cooperative target and the desired camera shooting point, i.e., the camera coordinate system;

[0025] S32: According to the inter-system coordinate transformation framework of the robot base coordinate system, the manipulator end coordinate system, the camera coordinate system, the target cooperative target coordinate system and the gripper coordinate system in the drilling rig loading and unloading system, the pose matrix of the robot end camera coordinate system of the current posture is obtained.

[0026]

[0027] S33: Based on the transformation matrix between the current camera pose and the desired camera pose Get the complete transformation path of the transformation matrix for cooperative target recognition and positioning based on the current robot position:

[0028]

[0029] in It is the transformation matrix between the robot end coordinate system and the robot base coordinate system obtained by the robot teaching pendant. E T C is the hand-eye matrix obtained by the hand-eye calibration algorithm, is the transformation matrix between the cooperative target pose and the desired camera pose;

[0030] S34: The next posture of the robot end camera is calculated by the transformation matrix of cooperative target recognition and positioning, so that the relative posture between the current camera shooting position and the cooperative target image satisfies the fixed constraint between the initial camera shooting point and the initial cooperative target. The robot end posture transformation matrix of the next posture is:

[0031]

[0032] Furthermore, step S4 specifically includes:

[0033] S41: After the relative posture between the camera shooting position and the cooperative target image is completed to meet the precise positioning between the initial camera shooting point and the initial cooperative target, the transformation matrix is obtained according to the camera shooting point and drill rod placement point obtained by manual teaching. Get the description of the drill rod placement position in the robot's world coordinate system, that is, the drill rod placement point:

[0034]

[0035] in B T H Represents the transformation matrix of the drill rod placement position in the robot base coordinate system, represents the transformation matrix for cooperative target recognition and positioning based on the current position, represents the inverse transformation matrix of the hand-eye matrix, Represents the transformation matrix of the drill rod placement point in the robot end coordinate system.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a vision-based automatic loading and unloading drill rod positioning device for a tunnel drilling robot for a coal mine, comprising the following units:

[0037] The calibration unit is used to calibrate the camera and the robot's hand-eye calibration to obtain the hand-eye matrix;

[0038] The constraint unit is used to teach the camera's photo point and the drill rod placement point. It collects an image containing the cooperative target as the desired image. Based on the robot's end-point posture, the hand-eye matrix, and the desired image information, it sets the relative posture between the camera's photo point and the cooperative target as a fixed constraint condition, and establishes a vision-based positioning model for loading and unloading drill rods.

[0039] A control unit is used to control the movement of the robot through the robot vision system so as to satisfy the fixed constraint conditions between the camera shooting point and the cooperative target;

[0040] The calculation unit is used to calculate the drill rod placement position based on the posture of the robot after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the gripper position to place or grab the drill rod.

[0041] In addition, in order to achieve the above-mentioned purpose, the present invention also provides an electronic device, including a memory, a processor and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the steps of the method for automatically loading and unloading drill rod positioning of a tunnel drilling robot for coal mines are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for automatically loading and unloading drill rods and positioning the coal mine tunnel drilling robot are implemented.

[0043] The technical solution provided by the present invention has the following beneficial effects:

[0044] After establishing a positioning model for loading and unloading drill rods based on vision, the method of the present invention utilizes the principle that the inherent positional relationship between the cooperative target and the drill rod loading and unloading clamp remains unchanged, and minimizes the error between the expected relative posture and the actual relative posture of the camera and the cooperative target through image processing technology and robot coordinate system conversion, thereby achieving the purpose of calculating the ideal position of the drill rod loading and unloading clamp. Moreover, during the actual drilling process, the drill rig's own posture changes due to factors such as rotation, lifting, and movement, and no repeated teaching is required, and precise positioning can still be completed autonomously. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0046] Figure 1 This is a flow chart of the vision-based method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for coal mines according to the present invention;

[0047] Figure 2 This is a schematic diagram of the automatic loading and unloading of drill rods by the tunnel drilling robot for coal mines according to the present invention;

[0048] Figure 3 Schematic diagram of the conversion relationship between 3D target coordinates and 2D image coordinates of the present invention;

[0049] Figure 4 Schematic diagram of the coordinate system relationship of the drill rod loading and unloading system of the present invention;

[0050] Figure 5 It is the P3P geometric model of the present invention;

[0051] Figure 6 Schematic diagram of the transformation relationship of the drill rod loading and unloading system of the present invention;

[0052] Figure 7 Schematic diagram of the coordinate system transformation relationship of the present invention, including (a) the posture transformation of coordinate system B and coordinate system A; (b) the posture transformation of coordinate system C and coordinate systems A and B;

[0053] Figure 8 Schematic diagram of the coordinate system relationship after the drilling rig posture changes of the present invention;

[0054] Figure 9 This is a structural diagram of the vision-based automatic loading and unloading drill rod positioning device for a tunnel drilling robot used in coal mines according to the present invention;

[0055] Figure 10 It is a structural schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0056] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0057] Schematic diagram of automatic loading and unloading of drill rods by tunnel drilling robots in coal mines Figure 2 As shown, in this system, the drill rod loading process requires relying on the method of the present invention to move the robot with the drill rod to the drill rod loading and unloading clamp to complete the drilling operation; during the drill unloading process, it is necessary to first move the robot to the drill rod loading and unloading clamp to grab the drill rod, and then move the drill rod to the drill rod warehouse with the drill rod.

[0058] refer to Figure 1 The present invention provides a vision-based method for automatically loading and unloading drill rod positioning of a tunnel drilling robot for coal mines, comprising the following steps:

[0059] S1: Calibrate the camera and perform hand-eye calibration to obtain the hand-eye matrix;

[0060] S2: Teach the camera's photo point and the drill rod placement point, collect an image containing the cooperative target as the desired image, set the relative posture between the camera's photo point and the cooperative target as a fixed constraint based on the robot's end-point posture, hand-eye matrix, and desired image information, and establish a vision-based positioning model for loading and unloading drill rods.

[0061] S3: The robot vision system controls the robot movement to satisfy the fixed constraints between the camera shooting point and the cooperative target;

[0062] S4: Calculate the drill rod placement position based on the robot's posture after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the gripper position to place or grab the drill rod.

[0063] Step S1 specifically includes:

[0064] S11: Calibrate the two-dimensional camera. Based on multiple sets of images containing cooperative targets acquired by the camera, establish a mapping relationship between objects in the three-dimensional space and the two-dimensional camera plane, and calculate the camera's intrinsic parameter information.

[0065] S12: Fix the camera at the end of the robot, move it to different positions, and record images to calculate the transformation relationship between the robot end flange and the camera center, that is, the hand-eye matrix E T C .

[0066] Preferably, the camera is connected to the last joint link of the robot and fixed somewhere in the robot workspace.

[0067] Step S2 specifically includes:

[0068] S21: Using the robot teaching pendant, move the robot carrying the drill rod to the drilling rig clamping slot and record it as the drill rod placement point;

[0069] S22: Move the robot to the top of the drilling rig and record it as the camera photo point;

[0070] S23: Acquire an image containing the cooperative target as a desired image;

[0071] S24: According to the robot end posture, hand-eye matrix, and expected image information, the relative posture between the camera shooting point and the cooperative target is set as a fixed constraint condition, and a vision-based positioning model for loading and unloading drill rods is established.

[0072] Step S3 specifically includes:

[0073] S31: Based on the vision-based positioning model for loading and unloading drill rods, the camera intrinsic parameter information and the P3P algorithm are used to solve the transformation matrix of the cooperative target in the camera coordinate system in the images collected during the robot loading and unloading drill rods. The relative posture transformation matrix between the cooperative target and the initial camera in the initial setting expected image is: That is, the system fixed agreed conditions; according to the relative posture of the relative cooperative target obtained in the collected image and the set expected image, the transformation matrix between the current camera coordinate system and the camera coordinate system under the ideal fixed constraint is obtained according to the transformation relationship between the coordinate systems

[0074]

[0075] in, Represents the transformation matrix of the cooperative target in the current camera coordinate system, Represents the fixed constraint between the cooperative target and the desired camera shooting point, i.e., the camera coordinate system;

[0076] S32: According to the inter-system coordinate conversion framework of the robot base coordinate system, the manipulator end coordinate system, the camera coordinate system, the target cooperative target coordinate system and the gripper coordinate system in the drilling rig loading and unloading system, the pose matrix of the robot end camera coordinate system of the current posture is obtained.

[0077]

[0078] S33: Based on the transformation matrix between the current camera pose and the desired camera pose Get the complete transformation path of the transformation matrix for cooperative target recognition and positioning based on the current robot position:

[0079]

[0080] in It is the transformation matrix between the robot end coordinate system and the robot base coordinate system obtained by the robot teaching pendant. E T C is the hand-eye matrix obtained by the hand-eye calibration algorithm, is the transformation matrix between the cooperative target pose and the desired camera pose;

[0081] S34: The next posture of the robot end camera is calculated by the transformation matrix of cooperative target recognition and positioning, so that the relative posture between the current camera shooting position and the cooperative target image satisfies the fixed constraint between the initial camera shooting point and the initial cooperative target. The robot end posture transformation matrix of the next posture is:

[0082]

[0083] Step S4 specifically includes:

[0084] S41: After the relative posture between the camera shooting position and the cooperative target image is completed to meet the precise positioning between the initial camera shooting point and the initial cooperative target, the transformation matrix is obtained according to the camera shooting point and drill rod placement point obtained by manual teaching. Get the description of the drill rod placement position in the robot's world coordinate system, that is, the drill rod placement point:

[0085]

[0086] in B T H Represents the transformation matrix of the drill rod placement position in the robot base coordinate system, represents the transformation matrix for cooperative target recognition and positioning based on the current position, represents the inverse transformation matrix of the hand-eye matrix, Represents the transformation matrix of the drill rod placement point in the robot end coordinate system.

[0087] Next, based on the above steps, the specific details of the implementation process are described in detail:

[0088] 1. Camera Model

[0089] In order to obtain image features, it is necessary to convert the three-dimensional coordinates into two-dimensional image coordinates. Consider Figure 3 The camera perspective projection model shown, assuming that the point P = (X, Y, Z) in space is projected on the image plane as a two-dimensional point p = (u, v), then we can get:

[0090] sp=K[R|T]P (I)

[0091]

[0092] Where s represents the scale factor of the image point, f x and f y is the camera's zoom focal length, γ is sometimes assumed to be 0, K represents the camera's intrinsic parameter matrix, u0 and v0 are the central reference points of the image coordinate system, and R and T are the three-dimensional rotation and three-dimensional translation used to calculate the camera's extrinsic parameters. In the method proposed in this invention, a cooperative target in three-dimensional space is captured by a camera and then projected onto the image plane according to the camera perspective projection model of formula (2). The relative pose between the camera and the cooperative target is calculated using the P3P algorithm.

[0093] 2. P3P algorithm solves relative posture

[0094] In vision-guided industrial robotics applications, a 2D camera is connected to the last joint link of the industrial robot and fixed somewhere in the robot's workspace. The position measurement of the robot and the cooperating target in the robot workcell is mathematically established using Cartesian coordinate systems and their relative poses. These include the robot base coordinate system (B), the end-of-arm coordinate system (E), the camera coordinate system (C), the target cooperating target coordinate system (P), and the gripper coordinate system (H). Figure 4 shown.

[0095] After camera calibration in a vision project, the camera's intrinsic parameter matrix is obtained. By taking a photo, the relationship between the camera and the target can be determined. Feature points are widely used in vision-based pose estimation. To further determine the pose of the target, the P3P algorithm is required. The P3P algorithm describes how to estimate the camera's pose when three 3D spatial points and their projected positions are known.

[0096] The P3P problem can be described as follows: given three 3D control points P1, P2, P3, three distances a = ||P2-P3||, b = ||P1-P3||, c = ||P1-P2||, three-dimensional points P1, P2, P3 correspond to three two-dimensional points p1, p2, p3, and the unknown distances S1, S2, S3 from p1, p2, p3 to the camera optical center are calculated using the camera intrinsic parameters. The geometric model corresponding to the P3P problem is as follows Figure 5 As shown, the equation can be described as follows:

[0097]

[0098] Where x, y, and z are the distances between the camera's optical center and the three-dimensional points P1, P2, and P3, respectively, and α, β, and γ are the angles between them. Using the principle of similar triangles and the law of cosines, we can solve for x, y, and z using α, β, γ, and a, b, and c.

[0099] The pose transformation relationship from the camera coordinate system to the cooperative target coordinate system is based on equation (3). Considering different combinations of signs, the solution to the P3P problem corresponds to four modes:

[0100] Model 1:

[0101] Model2:

[0102] Model 3:

[0103] Model 4:

[0104] The above four situations cannot be judged by the position of the cooperative target in the actual space. Therefore, in this embodiment, four points of the cooperative target are selected for calculation, three of which are used for P3P calculation and one point is used for verification. The solution with the smallest calculation error among the four groups is taken as the correct solution. Solving the transformation matrix of the camera coordinate system and the cooperative target coordinate system, we can get:

[0105]

[0106] where R 3×3 and T 3×1 They represent the rotation matrix and translation matrix from the target coordinate system to the camera coordinate system respectively.

[0107] Figure 6 Describes the relationship between the coordinates of the drill pipe loading and unloading system, E d Indicates the desired initial position of the robot end delivering the drill pipe, C d represents the camera pose at the desired end pose of the drill rod delivery robot, E cis the current robot end position, obtained through the robot teaching pendant, C c The camera pose in the current robot state is represented by the transformation relationship between the camera coordinate system and the robot end coordinate system can be obtained by hand-eye calibration, so the camera position can be calculated based on the known robot end position. Therefore, the pose recognition of the delivered drill rod is essentially to establish a relationship with the cooperative target through the camera, and use the P3P algorithm to solve the pose transformation matrix from the camera coordinate system to the cooperative target coordinate system based on the cooperative target size, cooperative target pixel coordinates, and camera internal parameters. C T P .

[0108] 3. Calculation of robot target movement pose based on inherent constraints

[0109] In Cartesian space, the position transformation relationship between coordinate system B (n, o, a) and coordinate system A (x, y, z) can be expressed using a 4×4 homogeneous transformation matrix A T B express:

[0110]

[0111] In (9), the data elements in the first three columns and the fourth column represent the direction and position of coordinate system B relative to coordinate system A, respectively. The transformation matrix A T B The inverse matrix of B T A , expressed as the direction and position of coordinate system A relative to coordinate system B as Figure 7 As shown in a, the coordinate system transformation equation can be established from the transformation matrix, and each matrix is used to solve an unknown coordinate system transformation. Figure 7 As shown in b, the coordinate system C initially coincides with the coordinate system A and is then transformed to a new position. In this case, the known coordinate system transformation matrix A T B and C T A , unknown coordinate system transformation C T A It can be determined as:

[0112] C T A = C T A × A T B (10)

[0113] Formula (10) corresponds to the transformation arrow in the figure multiplying each corresponding transformation matrix along the path, according to Figure 6 The coordinate transformation framework shown only needs to know the transformation matrix between the cooperative target pose and the desired camera pose You can get the complete transformation path, follow the arrow instructions, and set You can get:

[0114]

[0115] in It is the transformation matrix between the robot end coordinate system and the robot base coordinate system obtained by the robot teaching pendant. E T C is the hand-eye matrix obtained by the hand-eye calibration algorithm, is the transformation matrix between the cooperative target pose and the desired camera pose, which can be obtained by (8), C T E By solving E T C The inverse of can be obtained, thereby calculating the transformation matrix of cooperative target recognition and positioning based on the current position

[0116] After completing the precise positioning of the drill rod delivered by the robot based on the cooperative target monocular vision, the robot can The description of the delivery position in the robot world coordinate system can be obtained:

[0117]

[0118] After the drilling rig posture changes, the system coordinate system relationship is as follows Figure 8 As shown in formula (12), it can be seen that the method proposed in the present invention has no direct relationship with the change of the drilling rig posture when calculating the final delivery position, and the set spatial constraints will not change due to the change of the drilling rig posture. Therefore, after the horizontal directional drilling rig causes the default clamp position to change due to factors such as rotation, it can still complete the drill rod delivery task after using the cooperative target for precise positioning, based on the characteristic that the relative posture between the cooperative target and the clamp remains unchanged.

[0119] The key points of the present invention in implementing the above technical solution are:

[0120] 1. Using a vision-based automated drill rod loading and unloading method in coal mine drilling systems effectively addresses the issue of decreased model accuracy caused by cumulative sensor errors, mechanical wear, and model changes. This significantly improves the automation and intelligence of the drill rod loading and unloading process. The key technical approach lies in establishing a suitable robot vision model and constructing appropriate coordinate system transformations. This allows for adaptive and precise positioning of the drill rig after changes in posture, requiring only two points to be taught.

[0121] 2. A drill rod loading and unloading positioning method based on hand-on-eye visual recognition, using cooperative objectives, enables precise positioning of the drill rig after changes in its posture, requiring only two initial points. The key technical aspect is that this method adapts to fully automated positioning of drill rod loading and unloading under varying drill rig postures, improving system efficiency and reliability.

[0122] In order to better implement the above method of the present invention, a vision-based automatic loading and unloading drill rod positioning device for a coal mine tunnel drilling robot provided by the present invention is described below. The device described below and the method described above can be referenced to each other.

[0123] like Figure 9 As shown, a vision-based automatic loading and unloading drill rod positioning device for a tunnel drilling robot used in a coal mine includes the following units:

[0124] Calibration unit 001, used to calibrate the camera and robot hand-eye calibration to obtain a hand-eye matrix;

[0125] Constraint unit 002 is used to teach the camera shooting point and drill rod placement point, collect an image containing the cooperative target as the expected image, set the relative posture between the camera shooting point and the cooperative target as a fixed constraint condition based on the robot end-point posture, hand-eye matrix, and expected image information, and establish a vision-based positioning model for loading and unloading drill rods;

[0126] The control unit 003 is used to control the movement of the robot through the robot vision system so as to satisfy the fixed constraint conditions between the camera shooting point and the cooperative target;

[0127] The calculation unit 004 is used to calculate the drill rod placement position based on the posture of the robot after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the clamp position to place or grab the drill rod.

[0128] Furthermore, if Figure 10As shown, an example of a physical structure diagram of an electronic device is shown, which may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the steps of the above-mentioned method for automatically loading and unloading drill rods for a tunnel drilling robot for coal mines, specifically including: calibrating the camera and performing robot hand-eye calibration to obtain a hand-eye matrix; teaching the camera shooting point and the drill rod placement point, collecting an image containing a cooperative target as an expected image, and setting the relative posture between the camera shooting point and the cooperative target as a fixed constraint condition based on the robot end posture, the hand-eye matrix, and the expected image information, and establishing a vision-based positioning model for loading and unloading drill rods; the robot vision system controls the movement of the robot to satisfy the fixed constraint condition between the camera shooting point and the cooperative target; based on the posture of the robot after movement, the transformation relationship between the drill rod placement point and the camera shooting point, the drill rod placement position is calculated, and the robot joint is controlled to drive the robot to the clamp position to place or grab the drill rod.

[0129] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0130] On the other hand, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for automatically loading and unloading drill rods for a tunnel drilling robot for coal mines, specifically including: calibrating the camera and performing hand-eye calibration on the robot to obtain a hand-eye matrix; teaching the camera taking points and drill rod placement points, collecting an image containing a cooperative target as an expected image, and setting the relative posture between the camera taking point and the cooperative target as a fixed constraint condition based on the robot end posture, the hand-eye matrix, and the expected image information, and establishing a vision-based positioning model for loading and unloading drill rods; the robot vision system controls the movement of the robot to satisfy the fixed constraint condition between the camera taking point and the cooperative target; based on the posture of the robot after movement, the transformation relationship between the drill rod placement point and the camera taking point, the drill rod placement position is calculated, and the robot joint is controlled to drive the robot to the clamp position to place or grab the drill rod.

[0131] After the implementation of the present invention, the following beneficial effects are achieved:

[0132] 1. This invention provides a vision-based method for automatically positioning drill rods for coal mine tunnel drilling robots. This method addresses the low efficiency, automation, and intelligence levels associated with conventional automated drill rod positioning systems, which require extensive manual assistance. It also effectively avoids the unreliable automatic positioning caused by degradation of model accuracy during later use due to wear and errors. This vision-based method effectively addresses the shortcomings of conventional methods.

[0133] 2. The present invention constructs a visual positioning method based on fixed constraints, which solves the problem of traditional methods requiring repeated manual teaching after the drill rig posture changes. Currently, the commonly used automatic loading and unloading methods for coal mine drill rigs all require re-teaching of the loading and unloading points after the posture changes, which is very labor-intensive and time-consuming. While greatly reducing the initial teaching work, this method can still complete the vision-based coal mine tunnel drilling robot automatic loading and unloading drill rod positioning task after the drill rig posture changes.

[0134] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0135] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.

[0136] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vision-based method for automatically loading and unloading drill rod positioning for a tunnel drilling robot in a coal mine, characterized in that: The following steps are involved: S1: Calibrate the camera and robot hand-eye calibration to obtain the hand-eye matrix; S2: Teach the camera's photo point and the drill rod placement point, collect an image containing the cooperative target as the desired image, set the relative posture between the camera's photo point and the cooperative target as a fixed constraint based on the robot's end-point posture, hand-eye matrix, and desired image information, and establish a vision-based positioning model for loading and unloading drill rods. S3: The robot vision system controls the robot movement to satisfy the fixed constraints between the camera shooting point and the cooperative target; S4: Calculate the drill rod placement position based on the robot's posture after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the gripper position to place or grab the drill rod; Step S3 specifically includes: S31: Based on the vision-based positioning model for loading and unloading drill rods, the camera intrinsic parameter information and the P3P algorithm are used to solve the transformation matrix of the cooperative target in the camera coordinate system in the images collected during the robot loading and unloading drill rods. The relative posture transformation matrix between the cooperative target and the initial camera in the initial setting expected image is: That is, the system fixed agreed conditions; according to the relative posture of the relative cooperative target obtained in the collected image and the set expected image, the transformation matrix between the current camera coordinate system and the camera coordinate system under the ideal fixed constraint is obtained according to the transformation relationship between the coordinate systems in, Represents the transformation matrix of the cooperative target in the current camera coordinate system, Represents the fixed constraint between the cooperative target and the desired camera shooting point, i.e., the camera coordinate system; S32: According to the inter-system coordinate transformation framework of the robot base coordinate system, the manipulator end coordinate system, the camera coordinate system, the target cooperative target coordinate system and the gripper coordinate system in the drilling rig loading and unloading system, the pose matrix of the robot end camera coordinate system of the current posture is obtained. S33: Based on the transformation matrix between the current camera pose and the desired camera pose Get the complete transformation path of the transformation matrix for cooperative target recognition and positioning based on the current robot position: in It is the transformation matrix between the robot end coordinate system and the robot base coordinate system obtained by the robot teaching pendant. E T C is the hand-eye matrix obtained by the hand-eye calibration algorithm, is the transformation matrix between the cooperative target pose and the desired camera pose; S34: The next posture of the robot end camera is calculated by the transformation matrix of cooperative target recognition and positioning, so that the relative posture between the current camera shooting position and the cooperative target image satisfies the fixed constraint condition between the initial camera shooting point and the initial cooperative target; The robot end posture transformation matrix of the next posture is: C T E The hand-eye matrix E T C The inverse matrix of Step S4 specifically includes: S41: After the relative posture between the camera shooting position and the cooperative target image is completed to meet the precise positioning between the initial camera shooting point and the initial cooperative target, the transformation matrix is obtained according to the camera shooting point and drill rod placement point obtained by manual teaching. Get the description of the drill rod placement position in the robot's world coordinate system, that is, the drill rod placement point: in B T H Represents the transformation matrix of the drill rod placement position in the robot base coordinate system, represents the transformation matrix for cooperative target recognition and positioning based on the current position, represents the inverse transformation matrix of the hand-eye matrix, Represents the transformation matrix of the drill rod placement point in the robot end coordinate system.

2. The method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for coal mines according to claim 1, characterized in that: Step S1 specifically includes: S11: Calibrate the two-dimensional camera. Based on multiple sets of images containing cooperative targets acquired by the camera, establish a mapping relationship between objects in the three-dimensional space and the two-dimensional camera plane, and calculate the camera's intrinsic parameter information. S12: Fix the camera at the end of the robot, move it to different positions, and record images to calculate the transformation relationship between the robot end flange and the camera center, that is, the hand-eye matrix E T C .

3. The method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for coal mines according to claim 1, characterized in that: Step S2 specifically includes: S21: Using the robot teaching pendant, move the robot carrying the drill rod to the drilling rig clamping slot and record it as the drill rod placement point; S22: Move the robot to the top of the drilling rig and record it as the camera photo point; S23: Acquire an image containing the cooperative target as a desired image; S24: According to the robot end posture, hand-eye matrix, and expected image information, the relative posture between the camera shooting point and the cooperative target is set as a fixed constraint condition, and a vision-based positioning model for loading and unloading drill rods is established.

4. The method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for coal mines according to claim 1, characterized in that: Before step S1, the camera is set to be connected to the last joint link of the robot and fixed somewhere in the robot workspace.

5. A vision-based automatic loading and unloading drill rod positioning device for a coal mine tunnel drilling robot, used to implement the steps of the automatic loading and unloading drill rod positioning method for a coal mine tunnel drilling robot according to any one of claims 1 to 4, characterized in that: The following units are included: The calibration unit is used to calibrate the camera and the robot's hand-eye calibration to obtain the hand-eye matrix; The constraint unit is used to teach the camera's photo point and the drill rod placement point. It collects an image containing the cooperative target as the desired image. Based on the robot's end-point posture, the hand-eye matrix, and the desired image information, it sets the relative posture between the camera's photo point and the cooperative target as a fixed constraint condition, and establishes a vision-based positioning model for loading and unloading drill rods. A control unit is used to control the movement of the robot through the robot vision system so as to satisfy the fixed constraint conditions between the camera shooting point and the cooperative target; The calculation unit is used to calculate the drill rod placement position based on the posture of the robot after movement, the transformation relationship between the drill rod placement point and the camera shooting point, and control the robot joints to drive the robot to the gripper position to place or grab the drill rod.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for a coal mine according to any one of claims 1 to 4 are implemented.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically loading and unloading drill rods and positioning a tunnel drilling robot for a coal mine as described in any one of claims 1 to 4 are implemented.

Images