Drilling robot end effector and target object relative position positioning method based on machine vision

Abstract

The invention provides a drilling robot end effector and target object relative position positioning method based on machine vision, and the method comprises the steps: hanging a Zhang Zhengyou calibration plate at the tail end of a mechanical arm, carrying out the camera calibration of a binocular vision camera, and obtaining the internal reference of the camera; placing a Zhang Zhengyou calibration plate on a plane position, controlling an end effector to reach a plurality of fixed feature angular point positions on the calibration plate, collecting calibrated image data, and recording feature angular point data; performing hand-eye calibration calculation based on the obtained image data and feature angular point data to obtain a hand-eye calibration conversion matrix; different positions of a Charuco mark code under a camera coordinate system are recorded, a calculation result of hand-eye calibration is converted into a position under a robot coordinate system, and on the basis of data of an end effector under different positions, conversion matrix calculation of relative positions is carried out on the end effector and the Charuco mark code, so that a positioning result is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical fields of visual positioning and industrial robots, and particularly relates to a method for positioning the relative position between the end effector of a drilling robot and a target object based on machine vision. Background Art

[0002] Robotic-assisted surgery can help doctors perform more repeatable and accurate surgeries through patient-specific surgical plans. The robotic system utilizes advanced imaging technologies and robotic operating systems to provide doctors with a more precise and safer surgical operation platform. However, there is currently little overall simulation research on robotic skull drilling, and there are also insufficient studies on the three-dimensional visualization simulation, drilling trajectory planning, and verification of the surgical process of the drilling surgical robot. There is a need for real-time monitoring of the drilling depth and the implementation of online monitoring technologies. To ensure the real-time monitoring of the surgical process, it is particularly necessary to ensure accuracy and safety. Currently, digital twins are widely used in the medical field. It can not only accurately simulate the operating state and performance of physical systems, but also predict and optimize the operation of the system through real-time data monitoring and analysis.

[0003] In order to achieve the real-time synchronous and precise operation of the robot under digital twins, it is necessary to ensure that the relative position between the drilling robot and the target object in the virtual environment is consistent with the relative position relationship between the physical models. However, it is very difficult to achieve this by manual measurement alone, and there are often errors of dozens of millimeters, which will make the safety of the drilling surgical robot unable to be guaranteed. Therefore, it is necessary to use the hand-eye calibration method to obtain the position of the end effector of the robot, and at the same time obtain the position information of the target object from the perspective of the camera, so that the relative position relationship between the two can be reconstructed in the virtual environment to ensure the smooth progress of the drilling process.

[0004] Currently, the positioning methods of most digital twin systems are to place a marker on the operating platform as the origin of the base coordinate system recognized by the camera, and obtain the position information of each object in this coordinate system. Or use the workpiece calibration method of the robot itself to obtain the position information of the workpiece in the robot coordinate system. However, the above methods have very high requirements for the position and shape of the workpiece. For irregular objects with randomly placed positions and irregular shapes, more suitable methods are needed to meet the requirements of position accuracy. Summary of the Invention

[0005] To overcome the defects and deficiencies of the existing technologies, the present invention proposes a method for positioning the relative position between the end effector of a drilling robot and a target object based on machine vision, and on this basis, improves the digital twin system to achieve more comprehensive process support for robot drilling and real-time monitoring of position information. In the solution provided by the present invention, a checkerboard on a plane is used to calibrate the robot with an end effector and a camera. After obtaining the calibration transformation matrix of the robot and camera coordinate systems, the position information of the end point of the robot in the camera coordinate system is obtained in real time by using the Charuco marker corner recognition algorithm and the calibration transformation matrix. At the same time, the camera combines the recognition method of the marker ball to position the target object, obtains the position information of the target object in the camera coordinate system, and realizes the positioning of the relative position relationship between the end effector and the target object. It effectively meets the positioning requirements of irregular target objects, and at the same time ensures the safety and accuracy of the drilling operation of the end effector.

[0006] The technical solution specifically adopted by the present invention to solve its technical problems is as follows:

[0007] A method for positioning the relative position between the end effector of a drilling robot and a target object based on machine vision:

[0008] The end effector is connected to the end of the robotic arm of the drilling robot;

[0009] The Charuco marker is mounted on the end effector and has a fixed relative position with the end effector, and is used as a marker tool for the camera to identify and track the end point of the end effector;

[0010] The binocular vision camera is used to perform real-time tracking and monitoring of the Charuco marker;

[0011] The positioning method includes:

[0012] Hang the Zhang Zhengyou calibration board at the end position of the robotic arm to calibrate the binocular vision camera and obtain the internal parameters of the camera;

[0013] Place the Zhang Zhengyou calibration board in a planar position, control the end effector to reach several fixed feature corner positions on the calibration board, collect the calibrated image data, and record the feature corner data; based on the obtained image data and feature corner data, perform hand-eye calibration calculation to obtain the hand-eye calibration transformation matrix;

[0014] By recording different positions of the Charuco marker in the camera coordinate system and converting them into positions in the robot coordinate system through the calculation results of hand-eye calibration, based on the data of the end effector at different positions, calculate the transformation matrix of the relative position between the end effector and the Charuco marker to obtain the positioning result.

[0015] Furthermore, a four-point calibration method is used to calibrate the end effector to obtain the position and posture relationship matrix of the end effector with respect to the robot coordinate system.

[0016] Furthermore, the specific process of the hand-eye calibration calculation includes:

[0017] Adjust the posture of the end effector to be vertically downward and keep it unchanged, and collect the left and right view images of Zhang Zhengyou's calibration plate in the binocular vision camera;

[0018] Control the robot arm to move toward the target calibration plate, and make the end point of the end effector reach the characteristic corner point position of Zhang Zhengyou calibration plate, use the end point of the end effector to touch the preset n characteristic point positions of Zhang Zhengyou calibration plate in a certain order, and record the n spatial coordinates of these characteristic points in the robot base coordinate system

[0019] Hand-eye calibration calculation is performed based on the collected image of the Zhang Zhengyou calibration plate and the feature point data of the Zhang Zhengyou calibration plate.

[0020] Furthermore, the process of performing hand-eye calibration calculation based on the collected image of the Zhang Zhengyou calibration plate and the feature point data of the Zhang Zhengyou calibration plate includes:

[0021] Perform corner point detection on the image data of Zhang Zhengyou's calibration plate, and obtain the pixel coordinates of the preset n feature corner points under the left and right visual cameras and

[0022] The pixel coordinates of the feature corner points under the left and right view cameras are matched one by one, and the intrinsic parameter matrix obtained by the joint camera calibration is calculated by the PnP algorithm to obtain the three-dimensional point coordinate information of n feature corner points in the camera coordinate system.

[0023] According to the pose data of the end point when the end point of the end effector touches the feature point of the chessboard, the position information of the feature corner point in the robot coordinate system is obtained.

[0024] Based on the position information data of the feature corner points in the robot coordinate system and the camera coordinate system and Construct the transformation matrix relationship:

[0025]

[0026] The SVD eigenvalue decomposition method is used to solve the matrix and obtain the hand-eye calibration matrix

[0027] Pair Matrix Perform singular value decomposition:

[0028]

[0029] Calculate the rotation matrix R and translation vector t from U and V:

[0030]

[0031] Among them C b and C c is the centroid of the point set in the robot coordinate system and the camera coordinate system.

[0032] Furthermore, the process of calculating the transformation matrix of the relative position between the end effector and the Charuco marking code includes:

[0033] Through Charuco corner point recognition, the pixel coordinates of the four Charuco corner points under the left and right visual cameras are obtained;

[0034] The pixel coordinates of the four corner points are averaged, and the left and right pixel coordinates of the average value are combined with the intrinsic parameter matrix obtained by camera calibration to calculate the PnP algorithm to obtain the three-dimensional point coordinate information of the average feature corner point in the camera coordinate system;

[0035] Combine the 3D point coordinate information of the average feature corner point of the Charcuo marker code with the hand-eye calibration conversion matrix to obtain the 3D point coordinate information of the average feature corner point of the Charcuo marker code in the robot coordinate system;

[0036] One-to-one correspondence with the recorded position of the end effector, calculate the conversion matrix between the Charuco marker code and the end effector

[0037] Furthermore, given the known transformation matrix In this case, the corner points of the Charuco marker code are tracked in real time, and the position information of the corner points in the camera coordinate system is solved, so as to infer the position information of the end point of the robot end effector in the camera coordinate system.

[0038] Furthermore, at least three feature marker balls that are not in the same plane are arranged on the target object; the orientation setting of the binocular vision camera should satisfy the real-time tracking and monitoring of the Charuco marker code and the acquisition of the position of the feature marker ball; the image features of the marker ball on the target object are acquired by the camera in real time, and the coordinate position of the marker ball is calculated by the triangulation method, and the accuracy and robustness of positioning are improved by the coordination of multiple marker balls.

[0039] Further, the method for obtaining the image features of the marker balls on the target object in real time through the camera and calculating the coordinate positions of the marker balls using the triangulation method is specifically as follows: capture the environmental image containing the marker balls through the camera, identify the edges, positions, and directions of the marker balls through the edge detection algorithm, and classify them according to the features of the marker balls; use the known internal and external parameters of the camera to convert the coordinates of the marker balls in the two-dimensional image into positions in the three-dimensional space through the triangulation method, and calculate the coordinate positions and postures of each marker ball in the three-dimensional space.

[0040] In addition, an electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for positioning the relative position between the end effector of a drilling robot based on machine vision and a target object as described above.

[0041] A non-transitory computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method for positioning the relative position between the end effector of a drilling robot based on machine vision and a target object as described above.

[0042] Aiming at the defects and deficiencies of the prior art, the present invention and its preferred solutions effectively reduce the time required for robot hand-eye calibration, place the end point of the end effector in the camera coordinate system for real-time monitoring of the position, can greatly reduce the influence brought by the offset error of the robot itself, and at the same time, the corner recognition of the Charuco marker code combined by multiple Aruco marker codes can better ensure the recognition accuracy. Placing the end effector and the target object in the same coordinate system can also more accurately locate the relative position relationship between the two. In addition, the calibration method of the present invention also fully considers the rigid transformation between coordinate systems, does not require the use of the robot's calibration method and consideration of the shape of the target object, and at the same time ensures that as long as the camera can recognize the marker code and the marker, the two can be located, without setting a certain fixed distance, avoiding the problem of difficult distance setting between the object and the marker. Further, a complete digital twin system of the drilling robot is established through the high-fidelity simulation of the Unity3D platform, and the position change curve of the end point of the end effector of the robot is displayed in real time and efficiently on the platform, providing strong help for ensuring the safety of the drilling process. Description of the Drawings

[0043] The following further details the present invention in conjunction with the drawings and specific embodiments:

[0044] Figure 1 It is the device diagram corresponding to the calibration scheme of the embodiment of the present invention;

[0045] Figure 2 It is the calibration checkerboard diagram of the embodiment of the present invention;

[0046] Figure 3 This is the calibration process diagram of the realsense D435 camera in the embodiment of the present invention;

[0047] Figure 4 This is the calibration process diagram of the robot's hand-eye in the embodiment of the present invention;

[0048] Figure 5 This is the schematic diagram of the Charuco marker code in the embodiment of the present invention;

[0049] Figure 6 This is the recognition diagram of the corner point position of the marker code in the embodiment of the present invention;

[0050] Figure 7 This is the comparison diagram of the position information between the end point obtained by solving and the teach pendant in the embodiment of the present invention;

[0051] Figure 8 This is the relative position relationship diagram obtained in real time from the video stream in the embodiment of the present invention. Detailed implementation manners

[0052] To make the features and advantages of the present invention more obvious and understandable, specific embodiments are given below and described in detail as follows:

[0053] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanations for the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0054] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present application. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0055] As Figure 1 shown, the embodiment of the present invention first provides a device design for positioning the relative position between the end effector of a drilling robot and a target object based on machine vision:

[0056] For a six-axis robot with a six-axis robotic arm as the main body, its base is fixed at a suitable position, and an end effector is installed and connected through a flange at the end flange of the robot.

[0057] Assume that the target object is placed on the operating platform, and three feature marking balls that are not in the same plane are set on the target object. It is necessary to ensure that the positions of the marking balls do not slip and are clearly visible in the camera's view. During the actual use of the solution, the above "target object" can be understood as the part to be operated on. For example, in skull drilling-related surgeries, this corresponds to the skull part. At this time, the above-mentioned "placement of the target object" should be understood as the actual position of the part to be operated on the operating table.

[0058] The vision drilling device includes a binocular vision camera, a Zhang Zhengyou calibration board, a Charuco marker code, and a bone drilling device; among them, the bone drilling device is connected to the end flange of the robotic arm as the end effector through an end connector;

[0059] The Charuco marker code is a marking tool used to be recognized and tracked by the camera at the end point of the end effector. Therefore, it needs to be clearly visible within the camera's view and can be tracked in real time. So the marker code is mounted on the end effector through a designed bone drill connector, and the relative position relationship between the marker code and the end effector is fixed;

[0060] The binocular vision camera is used to perform real-time tracking and monitoring of the marker code on the end effector of the robot. At the same time, it is also necessary to obtain the specific positions of the marking balls of the target object. When placing the vision camera directly in front of the end effector, there needs to be a certain angular offset to ensure that the position information of the end effector and the target object is visible within the camera's view. After the binocular camera determines the position, it remains fixed;

[0061] During the hand-eye calibration process, a Zhang Zhengyou calibration board needs to be placed on the platform for calibration. After the calibration of the robotic arm and the camera coordinate system is completed, the checkerboard calibration board is removed.

[0062] Based on the design of the above device (mainly the components of each device and their relative position relationships), the embodiment of the present invention also provides a method for positioning the relative position between the end effector of a drilling robot based on machine vision and a target object, including:

[0063] Step S1: First, hang the Zhang Zhengyou calibration board at the end position of the robotic arm to perform camera calibration on the binocular vision camera and obtain the internal parameters of the camera;

[0064] Step S2: Place the Zhang Zhengyou calibration board in a planar position, control the end effector of the robotic arm to reach several fixed feature corner points on the calibration board, collect the calibrated image data, and record the feature corner point data at the same time; based on the obtained image data and feature corner point data, perform hand-eye calibration calculation;

[0065] Specifically, as Figure 2As shown, considering that the intrinsic characteristics of the binocular vision camera remain unchanged, in this embodiment, the internal parameters of the camera are obtained through checkerboard calibration (i.e., Zhang Zhengyou calibration board):

[0066] Mount the checkerboard at the position of the end effector of the robot, move the robot to an appropriate position, and make the checkerboard occupy more than half of the camera's field of view to ensure that the camera can clearly and accurately capture the content of the checkerboard;

[0067] By identifying the position coordinates of the corner points at the intersections of adjacent squares and establishing a plane coordinate system with the position of the first corner point in the upper left corner as the origin of the checkerboard, calculate the position of the camera under this checkerboard;

[0068] Continuously change the position and attitude of the checkerboard within the camera's field of view, and use the camera to capture the scenes containing the calibration board, respectively obtaining the position information of the camera in different checkerboard postures. Obtaining about 15 or more images can yield accurate internal parameters and distortion parameters of the camera, which also marks the completion of the camera calibration. Taking the calibration process of the realsense D435 camera as an example, its implementation is shown as Figure 3 shown.

[0069] The end effector calibration is achieved by using the four-point calibration method. A reference calibration process is as follows: Place a pen tip on the working platform, control the robot to move so that the end effector touches the tip of the pen. Each touch has a relatively large difference in posture compared to the previous touches. Touch the tip four times in this way and record the pose information of each touch to obtain the pose relationship matrix of the end effector with respect to the robot coordinate system.

[0070] Step S21: As Figure 4 shown, adjust the posture of the end effector to be vertically downward and keep it unchanged. Under this fixed posture, collect the left and right view images of the checkerboard calibration board in the binocular vision camera. The checkerboard is placed on the working platform, and the binocular camera is slightly downward so that the checkerboard and the end point of the end effector are visible within the camera's field of view, and keep the camera stationary;

[0071] Step S22: The robotic arm moves towards the target checkerboard and makes the end point of the end effector reach the characteristic corner point position of the Zhang Zhengyou calibration board. Use the end point to poke the characteristic points of the preset n checkerboard calibration boards in a certain order and record the n spatial coordinates of these characteristic points in the robot base coordinate system When reaching the position of each corner point, observe whether the end point has reached the target position and has a certain touch with the target point to avoid calibration deviation caused by not being in place;

[0072] While recording the position information of the feature corner points in the robot coordinate system, it is necessary to use a binocular vision camera to capture the image information of the chessboard in advance, use the corner detection algorithm to detect and identify the preset feature corner points and obtain their pixel coordinates, and then use the pixel coordinates and camera intrinsic parameters to combine them. After calculation by the stereo vision algorithm, the three-dimensional point position information of the target detection corner points in the camera coordinate system can be obtained.

[0073] Step S23: performing hand-eye calibration calculation based on the acquired image of the checkerboard calibration plate and the feature point data of the checkerboard calibration plate.

[0074] Specifically, the main process of hand-eye calibration calculation based on the collected image of the checkerboard calibration plate and the feature point data of the checkerboard calibration plate is:

[0075] First, fix the chessboard on the work platform and take a picture of the environment containing the chessboard with a camera. Perform corner point detection on the image data of the chessboard calibration plate in Matlab and obtain the pixel coordinates of the preset n feature corner points. and The pixel coordinates of the feature corner points under the left and right view cameras are matched one by one, and the intrinsic parameter matrix obtained by the joint camera calibration is calculated by the PnP algorithm to obtain the three-dimensional point coordinate information of n feature corner points in the camera coordinate system.

[0076] After obtaining the above n feature points, give the robot instructions to make the end effector touch the corresponding feature corner point. According to the pose data of the end point when the end effector pokes the checkerboard feature point, the position information of the feature corner point in the robot coordinate system can be collected.

[0077] Based on the position information data of the feature corner points in the robot coordinate system and the camera coordinate system and The transformation matrix relationship can be constructed:

[0078]

[0079] The above matrix is ​​solved using the SVD eigenvalue decomposition method to obtain the hand-eye calibration matrix

[0080] First calculate the center of mass of the point set in the robot coordinate system and the camera coordinate system:

[0081]

[0082] Subtract the corresponding centroid from each point to get the point set after removing the centroid:

[0083]

[0084] Construct a matrix Its form is:

[0085]

[0086] For the matrix Perform singular value decomposition:

[0087]

[0088] Calculate the rotation matrix R and translation vector t through U and V:

[0089]

[0090] Through the above steps, we obtain the rotation matrix R and translation vector t from the robot coordinate system to the camera coordinate system. Using these transformation parameters, points in the robot coordinate system can be transformed into the camera coordinate system:

[0091] P c = RP b + t.

[0092] Compared with the traditional hand-eye calibration method, the steps S23 designed in this embodiment can calibrate the robot and camera coordinate systems under the condition of mounting the end effector, without the need to mount the checkerboard on the robot flange and frequently replace the end effector and checkerboard.

[0093] The calculation process of the transformation matrix of the relative position between the end effector and the Charuco marker code in step S3 is as follows:

[0094] As Figure 5 shown, the Charuco marker code is composed of four Aruco marker codes and five pure black grids combined alternately. This kind of marker combines the geometric structure of the traditional checkerboard and the coding characteristics of the Aruco marker. Its advantage is that it will not be unable to be recognized due to the occlusion of a certain marker code, and it can provide more accurate corner recognition in a complex environment. Moreover, the color contrast at the intersection position of the Aruco marker code and the black grid is obvious, which can be better captured by the camera, effectively improving the corner recognition and accuracy, and thus ensuring the position accuracy of the end effector;

[0095] Referring to Figure 1 , because the Charuco marker code is mounted on the connecting piece of the end effector, that is, their relative position relationship is fixed. As long as the transformation matrix relationship between the marker code and the end point of the end effector can be solved, the position information of the end point in the camera coordinate system can be obtained by the way of the camera recognizing the marker code;

[0096] Write Charuco corner point recognition code. Specifically, the edge algorithm and triangulation method are used to identify and detect corner points, and obtain the pixel coordinates of the four corner points of Charuco under the left and right visual cameras. First, the pixel coordinates of the four corner points are averaged, and the intrinsic parameter matrix obtained by calibrating the left and right pixel coordinates of the average value is calculated by the PnP algorithm to obtain the three-dimensional point coordinate information of the average feature corner point in the camera coordinate system.

[0097] Combine the 3D point coordinate information of the average feature corner point of the Charcuo marker code with the hand-eye calibration conversion matrix obtained by step S2. The specific calculation method is to multiply the coordinate information of the point position by the inverse matrix of the hand-eye calibration conversion matrix to obtain the 3D point coordinate information of the average feature corner point of the Charcuo marker code in the robot coordinate system, and compare the position information of the marker code and the end point in the robot coordinate system. Different from the previous hand-eye calibration, the points collected this time are multiple random points in space, not just the point information on the chessboard, so that the conversion matrix between the end point and the marker code is more accurate.

[0098] The random marker code position information obtained above is matched one by one with the recorded position of the end effector, and the conversion matrix between the Charuco marker code and the end effector is calculated. Given a known transformation matrix In this case, we can use the Charuco marker code to infer the position information of the end point of the robot's end effector; next, in order to meet the requirements of real-time monitoring, a single image recognition corner point information can no longer meet the needs, so it is necessary to write code in the camera's video stream to detect corner point change information in real time and perform corner point recognition on each frame of the image, thereby realizing real-time tracking of the marker code.

[0099] like Figure 6 As shown, the four corner points of the marker code can be clearly identified and tracked in the camera's field of view, and the position information of the end point can be obtained through the position of the marker code.

[0100] Step S4: The camera identifies the corner points of the Charuco marker code, tracks the corner points of the marker code in real time, and calculates the position information of the corner points in the camera coordinate system, thereby inferring the position information of the end point of the robot end effector in the camera coordinate system;

[0101] like Figure 7 As shown, the end point position on the mobile robot is recorded and its position information is compared with the end point position obtained by identifying the Charuco marker code and solving it with the camera. It can be seen that the recognition accuracy can be maintained within 1mm on average, which effectively meets the drilling requirements.

[0102] Step S5: Obtain the image features of the marker balls on the target object through a camera, calculate the coordinate positions of the marker balls using the triangulation method, and improve the positioning accuracy and robustness through the cooperation of multiple marker balls.

[0103] As a preferred solution of this embodiment, in step S5, the process of obtaining the image features of the marker balls on the target object through a camera, calculating the coordinate positions of the marker balls using the triangulation method, and improving the positioning accuracy and robustness through the cooperation of multiple marker balls is as follows:

[0104] As Figure 8 shown, this embodiment preferably uses red marker balls with obvious visual features, arranges them at specific positions on the target object, and ensures that the features of each marker ball have sufficient distinctiveness;

[0105] Capture the environmental image containing the marker balls through a camera, identify the edges, positions, and directions of the marker balls through an edge detection algorithm, and classify them according to the features of the marker balls.

[0106] Utilize the known internal and external camera parameters to convert the coordinates of the marker balls in the two-dimensional image into positions in the three-dimensional space using the triangulation method. Calculate the coordinate positions and postures of each marker ball in the three-dimensional space. At this time, the system improves the positioning accuracy and robustness through the cooperation of multiple marker balls.

[0107] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is used to implement one or more instructions, specifically used to load and execute one or more instructions in the computer storage medium to implement the above method.

[0108] It should be further noted that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and when the computer program is run by a processor, it executes the above-mentioned method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0109] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0110] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

[0111] The present invention is not limited to the above-mentioned optimal implementation manner. Anyone can derive various other forms of a method for positioning the relative position between the end effector of a drilling robot based on machine vision and a target object under the inspiration of the present invention. All equivalent changes and modifications made within the scope of the patent application of the present invention shall fall within the scope covered by the present invention.

Claims

1. A method for relative positioning between an end effector of a drilling robot and a target object based on machine vision, characterized in that: The end effector is connected to the end of the mechanical arm of the drilling robot; The Charuco marker is mounted on the end effector and fixed relative to the end effector, and is used as a marking tool for the camera to identify and track the end point of the end effector; The binocular vision camera is used to track and monitor the Charuco marking code in real time; Positioning methods include: Hang the Zhang Zhengyou calibration plate at the end of the robotic arm, calibrate the binocular vision camera, and obtain the camera's internal parameters; Place the Zhang Zhengyou calibration plate on a plane, control the end effector to reach several fixed feature corner points on the calibration plate, collect the calibrated image data, and record the feature corner data; perform hand-eye calibration calculation based on the acquired image data and feature corner data to obtain the hand-eye calibration conversion matrix; By recording the different positions of the Charuco marker code in the camera coordinate system and converting the calculation results of the hand-eye calibration into the position in the robot coordinate system, the positioning result is obtained by calculating the transformation matrix of the relative position of the end effector and the Charuco marker code based on the data of the end effector at different positions.

2. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 1, characterized in that: For the calibration of the end effector, a four-point calibration method is used to obtain the position and posture relationship matrix of the end effector with respect to the robot coordinate system.

3. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 1, characterized in that: The specific process of the hand-eye calibration calculation includes: Adjust the posture of the end effector to be vertically downward and keep it unchanged, and collect the left and right view images of Zhang Zhengyou's calibration plate in the binocular vision camera; Control the robot arm to move toward the target calibration plate, and make the end point of the end effector reach the characteristic corner point position of Zhang Zhengyou calibration plate, use the end point of the end effector to touch the preset n characteristic point positions of Zhang Zhengyou calibration plate in a certain order, and record the n spatial coordinates of these characteristic points in the robot base coordinate system Hand-eye calibration calculation is performed based on the collected image of the Zhang Zhengyou calibration plate and the feature point data of the Zhang Zhengyou calibration plate.

4. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 3 is characterized in that: The process of performing hand-eye calibration calculation based on the collected image of the Zhang Zhengyou calibration plate and the feature point data of the Zhang Zhengyou calibration plate includes: Perform corner point detection on the image data of Zhang Zhengyou's calibration plate, and obtain the pixel coordinates of the preset n feature corner points under the left and right visual cameras and The pixel coordinates of the feature corner points under the left and right view cameras are matched one by one, and the intrinsic parameter matrix obtained by the joint camera calibration is calculated by the PnP algorithm to obtain the three-dimensional point coordinate information of n feature corner points in the camera coordinate system. According to the pose data of the end point when the end point of the end effector touches the feature point of the chessboard, the position information of the feature corner point in the robot coordinate system is obtained. Based on the position information data of the feature corner points in the robot coordinate system and the camera coordinate system and Construct the transformation matrix relationship: The SVD eigenvalue decomposition method is used to solve the matrix and obtain the hand-eye calibration matrix Pair Matrix Perform singular value decomposition: Calculate the rotation matrix R and translation vector t from U and V: Among them C b and C c is the centroid of the point set in the robot coordinate system and the camera coordinate system.

5. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 4, characterized in that: The conversion matrix calculation process for the relative position of the end effector and the Charuco marking code includes: Through Charuco corner point recognition, the pixel coordinates of the four Charuco corner points under the left and right visual cameras are obtained; The pixel coordinates of the four corner points are averaged, and the left and right pixel coordinates of the average value are combined with the intrinsic parameter matrix obtained by camera calibration to calculate the PnP algorithm to obtain the three-dimensional point coordinate information of the average feature corner point in the camera coordinate system; Combine the 3D point coordinate information of the average feature corner point of the Charcuo marker code with the hand-eye calibration conversion matrix to obtain the 3D point coordinate information of the average feature corner point of the Charcuo marker code in the robot coordinate system; One-to-one correspondence with the recorded position of the end effector, calculate the conversion matrix between the Charuco marker code and the end effector 6. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 5, characterized in that: Given a known transformation matrix In this case, the corner points of the Charuco marker code are tracked in real time, and the position information of the corner points in the camera coordinate system is solved, so as to infer the position information of the end point of the robot end effector in the camera coordinate system.

7. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 1, characterized in that: At least three feature marker balls that are not in the same plane are set on the target object; the orientation setting of the binocular vision camera should meet the requirements of real-time tracking and monitoring of the Charuco marker code and obtaining the position of the feature marker ball at the same time; The image features of the marker ball on the target object are acquired by the camera in real time, and the coordinate position of the marker ball is calculated using triangulation. The accuracy and robustness of positioning are improved by coordinating multiple marker balls.

8. The method for relative positioning between the end effector of a drilling robot and a target object based on machine vision according to claim 7, characterized in that: The method of acquiring the image features of the marker ball on the target object in real time through the camera and calculating the coordinate position of the marker ball by using the triangulation method is specifically as follows: capturing the environment image containing the marker ball through the camera, identifying the edge, position and direction of the marker ball through the edge detection algorithm, and classifying the marker ball according to the features of the marker ball; using the known camera intrinsics and extrinsics, converting the coordinates of the marker ball in the two-dimensional image into the position in the three-dimensional space through the triangulation method, and calculating the coordinate position and posture of each marker ball in the three-dimensional space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for relative positioning of a drilling robot end effector and a target object based on machine vision are implemented as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for relative positioning of a drilling robot end effector and a target object based on machine vision as described in any one of claims 1 to 8.

Images