Control device and method for a robot, robot

Abstract

The present disclosure provides a robot control device and method, and a robot. The robot control device comprises: an acquisition module configured to acquire shape data of a target object by using a navigation system of the robot; a processing module configured to process the target object by using a processing system of the robot according to the shape data and a position to be drilled on the target object, so as to form an implantation hole on the target object; and an implantation module configured to implant an implantation tool into the target object through the implantation hole by using an implantation system of the robot. The present disclosure can solve the problem that poor repeatability of manual operation leads to difficulty in achieving precise implantation, and can achieve precise implantation. Moreover, the implantation process is repeatable, and stable implantation effect can be ensured.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more particularly to a robot control device and method, and a robot. Background Technology

[0002] With the development of robotics technology, in scenarios such as workpiece processing and medical operations, it is often necessary to accurately implant parts, medical devices, etc. into specific locations inside objects.

[0003] In related technologies, such implantation procedures often rely on manual micromanipulation, mechanical drilling, and experience-based alignment. However, manual operations have poor repeatability, and the implantation effect depends entirely on the operator's experience. Especially in the process of micro-hole opening and implantation of precision devices, it is difficult to ensure the consistency of operations between different experiments and different operators, making it difficult to achieve precise implantation. Summary of the Invention

[0004] This disclosure provides a robot control device and method, as well as a robot, to at least solve the problem of poor repeatability in manual operations in related technologies, which makes precise implantation difficult. The technical solution of this disclosure is as follows: According to a first aspect of this disclosure, a robot control device is provided, the control device comprising: a data acquisition module for acquiring shape data of a target object using the robot's navigation system; a processing module for processing the target object using the robot's processing system based on the shape data and the position of the hole to be drilled on the target object, thereby forming an implantation hole on the target object; and an implantation module for implanting an implantation tool into the target object through the implantation hole using the robot's implantation system.

[0005] According to a second aspect of this disclosure, a robot is provided, the robot comprising: a control device according to an embodiment of this disclosure.

[0006] According to a third aspect of this disclosure, a robot control method is provided, the control method comprising: acquiring shape data of a target object using the robot's navigation system; processing the target object using the robot's processing system based on the shape data and the position of a hole to be drilled on the target object to form an implantation hole; and implanting an implantation tool into the target object through the implantation hole using the robot's implantation system.

[0007] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a robot control method according to embodiments of this disclosure.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement a robot control method according to embodiments of this disclosure.

[0009] The technical solution provided in this disclosure brings at least the following beneficial effects: According to the robot control scheme disclosed herein, the robot's navigation system can be used to collect the shape data of the target object, and based on the shape data and the position of the hole to be opened on the target object, the robot's processing system can be used to process the target object to form an implantation hole. Thus, the robot's implantation system can be used to implant the implantation tool into the target object through the implantation hole. This implantation process can be automatically realized by the robot, without relying on human experience, and can achieve precise implantation. Moreover, the implantation process is repeatable, ensuring a stable implantation effect.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0012] Figure 1 This is a schematic block diagram of a robot control device according to exemplary embodiments of the present disclosure.

[0013] Figure 2A and Figure 2B These are schematic structural diagrams of a robot from two angles according to exemplary embodiments of the present disclosure.

[0014] Figure 3 This is a schematic flowchart illustrating the calibration module performing position calibration according to an exemplary embodiment of the present disclosure.

[0015] Figure 4 This is a schematic flowchart illustrating the processing of an implantation hole by a processing module according to an exemplary embodiment of the present disclosure.

[0016] Figure 5 This is a schematic flowchart of the implantation process in the implantation planning mode according to exemplary embodiments of the present disclosure.

[0017] Figure 6 This is a schematic flowchart of the implantation process under the pre-implantation planning mode according to exemplary embodiments of the present disclosure.

[0018] Figure 7This is a schematic flowchart of a robot control method according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0019] In order to enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0020] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0021] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0022] As mentioned earlier, in related technologies, the poor repeatability of manual operations makes it difficult to achieve precise implantation.

[0023] Taking medical implantation as an example, in brain-computer interfaces, neuroscience experiments, and minimally invasive neurosurgical implantation, it is often necessary to accurately implant flexible brain electrodes, neural probes, micro-optical fibers, electrochemical sensors, or other flexible devices into specific brain regions.

[0024] However, traditional implantation methods rely on manual micromanipulation, mechanical drilling, and experiential alignment. The surgical process depends on the operator's experience, especially in the processes of opening micropores in the skull of small animals, puncturing the meninges, and implanting flexible electrodes. It is difficult to ensure consistency between different experiments.

[0025] Furthermore, even if traditional systems can complete intraoperative registration, they are often only used for positioning or path planning, and the registration results are not directly used for laser focus control, opening termination judgment, and subsequent implantation posture planning. Intraoperative navigation and processing lack a closed-loop connection.

[0026] Furthermore, in traditional implantation methods, it is difficult to reliably determine the depth of laser perforation and the state of penetration. Since there are differences in skull thickness, curvature, and local tissue condition, if processing is carried out with fixed power and fixed time, insufficient perforation or excessive ablation may occur, which may affect the safety of the meninges and brain tissue.

[0027] Furthermore, traditional implantation methods lack the ability to quickly plan intraoperatively without preoperative imaging. In some small animal experiments or rapid verification scenarios, there may be no magnetic resonance imaging (MRI) or computed tomography (CT) image data, making it difficult for existing methods to visually select points during surgery and convert them into three-dimensional processing points.

[0028] Therefore, a surgical robot system is needed that can unify coordinate calibration, intraoperative 3D reconstruction, laser closed-loop opening, and low-torsion implantation of flexible electrodes. Similarly, the same problems exist in industrial processing and assembly, especially in the machining of precision workpieces.

[0029] In view of the above problems, exemplary embodiments of this disclosure provide a robot control device, a robot, a robot control method, a computer-readable storage medium, and a computer program product that can solve or at least alleviate the above problems.

[0030] For ease of description, the concepts used in this article will first be explained: Brain-Computer Interface (BCI): A direct connection created between the human or animal brain and an external device, enabling information exchange between the brain and the device.

[0031] Tool Center Point (TCP): The actual point of action of the end effector of a robotic arm, such as the tip of an implanted needle, the electrode gripping point, or the laser focus.

[0032] Iterative Closest Point (ICP) algorithm: used for registration between point clouds or between point clouds and 3D models.

[0033] Fast Point Feature Histograms (FPFH): Used for describing local geometric features in coarse registration of point clouds.

[0034] PnP (Perspective-n-Point): A method for solving camera pose by mapping 3D points to 2D image points.

[0035] Hand-eye calibration: The process of solving for fixed external parameters between the camera coordinate system and the end effector coordinate system of the robotic arm.

[0036] Coaxial monitoring camera: A camera arranged coaxially or nearly coaxially with the laser processing optical path, used to observe the processing point, light spot, hole area status and processing termination characteristics.

[0037] Minimum rotational attitude planning: Under the constraint of keeping the tool axis aligned with the target direction, select the rotation angle around the axis that minimizes the change in end-effector attitude.

[0038] The robot control scheme and robot according to the embodiments of this disclosure relate to the fields of medical robots, computer-aided surgery and brain-computer interface devices, such as an automated surgical robot system for neural electrodes, flexible brain-computer interface devices, micro-implantable devices, etc.

[0039] Specifically, embodiments of this disclosure provide an integrated system that combines intraoperative 3D navigation, laser micro-opening craniotomy, flexible device posture alignment, and robot implantation. This system can achieve a closed-loop automated process from preoperative planning or intraoperative point selection to laser opening and then to flexible electrode implantation through global coordinate reference, laser in-situ physical calibration points, multi-view 3D point cloud reconstruction, registration of preoperative images and intraoperative surfaces, coaxial visual feedback, and minimum rotational posture planning.

[0040] In a first aspect of an exemplary embodiment of this disclosure, a control device for a robot is provided.

[0041] The robot control device according to the exemplary embodiments of this disclosure can be applied to control surgical robots for surgical implantation and / or industrial robots for industrial operations. For example, it can control surgical robots to implant brain-computer interface devices, micro-implantable devices, etc.; or, it can control industrial robots to implant one workpiece into another workpiece during industrial processing and assembly.

[0042] like Figure 1 As shown, the robot control device 100 according to an embodiment of the present disclosure may include a data acquisition module 110, a processing module 120, and an implantation module 130.

[0043] The acquisition module 110 can be used to acquire shape data of a target object using the navigation system 210 of the robot 200.

[0044] The processing module 120 can be used to process the target object according to the shape data and the position of the hole to be opened on the target object, using the robot's processing system 220, so as to form an implantation hole on the target object.

[0045] The implantation module 130 can be used to implant an implantation tool into a target object through an implantation hole using the robot's implantation system 230.

[0046] As an example, Figure 2A and Figure 2B Example structures of a robot from different perspectives are shown according to embodiments of this disclosure.

[0047] like Figure 2A and Figure 2B As shown, the robot 200 may include a navigation system 210, a processing system 220, and an implantation system 230. The navigation system 210 can be used to collect shape data of the target object S, the processing system 220 can be used to process the target object S to form an implantation hole on the target object, and the implantation system 230 can be used to implant an implantation tool into the target object through the implantation hole.

[0048] Here, the target object S may include, but is not limited to, an animal head, tissue surface, workpiece, etc.

[0049] As an example, robot 200 may also include a global reference platform 240 and an object fixing platform 250.

[0050] Here, the global reference platform 240 can serve as the base or pedestal of the robot 200, supporting the navigation system 210, the processing system 220, the implantation system 230, and the object fixing platform 250.

[0051] The global reference platform 240 may include a three-axis macro-motion slide table. During the control process of the control device, a world coordinate system {W} can be established based on the global reference platform 240. For example, the world coordinate system {W} may include an X-axis, a Y-axis, and a Z-axis. The X-axis and Z-axis can be used to adjust the position of the machining components in the machining system 220, and the Y-axis can be used to switch the object fixing platform 250 between the navigation station, the laser station, and the implantation station, which will be described in detail below. For the world coordinate system {W}, if a certain axis does not exist or is fixed, the corresponding transformation can be set as an identity or a constant. In the embodiments of this disclosure, the slide table refers to a system composed of a movable machining component and a movable object fixing platform. The position of the slide table can be represented by the position of the movable machining component on the X-axis and Z-axis and the position of the object fixing platform on the Y-axis. For example, the slide table is in position... Indicates: Fabrication components such as galvanometers in the world coordinate system The coordinates below are The object is fixed in the world coordinate system. The coordinates below are ),in, Indicates the machining components in the world coordinate system The Y-axis coordinate is fixed. and The object is fixed in the world coordinate system. The X-axis and Z-axis coordinates are fixed.

[0052] The object fixing platform 250 can be used to hold the target object to be processed or operated, and the target object can be fixed on the object fixing platform 250.

[0053] Furthermore, the object fixing platform 250 can be movable relative to the global reference platform 240, for example, it can move between navigation stations, machining stations, and implantation stations. Figure 2A and Figure 2B As shown, the object fixing platform 250 can move along... Figure 2A and Figure 2B It moves along the Y-axis to switch between the navigation station, laser station, and implantation station.

[0054] During the control process of the control device, the coordinate system of the object fixing platform can be defined based on the object fixing platform 250 or the stereo positioning instrument. The target object, such as an experimental animal, or the calibration component or sample, such as a calibration plate, remains fixed relative to the object fixation platform 250 during implantation. The target object, calibration component, or sample can move together with the object fixation platform 250 or be driven by the object fixation platform 250.

[0055] The navigation system 210 may include a navigation robotic arm 211 and a navigation camera 212 mounted on the navigation robotic arm 211. The navigation camera 212 can be used to collect shape data of the target object S, and the navigation robotic arm 211 can be used to drive the navigation camera 212 to move.

[0056] Here, the navigation system 210 can be set above the global reference platform 240 and the object fixing platform 250 to collect image data of the target object, calibration parts, etc., fixed on the object fixing platform 250.

[0057] by Figure 2A and Figure 2B For example, the navigation system 210 can be connected to the first support 260, and the base of the navigation robotic arm 211 can be fixed to the first support 260. The end of the navigation robotic arm 211 can be in the vertical direction (e.g.) Figure 2A The end of the navigation robot 211 is closer to the object fixing platform 250 relative to the base of the navigation robot 211 in the Z-axis direction. For example, when the object fixing platform 250 is located at the navigation station, the end of the navigation robot 211 can be located directly above the object fixing platform 250.

[0058] As an example, the navigation camera 212 can be mounted on the end of the navigation robotic arm 211. The navigation camera 212 can be a 3D sensing device such as a structured light camera, but it is not limited to this. The navigation camera 212 can also be any device that can output a 3D point cloud or depth map, such as a ToF camera, a binocular stereo vision camera, or a laser line scan profiler.

[0059] During the control process of the control device, the navigation camera coordinate system {SC} can be defined based on the navigation camera 212, the navigation arm base coordinate system {B1} can be defined based on the base of the navigation robotic arm 211, and the navigation arm end coordinate system {E1} can be defined based on the end of the navigation robotic arm 211 (e.g., the end flange).

[0060] As an example, the navigation system 210 described above can be used to acquire multi-view 3D point clouds and complete intraoperative reconstruction (which will be described in detail below).

[0061] The processing system 220 may include a monitoring camera 221 and a processing component 222. The processing component 222 can be used to perform processing operations on the processing station, and the monitoring camera 221 can be used to acquire image data of the processing station.

[0062] The machining component 222 can be movable, for example, the object fixing platform can be in a first direction (e.g., Figure 2A The machining component 222 can move upwards along the Y-axis, and can move in a second direction and a third direction that are perpendicular to the first direction, respectively. The second direction and the third direction are perpendicular to each other, for example, respectively. Figure 2A The X and Z axes. For example, the machining component 222 can be carried by a drive device (not shown) that can drive the machining component 222 to move in two directions perpendicular to each other. The drive device can be implemented by, for example, existing linear actuators and / or other drive mechanisms. The embodiments of this disclosure do not particularly limit the implementation of the drive device.

[0063] As an example, the processing component 222 can utilize a laser to perform processing operations. For instance, the processing system 220 can be a laser processing system that uses a laser for etching and / or drilling. For example, the processing system 220 may include a laser 223, and the processing component 222 may be a galvanometer module. The laser 223 may be, for example, including but not limited to, an ultraviolet laser or other lasers suitable for bone tissue micromachining. Depending on the processing object, the laser 223 can be any laser source suitable for micromachining bone tissue, hard tissue, or thin film materials. The monitoring camera 221 can be a coaxial monitoring camera, which can be fixedly connected to the processing component 222. The monitoring camera 221 can move synchronously when the processing component 222 moves. For example, the monitoring camera 221 and the processing component 222 can be carried and driven by the same drive device.

[0064] Here, the machining system 220 can be set above the global reference platform 240 and the object fixing platform 250 to perform machining operations on the target object, calibration part, etc., fixed on the object fixing platform 250.

[0065] by Figure 2A and Figure 2B For example, the machining system 220 can be connected to the second support 270, for instance, it can be fixed to the second support 270, and the machining assembly 222 can be in a horizontal direction perpendicular to the direction of movement of the object fixing platform 250 (e.g., Figure 2A (X-axis direction) and vertical direction (e.g.) Figure 2A The machining component 222 can be located directly above the object fixing platform 250 when the object fixing platform 250 is located at the machining station.

[0066] During the control process of the control device, the coordinate system {LS} of the processing component can be defined based on the processing component 222 such as the laser galvanometer, and the coordinate system {CC} of the monitoring camera can be defined based on the monitoring camera 221.

[0067] As an example, the aforementioned processing system 220 can be used for calibration point generation, skull micropore opening, processing status observation, and closed-loop termination determination (which will be described in detail below).

[0068] The implantation system 230 may include an implantation robotic arm 231, and an implantation tool (or implantation actuator) P may be located at the end of the implantation robotic arm 231. The implantation robotic arm 231 may be used to drive the implantation tool P to move.

[0069] Here, the implantation system 230 can be set above the global base platform 240 and the object fixing platform 250 to perform implantation on the target object fixed on the object fixing platform 250.

[0070] by Figure 2A and Figure 2B For example, the implantation system 230 can be connected to the first support 260, and the base of the implantation robotic arm 231 can be fixed to the first support 260. The end of the implantation robotic arm 231 can be in the vertical direction (e.g.) Figure 2A The end of the implantable robotic arm 231 is closer to the object fixation platform 250 in the Z-axis direction relative to the base of the implantable robotic arm 231. For example, when the object fixation platform 250 is located at the implantation station, the end of the implantable robotic arm 231 can be located directly above the object fixation platform 250.

[0071] As an example, the implantable robotic arm 231 may also be provided with at least one of a micro-motion platform, a voice coil motor, a linear actuator or other precision propulsion mechanism, an implantation needle, an electrode clamping mechanism or a flexible device holder, and a binocular microscopic vision module.

[0072] Furthermore, the navigation robotic arm 211 and the implanted robotic arm 231 can move in the direction of motion of the object fixing platform 250 (e.g., Figure 2A Located on both sides of the first bracket 260 on the Y-axis.

[0073] During the control process of the control device, the implantation arm base coordinate system {B2} can be defined based on the base of the implantation robotic arm 231, the implantation arm end coordinate system {E2} can be defined based on the end of the implantation robotic arm 231 (e.g., the end flange), and the implantation application end (or needle tip) coordinate system {A} can be defined based on the implantation tool (or implantation actuator).

[0074] As an example, the implantation system 230 described above can be used to align the implantation needle or flexible electrode with the laser micropore and complete the implantation in the planned direction (which will be described in detail below).

[0075] In some embodiments, the various structures in the robot 200 can be modularly mounted on the global reference platform 240. The global reference platform 240 may include a welded frame, a cast base, or a profile frame, on which a mounting reference surface, linear guide rails, and a protective cover are provided to support the navigation system 210, the machining system 220, the implantation system 230, and the object fixing platform 250. The bottom of the global reference platform 240 may be equipped with adjustable legs, casters, and vibration damping components to enable equipment movement, leveling, and vibration reduction.

[0076] The object fixing platform 250 is movably mounted on a linear guide rail extending along the Y-axis via a slider and is driven by a servo motor via a ball screw, synchronous belt, or linear motor, thereby moving between the navigation station, machining station, and implantation station. Fixtures, stereoscopic locators, animal head restrainers, or workpiece clamping mechanisms can be installed on the object fixing platform 250 to restrict the movement of the target object S relative to the object fixing platform 250. The object fixing platform 250 can also be configured with encoders and limit switches to detect its position and limit its range of motion.

[0077] The navigation robotic arm 211 and the implantation robotic arm 231 can each be a series multi-degree-of-freedom robotic arm with multiple arm bodies and multiple rotary joints. Each rotary joint can include a servo motor, a reducer, an encoder, and a brake. The bases of the navigation robotic arm 211 and the implantation robotic arm 231 can be fixed to different sides of the first bracket 260 by bolts, locating pins, or mounting plates. The navigation camera 212 can be mounted on the end flange of the navigation robotic arm 211 via a camera bracket and communicates with the control device via Ethernet, USB, or a dedicated camera interface. The implantation tool P can be set at the end of the implantation robotic arm 231 via a quick-connect coupling, a gripper, or a tool mount. The implantation tool P can also be connected to a linear actuator, a micro-motion platform, or a voice coil motor to perform precision feed along the implantation direction.

[0078] The processing system 220 can be mounted on the second support 270. The processing component 222 can be mounted on a cross slide consisting of an X-axis linear motion module and a Z-axis linear motion module, and driven by a servo motor, ball screw, or linear motor. The laser 223 can be fixed to the second support 270 or an independent mounting platform, and transmits the laser to the processing component 222 via optical fiber, a reflector group, or a light guide assembly. The processing component 222 may include a galvanometer, a focusing lens, and a protective lens. The monitoring camera 221 can be connected to the processing component 222 via a rigid mounting base and arranged coaxially or nearly coaxially with the laser optical path; a beam splitter can be installed between the monitoring camera 221 and the laser optical path to allow the monitoring camera 221 to acquire images of the processing area.

[0079] Furthermore, although the above reference Figure 2A and Figure 2B An example structure of a robot is shown, but the embodiments of this disclosure are not limited thereto. The control schemes of the embodiments of this disclosure can also be applied to robots of other structures and models. For example, the positional relationship and specific structure of the navigation system, processing system and implantation system can be changed.

[0080] The above combination Figure 2A and Figure 2B Examples of the construction of coordinate systems in the control process of the control device according to embodiments of the present disclosure are described. For ease of description, the coordinate systems and homogeneous transformation conventions used herein are first described.

[0081] Specifically, this article adopts This indicates moving the point from the coordinate system Transform to coordinate system The homogeneous transformation matrix. If coordinate system If the homogeneous point is below, then the coordinate system... The point below , can be represented as: Among them, the homogeneous transformation matrix It can be expressed by the following formula (1): (1) in, Indicates from coordinate system To coordinate system The rotation matrix, Indicates from coordinate system To coordinate system The translation vector.

[0082] Based on the above equation (1), any point is transferred from the coordinate system To coordinate system It can be represented as: ,in, This indicates that the point is in the coordinate system. The coordinates below, This indicates that the point is in the coordinate system. The coordinates below, and All can be represented as In form, This represents the coordinate value in the corresponding coordinate system.

[0083] by Figure 2A and Figure 2B For example, in a simplified embodiment, the object fixing platform can be... Axis translation can be expressed as: ,in, The object is fixed in the world coordinate system. of On-axis movement distance The processing components can be... Axis translation can be expressed as: ,in, Indicates the machining components in the world coordinate system of shaft and The distance moved on the axis is respectively and .

[0084] exist Figure 2A and Figure 2B In a robot system, the coordinate transformation chain may include, for example: , , , .

[0085] It should be noted that the coordinate systems constructed in the embodiments of this disclosure described above can have arbitrary origin positions and coordinate axis directions, as long as each coordinate system can represent the position of its corresponding component or structure. Furthermore, the construction method of the coordinate systems is not limited to the above examples and can be constructed according to actual needs.

[0086] In embodiments of this disclosure, the control device 100 may further include a calibration module 140, which may be used to unify the navigation system, the machining system (e.g., including a monitoring camera and machining components) and the implantation system (e.g., together with an implantation actuator) into the same global coordinate frame.

[0087] As an example, the calibration of calibration module 140 may include camera intrinsic parameters and hand-eye calibration.

[0088] Specifically, navigation cameras can be calibrated (e.g., using the coordinate system mentioned above). The intrinsic and distortion parameters. Fixed extrinsic parameters between the navigation camera and the end effector (e.g., the end flange) of the navigation robot arm can also be obtained through hand-eye calibration (e.g., they can be expressed as...). This transformation can be used to convert point clouds acquired by navigation cameras to the coordinate system at the end of the navigation arm. Similarly, it can be used to calibrate surveillance cameras (e.g., the coordinate system described above). Internal reference and distortion parameters.

[0089] As an example, in an embodiment of this disclosure, a feature marker that serves as a global reference can be produced using a processing system, and the calibration module 140 can calibrate the positional relationship of each system based on such feature marker.

[0090] As an example, the control device also includes a calibration module 140, which is used to: process the robot's calibration parts using the machining system 220 to mark feature marks (such as...) on the calibration parts. Figure 3 Step S310); based on feature markers, position calibration is performed on the navigation system 210, implantation system 230, and processing system 220 (e.g., ...). Figure 3 Step S320 in the process.

[0091] Here, the calibration element can be fixed to the object fixing platform 250 used to fix the target object S. For example, as... Figure 2A and Figure 2B As shown, the calibration component can be a calibration plate fixed on the object fixing platform 250.

[0092] For example, calibration module 140 can use processing system 220, such as a laser processing system, to generate feature markers on a calibration plate fixed to object fixation platform 250, such as an animal platform or surgical platform. These feature markers can include, for example, asymmetrically distributed physical feature points. Such feature markers can be recognized by a navigation camera, such as a structured light camera, contacted by the tip or tool TCP of the implanted robotic arm, and observed by a monitoring camera. In this way, the coordinate relationships of the navigation robotic arm, implanted robotic arm, processing components such as laser galvanometers, object fixation platform, and vision sensors can be unified using the same set of physical feature markers as a common reference.

[0093] Here, in the example of generating asymmetrically distributed feature labels, the accuracy of camera recognition and the robustness of calibration can be improved. Furthermore, as an example, the number of feature labels can be greater than or equal to 6; for example, the number of feature labels can be greater than or equal to 20, thus further improving the robustness of calibration.

[0094] Furthermore, as an example, feature markers may include feature markers at multiple heights and / or feature markers on multiple inclined planes.

[0095] The following description uses processing system 220 as an example of a laser processing system to illustrate the process of generating feature markers.

[0096] Specifically, the calibration module 140 can control the laser emitted by the machining system 220 to be fixed on the object fixing platform (e.g., coordinate system). A set of asymmetric feature markers (e.g., feature points) are ablated in the calibration plate on the ) , where, for the ) These feature markers can record, for example, the processing components of the laser subsystem on the macro-motion platform (or global reference platform). Shaft and / or Initial position on the axis The object is fixed on the macro-motion platform (or global baseline platform). Position on the axis Such as the deflection commands or corresponding physical deflection amounts (or control quantities) of machining components like galvanometers. Here, the deflection amount can refer to the preset zero position (or initial position) relative to the processing component. offset of ) Indicates in Offset on the axis Indicates in Offset on the axis.

[0097] In general, the mapping of machining components such as galvanometers can be represented by the following equation (2): (2) in, and This indicates the control quantity of processing components such as galvanometers. The mapping of physical coordinates to the laser focal plane (or the plane being processed). This indicates the laser focal plane (or the plane being processed) in the coordinate system of the processing component. The height below.

[0098] As an example, if a simplified model is used, the focal plane can be set as... Furthermore, since the above control quantities are approximately consistent with the physical coordinates, we can obtain the following formula (3): (3) Thus, the first The position of a feature marker (e.g., an ablation point) in the fixed platform coordinate system of the object can be represented by the following formula (4): (4) in, Indicates the first The position (e.g., coordinates) of each feature marker in the fixed platform coordinate system of the object. Indicates the fixed platform coordinate system of the object { } to world coordinate system { Position transformation relationships (e.g., coordinate transformation matrix) of} Indicates from the coordinate system of the machining component { } to world coordinate system { Position transformation relationships (e.g., coordinate transformation matrix) of} This indicates the position of the fixed platform of the object relative to the global reference platform (or the robot's reference position) (e.g., the world coordinate system). }of (coordinates on the axis) This indicates the initial position of the machining component relative to the global reference platform (or the robot's reference position) (e.g., in the world coordinate system). }of shaft and (Initial coordinates on the axis).

[0099] Under the simplified model, equation (4) above can be expressed as equation (5) below: (5) The above point set It can be used for calibration of navigation robot arm bases, calibration of implanted robot arm bases, calibration of external parameters of monitoring cameras, and verification of machining accuracy.

[0100] Using the above method, global coordinate unification can be performed based on characteristic markers such as laser in-situ ablation points as a common physical reference. These characteristic markers can be identified by navigation cameras, contacted by application terminals such as needle tips or TCPs implanted in robotic arms, or imaged by monitoring cameras. Furthermore, their theoretical positions can be calculated based on the position of the macro-motion slide and control commands from the processing components. Therefore, such characteristic markers can serve as a common physical reference among multiple subsystems, including navigation, processing, implantation, and vision. Based on these characteristic markers, coordinate systems can be unified through rigid body registration, PnP, or other extrinsic parameter calibration methods.

[0101] Furthermore, although the above embodiments use laser to ablate feature marks on the calibration plate in the field, the embodiments of this disclosure are not limited to this. For example, high-precision calibration blocks, precision groove plates, three-dimensional calibration targets, checkerboard grids, dot arrays, etc. can also be used. As long as the reference can be visually recognized, touched by tools, and / or positioned by laser, it can be used as a global reference.

[0102] The following describes in detail an example process of the calibration module 140 performing position calibration on the navigation system 210, the implantation system 230, and the processing system 220 based on feature tags.

[0103] As an example, the calibration module 140 can be used to: control the object fixing platform to move to a navigation station for data acquisition by the navigation camera; use the navigation camera to acquire first shape data of the feature marks on the calibration component; and calibrate the positional relationship between the navigation robotic arm and the robot's reference position based on the first shape data.

[0104] Here, the reference position can be determined based on a global reference platform, for example, by the origin of the world coordinate system.

[0105] The first shape data can be, for example, 3D point cloud data of feature markers acquired by a navigation camera, which acts as a structured light camera. This first shape data can reflect the position of the feature markers relative to the navigation robot arm at the navigation station. Since the positional relationship between the navigation station and the reference position is determinable, the positional relationship between the navigation robot arm and the robot's reference position can be calibrated based on the positional relationship between the navigation station and the reference position and the first shape data.

[0106] For example, the calibration module 140 can be used to: determine a first position of the feature mark relative to the navigation robot arm when the object fixing platform is in the navigation station based on the first shape data; determine a second position of the feature mark relative to the reference position when the object fixing platform is in the navigation station based on the positional relationship between the navigation station and the reference position; and calibrate the positional relationship between the navigation robot arm and the reference position based on the first position and the second position.

[0107] Specifically, the object-fixed platform can be moved to the navigation station. At the navigation station, the position of the object-fixed platform on the Y-axis of the world coordinate system can be represented as... It can represent the positional relationship between the navigation station and the reference position (i.e., the second position mentioned above).

[0108] When the object is fixed on the platform and in the navigation position, the navigation camera can be used to scan feature markers and extract their three-dimensional center coordinates. Wherein, if the first... A feature marker in the navigation camera coordinate system The coordinates below are represented as Then it can be determined in the coordinate system of the navigation arm base. coordinates below (That is, the first position mentioned above) can be represented as: ,in, Represents the coordinate system from the end of the navigation arm To the navigation arm base coordinate system Position transformation relationships (e.g., coordinate transformation matrix). Indicates from the navigation camera coordinate system To the end coordinate system of the navigation arm Position transformation relationships (e.g., coordinate transformation matrices).

[0109] Furthermore, based on the positional relationship between the navigation station and the reference position, for example... , No. The theoretical position of each feature marker in the world coordinate system (i.e., the second position mentioned above) can be represented as: ,in, Indicates in The navigation station is located in the fixed platform coordinate system of the object. To the world coordinate system Position transformation relationships (e.g., coordinate transformation matrices).

[0110] In the simplified model, the second position mentioned above can be expressed as, for example, by the following equation (6): (6) Based on the point set at the first position mentioned above The set of points at the second position Rigid body registration can be performed to solve the extrinsic parameters of the navigation robot's base relative to the world coordinate system (e.g., the navigation robot's base coordinate system). With world coordinate system The positional transformation relationship between them can be expressed as, for example, as shown in the following formula (7): (7) in, Indicates from the navigation arm base coordinate system To the world coordinate system Position transformation relationship.

[0111] As an example, the rigid body registration method described above can employ, but is not limited to, the Kabsch algorithm, the Umeyama algorithm, or other least-squares rigid body registration methods.

[0112] By using the above method, the positional relationship between the navigation robot and the reference position can be calibrated by determining the first position of the feature marker relative to the navigation robot (e.g., the base of the navigation robot) and the second position of the feature marker relative to the reference position (e.g., the world coordinate system).

[0113] As an example, the calibration module 140 can be used to: control the object fixing platform to move to the implantation station for the implantation robot arm to perform implantation operations; control the implantation robot arm to drive the implantation tool to move so that the application end of the implantation tool contacts the feature mark, and record the pose data of the end of the implantation robot arm when the application end contacts the feature mark; and calibrate the positional relationship between the implantation robot arm and the robot's reference position based on the pose data.

[0114] Here, the application end of the implantation tool can be, for example, the end or tip of the tool that will first enter the target object. In the case of an implantation needle, the application end can be the needle tip. During the process described above, the implantation tool can be controlled to sequentially touch each feature marker, and the pose data of the implantation robot arm's end effector can be recorded at each touch. This pose data reflects the position of the feature marker relative to the implantation robot arm at the implantation station. Since the positional relationship between the implantation station and the reference position is determinable, the positional relationship between the implantation robot arm and the robot's reference position can be calibrated based on the positional relationship between the implantation station and the reference position, as well as the pose data.

[0115] For example, the calibration module 140 can be used to: determine the third position of the feature marker relative to the implanted robotic arm when the feature marker is contacted at the application end based on pose data; determine the fourth position of the feature marker relative to the reference position when the feature marker is contacted at the application end based on the positional relationship between the implantation station and the reference position; and calibrate the positional relationship between the implanted robotic arm and the reference position based on the third and fourth positions.

[0116] Specifically, the object fixing platform can be moved to the implantation station. At the implantation station, the position of the object fixing platform on the Y-axis of the world coordinate system can be represented as... It can represent the positional relationship between the implantation station and the reference position (i.e., the fourth position mentioned above).

[0117] When the object is fixed on the implantation platform, the implantation robot arm can control the application end of the implantation tool, such as the needle tip or TCP, to sequentially contact the feature marker and record the pose of the end flange of the implantation robot arm at each contact.

[0118] For example, if the TCP implantation tool is located in the application's coordinate system... If the origin is defined at the application end, such as the needle tip, then the origin lies in the implant arm base coordinate system. coordinates below (For example, the third position mentioned above) can be represented as:

[0119] in, Represents the coordinate system from the end of the implanted arm To the implant arm base coordinate system Position transformation relationships (e.g., coordinate transformation matrix). Indicates the coordinate system from the implanted application. To the end-effector coordinate system Position transformation relationships (e.g., coordinate transformation matrices), for this The calibration will be described in detail below.

[0120] Furthermore, based on the positional relationship between the implantation station and the reference position, for example , No. The theoretical position of each feature marker in the world coordinate system (i.e., the fourth position mentioned above) can be represented as: ,in, Indicates in The implantation station is located in the fixed platform coordinate system of the object. To the world coordinate system Position transformation relationships (e.g., coordinate transformation matrices).

[0121] Based on the point set at the third position mentioned above and the set of points at the fourth position Rigid body registration can be performed to solve the extrinsic parameters of the implanted robotic arm's base relative to the world coordinate system (e.g., the implanted arm base coordinate system). With world coordinate system The positional transformation relationship between them can be expressed as, for example, as shown in equation (8): (8) in, Indicates the coordinate system from the implant arm base. To the world coordinate system Position transformation relationship.

[0122] By using the above method, the positional relationship between the navigation robot and the reference position can be calibrated by determining the third position of the feature marker relative to the navigation robot (e.g., the base of the navigation robot) and the fourth position of the feature marker relative to the reference position (e.g., the world coordinate system).

[0123] As an example, the calibration module 140 can also be used to calibrate the positional relationship between the application end of the implantation tool and the end of the implantation robotic arm.

[0124] Specifically, the calibration module 140 can be used to determine the fixed positional relationship between the application end, such as the tip of the implantation needle, the electrode clamping point, or the actual implantation point, and the end of the implantation robotic arm (e.g., the end flange), for example, from the implantation application end coordinate system. To the end-effector coordinate system coordinate transformation matrix .

[0125] The calibration process for the positional relationship between the application end of the implantation tool and the end effector of the implantation robotic arm can employ, for example, but not limited to, the four-point method, the sphere center fitting method, the needle tip contact method, or other robot TCP calibration methods. Furthermore, since the positional relationship between the application end of the implantation tool and the end effector of the implantation robotic arm is fixed, any other existing calibration method can also be used to calibrate this positional relationship.

[0126] As an example, the calibration module 140 can be used to: control the object fixing platform to move to the processing station; use a monitoring camera to collect second shape data of feature marks; and calibrate the positional relationship between the monitoring camera and the processing components based on the second shape data.

[0127] Here, the object fixing platform can be moved to the processing station. At the processing station, a monitoring camera can be used to collect second shape data of the feature marker. The second shape data can be, for example, image data of the feature marker collected by monitoring camera 221. This second shape data can reflect the position of the feature marker relative to the processing component at the processing station. Since the positional relationship between the monitoring camera and the processing component can be fixed, the positional relationship between the monitoring camera 221 and the processing component 222 can be calibrated based on the position of the feature marker relative to the processing component 222 and the second shape data.

[0128] For example, calibration module 140 can be used to: determine the position of the feature marker relative to the processing component; and calibrate the positional relationship between the monitoring camera and the processing component based on the position of the feature marker relative to the processing component and the second shape data.

[0129] Specifically, surveillance cameras (e.g., surveillance camera coordinate system) ) and machining components (e.g., machining component coordinate system) This can maintain a rigid relationship, therefore it is necessary to calibrate the positional transformation relationship between the monitoring camera and the processing components, such as the monitoring camera coordinate system. coordinate system with machining components Coordinate transformation matrix between .

[0130] Here, the slide is in position In this case, the coordinate system of the surveillance camera can be determined (e.g., the camera's coordinate system). Relative to the object's fixed platform (e.g., the object's fixed platform coordinate system) The pose of ) can be represented, for example, by the following formula (9): (9) in, Indicates the coordinate system of the surveillance camera To the object's fixed platform coordinate system Position transformation relationship, Represents the fixed platform coordinate system of the object To the world coordinate system Position transformation relationship, Indicates the coordinate system of the machining component To the world coordinate system Position transformation relationship, Indicates the coordinate system of the surveillance camera To the machining component coordinate system Position transformation relationship.

[0131] During calibration, feature markers can be captured using a surveillance camera to obtain second shape data, and the pixel centers of each feature marker can be extracted from the second shape data, for example, as follows: Furthermore, the feature markers in the machining component coordinate system can be calculated based on the slide position and galvanometer commands. The position below, for example, can be represented as three-dimensional coordinates. .

[0132] Based on 3D points Two-dimensional pixels in the second shape data The correspondence can be obtained from the coordinate system of the machining component by methods such as PnP. To the coordinate system of the surveillance camera Position transformation relationships, such as obtaining coordinate transformation matrices. It can be expressed by the following formula (10): (10) in, Represents the perspective projection function. Based on the coordinate transformation matrix. The coordinate transformation matrix can be obtained through matrix inverse transformation. .

[0133] As an example, equation (10) above can also be solved using algorithms such as the Levenberg-Marquardt algorithm. This algorithm can reduce reprojection errors through nonlinear optimization. Alternatively, it can be solved using methods such as RANSAC-PnP to obtain the solution. This algorithm can eliminate the phenomenon of mismatch between three-dimensional points and two-dimensional points.

[0134] In the above manner, the shape data of feature marks can be collected at the processing station by the monitoring camera, and the positional relationship between the monitoring camera and the processing components can be calibrated. This makes the control device adaptable to processing systems of various structures or forms, reducing the requirements of the control scheme on the structural design of the robot.

[0135] In the embodiments of this disclosure, based on the calibration processes of the above-described calibration modules, the control device according to the embodiments of this disclosure can be adapted to robots of different models and structures. For example, the structure or placement of at least one of the robot's robotic arm, structured light camera, ToF camera, binocular camera, laser, galvanometer, and implanted actuator can be adapted to... Figure 2A and Figure 2B As shown, the control device can be migrated to different robots or experimental platforms simply by re-performing the corresponding calibration performed by the above calibration module.

[0136] Furthermore, in the embodiments of this disclosure, the shape data of the acquired target object can also be calibrated.

[0137] As an example, the calibration module 140 can also be used to: control the object fixing platform carrying the target object to move to the navigation station; use the navigation camera to collect the third shape data of the target object; and correct the third shape data based on the existing fusion model.

[0138] Here, the third shape data can be, for example, three-dimensional point cloud data of the target object acquired by a navigation camera that acts as a structured light camera.

[0139] As an example, work can begin by moving an object from a fixed platform to a navigation station, for example, with the Y-axis coordinate in the world coordinate system being... A navigation-guided robotic arm can carry a navigation camera to scan a target object, such as an animal's head or a sample surface, from multiple perspectives to obtain third shape data, which may include, for example, multiple frames of local point cloud data of the target object.

[0140] Here, we are focusing on the first [frame] of local point cloud data from multiple frames. Frame point cloud It can be transformed to the object's fixed platform coordinate system using the following formula (11): (11) in, Indicates the first Frame point cloud Transform to object fixed platform coordinate system The coordinates below, Indicates the position of the target object at the navigation station. Fixed platform coordinate system of the object To the world coordinate system Position transformation relationship, Indicates from the navigation arm base coordinate system To the world coordinate system Position transformation relationship, Represents the coordinate system from the end of the navigation arm To the navigation arm base coordinate system Position transformation relationship, Represents the fixed platform coordinate system of the object To the end coordinate system of the navigation arm Position transformation relationship.

[0141] In the embodiments of this disclosure, considering that errors may exist in the kinematics of the robotic arm, hand-eye calibration, and structured light reconstruction, direct splicing may lead to local misalignment. Therefore, the robot kinematics results can be used as initial values ​​to perform point-to-surface iterative nearest point (ICP) correction on the adjacent viewpoint point cloud or the current point cloud and the existing fusion model.

[0142] As an example, the point-to-surface ICP can be represented by the following equation (12): (12) in, This represents the corresponding point in the existing fusion model. This represents the normal vector of the corresponding point in the existing fusion model.

[0143] The calibration module can be based on the above-mentioned point-to-surface ICP transformation relationship. For the aforementioned third shape data (e.g., point cloud) The transformed point clouds from each frame are then fused to obtain the corrected data. .

[0144] It should be noted that although the above description uses multi-view point cloud fusion as an example, the embodiments disclosed herein are not limited to this. For example, any point cloud stitching method such as TSDF fusion, surfel fusion, or graph-optimized global point cloud stitching can also be used to obtain the corrected data.

[0145] In the embodiments of this disclosure, two implantation planning modes can be compatible during the implantation process: one is to determine the processing position of the hole based on the image data collected before implantation, and the other is to determine the processing position of the hole based on the target position selected by the user during the implantation process.

[0146] As an example, the processing module 120 can be used to: determine the location of the hole to be opened based on pre-acquired image data and shape data; and / or, determine the location of the hole to be opened based on received user instructions.

[0147] Here, taking the medical implantation scenario as an example, the implantation planning mode that determines the location of the opening based on pre-collected image data and shape data is more suitable for scenarios such as large animals, precise targeted implantation, and scenarios that need to avoid blood vessels or important brain regions; the implantation planning mode that determines the location of the opening based on received user instructions is more suitable for scenarios such as small animal experiments, rapid verification, or scenarios without preoperative MRI / CT.

[0148] Specifically, in the implantation planning mode where the location of the opening is determined based on pre-acquired image data and shape data, the processing module 120 can be used to: register the pre-acquired image data and shape data; and determine the location of the opening based on the registration result.

[0149] Here, the pre-acquired image data may include, but is not limited to, MRI image models and / or CT image models. In a surgical scenario, the image data may be, for example, preoperative image data. The shape data may be data collected by the navigation camera at the navigation station. For example, the shape data may include, but is not limited to, point cloud data of the target object obtained by the navigation camera such as a structured light camera or other 3D perception device.

[0150] Furthermore, the registration results of pre-acquired image and shape data can include, for example, the transformation relationship from the image coordinate system to the object fixing platform coordinate system. The processing position of the hole to be drilled can be represented by the implantation position and implantation direction.

[0151] For example, the processing module 120 can combine registration methods such as robotic arm kinematics, hand-eye calibration, multi-view point cloud fusion, FPFH and RANSAC coarse registration, and point-to-surface ICP fine registration to align the pre-acquired image data with the real space in the implantation, and obtain the transformation relationship from the image coordinate system to the object fixed platform coordinate system.

[0152] Specifically, in the pre-implantation imaging planning mode, pre-acquired image data such as MRI, CT or other medical images can be obtained in advance, and a model of the target object such as the skull, brain surface, blood vessels or target brain regions can be reconstructed. During the procedure, point cloud data of the target object's surface is obtained (e.g., corrected data). After that, the point cloud data can be compared with the model. Registration.

[0153] As an example, the point cloud data can be compared with the model in the following ways. Registration: the process of matching point cloud data with a model FPFH features were extracted; coarse registration was performed using the RANSAC method to obtain the initial pose; fine registration was performed using point-to-surface ICP; and the transformation relationship from the image coordinate system to the object's fixed platform coordinate system (e.g., coordinate transformation matrix) was obtained. This transformation relationship can be used as the result of the above registration.

[0154] The calibration module can be based on the registration results (e.g., the coordinate transformation matrix mentioned above). The planned implantation location (e.g., entry point) and implantation direction are transformed from the image coordinate system to the object-fixed platform coordinate system. For example, the implantation location and implantation direction can be represented as follows:

[0155]

[0156] in, This indicates the coordinates of the implantation location in the object's fixed platform coordinate system. Indicates the implantation direction in the fixed platform coordinate system of the object. This indicates the coordinates of the implantation site in the image coordinate system. Indicates the implantation direction in the image coordinate system. This indicates the position transformation relationship from the image coordinate system to the object's fixed platform coordinate system. This represents the rotation matrix that transforms the image coordinate system to the object's fixed platform coordinate system.

[0157] Furthermore, although the above registration example uses the RANSAC method for coarse registration and point-to-surface ICP for fine registration, the embodiments of this disclosure are not limited to this. For example, coarse registration can also use any point cloud registration method such as PCA principal direction registration, ISS, SHOT and other local features, semantic key point registration, etc.; and fine registration can also use any point cloud registration method such as point-to-point ICP, robust kernel function ICP, non-rigid registration, etc.

[0158] The above method enables registration between the pre-implantation image model and the shape data collected by the robot, thereby allowing the robot to perform implantation operations at a pre-selected implantation site.

[0159] In the implantation planning mode where the location of the hole to be opened is determined according to the received user instructions, the processing module 120 can be used to: use a monitoring camera to collect target image data of the target object; and determine the location of the hole to be opened according to the received user instructions.

[0160] Here, user instructions can direct the user to select a target location in the target image data.

[0161] For example, in scenarios where there is no preoperative imagery or where rapid intraoperative planning is required, the user can directly select a target pixel from the target image data acquired by the monitoring camera. The processing module 120 can back-project the target pixel into a spatial ray and find the intersection point with the three-dimensional surface model reconstructed from the target image data such as structured light to obtain the real three-dimensional processing point.

[0162] As an example, the processing module 120 can be used to: determine the processing direction from the processing component toward the target object based on the target location; and determine the location of the hole to be opened based on the shape data and the processing direction.

[0163] Specifically, in scenarios where preoperative imaging is unavailable or rapid intraoperative planning is required, the operator can click on the target location (e.g., the target pixel) in the image from the monitoring camera.

[0164] Here, the pose of the surveillance camera can be calculated. For example, at the current position of the slide table... At that time, the pose of the monitoring camera relative to the fixed platform coordinate system of the object can be represented by the following formula (13): (13) The unit direction of the processing direction in the coordinate system of the monitoring camera can be obtained by back projection based on the intrinsic parameters of the monitoring camera, for example, as shown in the following formula (14): (14) in, This represents the unit vector (or unit direction) of the machining direction in the coordinate system of the monitoring camera. This indicates the target location. As an example, before calculating the unit direction, distortion correction can also be performed on the target pixel clicked by the user. Here, existing methods can be used to perform distortion correction, and the embodiments of this disclosure do not impose any particular limitations on this, and the distortion correction process can also be omitted.

[0165] Furthermore, the optical center of the monitoring camera is in the coordinate system of the object-fixed platform. The lower position It can be represented as: Unit vector in the machining direction It can be represented as: .

[0166] Therefore, based on the above processing direction and the optical center position of the monitoring camera, the spatial ray corresponding to the target position can be determined, for example, by the following formula (15): (15) in, This represents the spatial ray corresponding to the target location. This can be a coefficient to be determined, which can characterize the length of the space ray, where, .

[0167] Given the aforementioned spatial ray, the location of the opening can be determined by the spatial ray and the three-dimensional model reconstructed based on the target image data, including, for example, the implantation location and implantation direction.

[0168] Specifically, let the three-dimensional surface model reconstructed based on the target image data be... In one example, if the model is a triangular mesh, the nearest positive intersection point between the ray and the mesh can be selected, for example, as follows:

[0169]

[0170] in, This represents the actual 3D processing point or implantation location (i.e., the intersection mentioned above). Indicates the determined coefficient .

[0171] In another example, if the model is a point cloud, neighboring points can be searched along a spatial ray, a local plane or quadratic surface can be fitted, and then the intersection of the spatial ray and the fitted surface can be calculated. .

[0172] Using the above method, the robot can be controlled to perform the implantation operation based on the shape data of the target object collected in real time and the implantation location selected by the user in real time, making the implantation process more flexible and adaptable to more implantation scenarios.

[0173] As an example, the processing module 120 can be used to: determine the processing position (e.g., based on the shape data of the target object) where the processing component performs the processing operation on the location of the opening to be made. Figure 4 Step S410); control the machining component to move to the machining position (e.g. Figure 4 Step S420); using a processing component, the target object is processed to form an implantation hole (e.g. Figure 4 Step S430 in the process.

[0174] When the location of the hole to be drilled is determined, the machining component can be moved to the machining location relative to the machining position of the machining component (e.g., the position of the machining point of the hole to be drilled relative to the coordinate system of the machining component) so as to drill a hole in the target object using the machining component.

[0175] For example, suppose the location of the hole to be opened is given. As shown in equation (16): (16) When the slide is in position In this case, the coordinates of the slide in the machining component coordinate system It can be expressed by the following formula (17): (17) Under the simplified translation model, the coordinates of the slide in the machining component coordinate system For example, it can be simplified to the following formula (18): (18) If it is desired that the processing point (e.g., laser focus) of the component satisfies the following formula (19): (19) Then it should satisfy: ; ; .

[0176] Therefore, when selecting the deflection amount of the processing component Then, the target position of the slide. It can be: ; ; .

[0177] As an example, in an example where the processing component is a laser galvanometer, if a galvanometer-centered processing strategy is adopted, i.e. ,and The target position of the slide is... It can be: ; ; .

[0178] Determine the target position of the slide. In this case, the slide can be controlled to move to the target position, so as to drive the processing components to the position where the target object will be processed.

[0179] As an example, in an example where the processing component performs processing operations using a laser, after the processing component moves to the processing position, the processing module 120 can be used to process the target object by executing at least one of the following processing strategies: in response to the location of the hole to be opened being a single point, keeping the laser focus position of the processing component unchanged, and processing the target object by adjusting the position of the processing component; or, in response to the location of the hole to be opened including multiple points or continuous patterns within the same field of view of the monitoring camera 221, keeping the position of the processing component unchanged, and processing the target object by adjusting the laser focus position of the processing component, wherein, when processing the target object by adjusting the laser focus position of the processing component, in response to the laser focus position of the processing component deflecting beyond a predetermined range, keeping the laser focus position of the processing component unchanged, and processing the target object by adjusting the position of the processing component.

[0180] Specifically, during the processing, two processing strategies can be considered: a slide-first processing strategy and a component-first processing strategy, and these two processing strategies can be switched.

[0181] As an example, the slide-priority machining strategy is suitable for single-point high-precision hole drilling. In this strategy, the laser focus position of the machining component remains constant, and the position of the machining component is adjusted first to reach a suitable machining position for processing the target object. For example, in this strategy, the galvanometer can be placed in the center; based on the target machining point (e.g., the target position mentioned above), the target position of the slide on the X, Y, and Z axes of the world coordinate system is calculated; the slide is controlled to move to the target position; visual verification is performed using a monitoring camera to confirm that the slide has reached the correct position; and the machining component is controlled to perform the machining operation. When the machining component is a laser galvanometer, this strategy can reduce galvanometer edge distortion and uneven spot energy.

[0182] As an example, a processing component-priority processing strategy is suitable for multi-point processing or continuous pattern processing within the same field of view. In this strategy, the position of the processing component remains constant, and the laser focus position of the processing component is adjusted to ensure it reaches the location on the target object where a hole is to be drilled, thus facilitating processing. For example, in this strategy, the position of the slide can be adjusted... Under fixed conditions, calculate the target point in the current machining component coordinate system. The lower position For example, it can be expressed by the following formula (20): (20) If position Represented as:

[0183] Then the offset command for the machining component can be solved: and In this way, the offset of the machining component can be obtained. : .

[0184] Here, in response to With focal plane height If the difference exceeds the depth of field range, the processing components can be adjusted. Position on the axis. If the machining component deflects beyond the predetermined range, such as by an offset... If the field of view exceeds the effective field of view of the galvanometer, the machining strategy of prioritizing the slide table described above can be automatically switched.

[0185] Using the above methods, different processing strategies can be selected according to actual needs, and processing strategies can be dynamically switched according to the processing process to achieve more precise processing.

[0186] Furthermore, as an example, in an example where the processing system is a laser processing system, the aforementioned implantation hole can be processed using a laser helical scanning micro-hole aperture method. Specifically, laser micro-hole aperture can employ a helical scanning or multi-turn circular scanning strategy, and the processing status can be judged during the processing (e.g., by using a coaxial monitoring camera) to achieve closed-loop termination.

[0187] In the aforementioned laser aperture process, the micro-hole diameter, hole wall quality, and heat-affected zone can be controlled by adjusting the maximum radius, pitch, scanning speed, laser power, repetition frequency, pulse width, and cooling time. However, the embodiments of this disclosure are not limited to laser helical scanning micro-hole aperture; in addition to helical scanning, the laser aperture trajectory can also employ any scanning method such as multi-turn circular scanning or Lissajous curve scanning.

[0188] Furthermore, in the embodiments of this disclosure, during the processing using methods such as those described above, a closed-loop opening and termination determination mechanism can also be employed to adaptively adjust the processing progress.

[0189] As an example, the processing module 120 can be used to process a target object by iteratively performing the following operations: using a processing component to perform the current processing on the processing location of the target object; using a monitoring camera to acquire an image of the processing location of the target object; based on the image of the processing location, determining whether the current state of the processing location meets the hole-opening termination condition; in response to the current state of the processing location meeting the hole-opening termination condition, stopping the iterative operation; in response to the current state of the processing location not meeting the hole-opening termination condition, using the processing component to perform the next processing on the processing location.

[0190] Specifically, the machining module 120 can calculate the control quantities of the macro-motion slide and machining components based on the position of the machining location (e.g., the target machining point) in the coordinate system of the fixed platform. As an example, a slide-priority machining strategy can be used to center the machining point on the machining component, or a machining component-priority machining strategy can be used to achieve rapid multi-point machining within the same field of view. During machining, a closed-loop control of "intermittent scanning - image acquisition - status judgment - continue or stop" can be performed through a monitoring camera to reduce the risk of over-ablation.

[0191] Here, in each iteration, a processing operation with a preset duration or a preset physical distance / area can be performed (e.g., a processing operation after completing a certain number of circles or a fixed scanning time). After completing this processing operation, an image of the processed position can be acquired using a monitoring camera, and the current state of the processed position in the image can be extracted. Based on the current state of the processed position, it can be determined whether the preset opening termination condition has been met, and whether to continue to perform the next processing operation until the opening termination condition is met, thus forming the final implantation hole.

[0192] The above method enables stepwise drilling. For example, after completing several revolutions or a fixed scanning time, the scanning is paused, and the hole depth is analyzed using images from the coaxial auxiliary camera. If no penetration is detected at the machined location (e.g., no light transmission or signs of critical deformation), the command moves down a small step in the Z-axis direction, refocuses, and continues scanning. This process can be repeated until a penetration signal is detected (e.g., abrupt reflection characteristics or the appearance of a bright spot), confirming hole completion. By employing such an adaptive drilling strategy, excessive ablation can be avoided, improving safety and drilling quality.

[0193] As an example, the current state of the aforementioned processed location can be obtained by identifying the optical features of the processed location, such as, but not limited to, at least one of bright spots, abrupt changes in reflection, edge morphology, light transmission features, aperture variations, and texture variations.

[0194] The above-mentioned hole termination conditions can be set according to actual needs.

[0195] As an example, the preset duration or preset physical distance / area can be set according to actual needs.

[0196] For example, the total time to complete the entire processing process can be estimated. By dividing the total time, multiple preset time periods can be obtained. In each processing session, in response to reaching the preset time period, the processing is paused and the status of the processed position is identified.

[0197] For example, the processing trajectory for forming the implantation hole or the processing area for forming the implantation hole can be determined. Multiple preset physical distances / areas can be obtained by dividing the processing trajectory or processing area. During each processing step, processing can be paused in response to reaching the physical distance / area corresponding to the current processing step, and the status of the processed position can be identified. For example, in the scenario of laser-guided drilling, the preset physical distance can be in the vertical direction (e.g., at...). Figure 2A A preset downward distance (in the Z-axis direction) can be used as a step size to drill downwards in each machining operation, for example, drilling downwards to a depth in the range of 0.05mm to 0.1mm each time.

[0198] Regarding the aforementioned closed-loop opening and termination judgment mechanism, taking the micro-open craniotomy process as an example, the processing module can execute the following closed-loop process: set laser parameters and scanning trajectory; perform a laser scan of a preset distance; pause the laser; monitor the camera to acquire images; preprocess the images of the opening area; extract optical features; determine whether the opening termination condition has been met; if not, adjust the laser focal plane or continue scanning; if met, stop the laser and record the opening status.

[0199] As an example, in the above process, the processing module can control the processing component to descend along the Z-axis in preset steps when no penetration is detected, such as descending to a depth of 0.05mm to 0.1mm each time, and can continue scanning after refocusing.

[0200] Furthermore, as an example, in an embodiment of this disclosure, before the processing laser is officially turned on, the processing module can also be used to control the monitoring camera to perform pre-processing verification. The pre-processing verification may include visual verification, and the position of the processing components or slide can be fine-tuned by using a low-power indicator based on the result of the visual verification.

[0201] Specifically, at the current position of the slide is In this case, the theoretical projection position of the target location in the image of the target object captured by the surveillance camera. It can be represented as: ,in, .

[0202] As an example, the machining module can perform pre-machining verification by comparing the theoretical projection positions mentioned above. If the deviation between the user-selected point or the low-power indicator spot is below a preset threshold, the current processing operation can be allowed; if the deviation exceeds the preset threshold, the position of the processing component or slide can be fine-tuned, and visual verification can be performed again after fine-tuning until the deviation is below the aforementioned preset threshold.

[0203] Furthermore, as described above, embodiments of this disclosure can provide two implantation planning modes, and an example process for determining the implantation location and direction in each implantation planning mode is described. The implantation module 130 can control the implantation robotic arm to perform the implantation operation based on the determined implantation location and direction.

[0204] As an example, the implantation module 130 can be used to: determine the target pose of the implantation tool based on the current pose of the implantation tool, such that the optical axis of the implantation tool is aligned with the implantation direction in the target pose and the rotation angle between the target pose and the current pose is minimized; determine the target pose of the implantation tool based on the position of the implantation hole and the target pose; and control the implantation robotic arm to move the implantation tool to the target pose.

[0205] For example, the location of the implantation hole (or implantation site) can be represented as: The implantation direction at this location can be represented as .

[0206] As mentioned above, in the pre-implantation image planning mode, the implantation location in the image coordinate system can be determined. and implantation direction Determine the implantation location in the fixed platform coordinate system of the object. and implantation direction ,For example and In the implantation point selection mode, This can be obtained by finding the intersection point between the back-projected ray of the pixel selected by the user in the image of the monitoring camera and the three-dimensional surface model of the target object. It can be set to a local surface normal, a preset vertical direction, or a user-specified direction.

[0207] As an example, before determining the target pose of the implantation tool based on the position of the implantation hole and the target pose, the implantation module 130 can transform the position of the implantation hole to the coordinate system of the implantation robot arm base. Specifically, this transformation occurs when the object fixing platform moves to the implantation station (e.g., the position described above). In the case of ), the location of the implantation hole in the world coordinate system. It can be represented as: The location of the implantation hole in the coordinate system of the implantation arm base. It can be represented as: The implantation direction in the implant arm base coordinate system It can be represented as: Here, in an embodiment where only a translation slide exists, It can be considered as an identity matrix.

[0208] As an example, the implantation module 130 can perform posture planning for the implantation tool by aligning the optical axis of the implantation tool with the implantation direction in the target posture and minimizing the rotation angle between the target posture and the current posture.

[0209] Specifically, in defining the coordinate system of the implanted application. Standard insertion direction For its part The axial direction can be represented, for example, by the following equation (21): (twenty one) The goal is to find the target pose of the implanted tool. This allows the product of the target orientation and the aforementioned standard insertion direction to be equal in the implant arm base coordinate system. The unit direction of implantation below For example, it can be represented by the following formula (22): (twenty two) Here, unit direction It can be expressed by the following formula (22): (twenty three) Current posture of the implantation tool It can be expressed by the following formula (24): (twenty four) The current coordinate system of the implantation tool in the implantation arm base. Below Axial direction For example, it can be expressed by the following formula (25): (25) Therefore, it can be based on unit direction and the above Axial direction Calculate the rotation axis of the implantation tool For example, it can be expressed by the following formula (26): (26) It can also be based on unit direction and the above Axial direction Calculate the rotation angle of the implantation tool For example, it can be expressed by the following formula (27): (27) like If both are in the same direction, then the identity matrix is ​​taken by alignment and rotation; if Furthermore, if the two are opposite, then choose any one that is consistent with the above. Axial direction Let the orthogonal unit vectors be the axis of rotation, and let... Otherwise, the relationship can be constructed as shown in equation (28): (28) Based on the above equation (28), the alignment rotation matrix can be constructed using the Rodrigues formula. For example, it can be expressed by the following formula (29): (29) in, For antisymmetric matrices, Represents the identity matrix.

[0210] Because there is a degree of freedom around the implantation direction, it is possible to further select the local area around the tool. Rotation angle of the axis This minimizes the rotational change of the target pose relative to the current pose. Here, the target pose... It can be expressed by the following formula (30): (30) Here, a matrix can be constructed. It can be expressed by the following formula (31): (31) Based on the matrix in equation (31) above The closed-form solution for the rotation angle can be obtained, which can be expressed by the following equation (32): (32) Finally, the target pose can be obtained as shown in equation (33). : (33) By using the above solution process, the rotation angle of the end effector can be reduced while ensuring that the insertion axis is aligned with the planned direction, thereby reducing the risk of cable twisting.

[0211] The above method allows for alignment and implantation with minimal rotational orientation. For implantation tools such as flexible electrodes, flexible probes, or implantable devices with leads, this method addresses the issues of bending, tangling, and twisting of the implantation tool. It optimizes the free rotation angle around the implantation direction while ensuring that the local insertion axis of the tool is consistent with the planned implantation direction, thereby minimizing the rotational change of the end effector relative to the current orientation and reducing the risk of torsion of flexible devices.

[0212] Given the location of the implantation hole and the target orientation, the target orientation of the implantation tool can be determined, thereby controlling the implantation robotic arm to move the implantation tool to that target orientation.

[0213] Specifically, the target pose of the implantation tool can be represented by the following equation (34): (34) Here, due to the existence of the following relationship: Therefore, the target pose of the end effector (e.g., the end flange) of the implanted robotic arm can be represented as: .

[0214] Given the target pose, the robot controller can... Perform inverse kinematics solutions and check at least one of the following: workspace accessibility; joint constraints; singular poses; and collision risk with object-fixed platforms, machining components, navigation arms, or other hardware.

[0215] Under the constraints of the above-mentioned inspection items, the implantation robotic arm can be driven to the target posture, and precision implantation can be performed by a micro-motion platform, linear actuator, or voice coil motor. As an example, during the implantation process, the end position of the implantation needle, electrode coil center, or flexible device can be tracked using sensors such as a binocular microscopic vision module or other sensors, and the deviation between the actual path and the planned path can be evaluated in real time.

[0216] The following will use medical implantation scenarios as examples to describe examples of the implantation process using the control device according to embodiments of the present disclosure in two implantation planning modes.

[0217] As an example, such as Figure 5 As shown, in the implantation planning mode, in step S501, the target object can be fixed to the object fixing platform. In step S502, the object fixing platform can be moved to the navigation station, and in step S503, the target object (e.g., an animal head or target area) can be scanned by a navigation camera such as a structured light camera. In step S504, a three-dimensional surface model of the target object in the object fixing platform coordinate system can be generated.

[0218] In step S505, the object fixing platform can be moved to the processing station, and in step S506, the target pixel point selected by the user in the image from the monitoring camera can be received. In step S107, the selected target pixel point can be back-projected into a spatial ray, and in step S508, the intersection point of the spatial ray and the three-dimensional model can be calculated to obtain the three-dimensional opening point. In step S509, the offset of the slide and the instruction indicating the processing component can be calculated. In step S510, the opening can be performed using the processing component, and a closed-loop termination judgment is executed.

[0219] In step S511, the object fixing platform can be moved to the implantation station, and in step S512, the application end of the implantation tool, such as a needle tip, wire loop, or key point of a flexible device, can be identified. In step S513, the implantation robotic arm can be aligned with the micropore and the implantation can be performed.

[0220] As an example, such as Figure 6 As shown, in the pre-implantation planning mode, in step S601, preoperative MRI / CT imaging can be performed, and in step S602, a model of the skull, brain surface, blood vessels, or target brain region can be reconstructed based on the imaging results. In step S603, the implantation location and direction as the entry point can be planned in the image coordinate system.

[0221] In step S604, the target object can be fixed to the object fixation platform. In step S605, the target object (e.g., an animal head or target area) can be scanned using a navigation camera such as a structured light camera to obtain an intraoperative surface point cloud. In step S606, the preoperative image model can be registered with the intraoperative point cloud. In step S607, the position transformation relationship between the image coordinate system and the object fixation platform coordinate system can be solved. In step S608, the planned implantation position and implantation direction can be transformed from the image coordinate system to the object fixation platform coordinate system.

[0222] In step S609, the object fixing platform can be moved to the processing station, and in step S610, the processing component can be used to make a hole and perform a closed-loop termination judgment.

[0223] In step S611, the object fixing platform can be moved to the implantation station, and in step S612, the implantation robot arm can be controlled to perform minimum rotational orientation alignment and flexible electrode implantation.

[0224] The robot control device according to embodiments of this disclosure can achieve high-precision coordinate unification. Specifically, by using feature markers such as laser in-situ ablation points as a common physical reference, the navigation system, processing system, implantation system, and monitoring camera can all be unified to the same world coordinate frame, reducing the accumulation of errors in multi-level calibration.

[0225] Furthermore, the robot control device according to the embodiments of this disclosure can not only complete the registration of preoperative images and intraoperative point clouds, but also directly use the registration results for the planning of the opening and implantation posture of the processing components, realizing a closed-loop mapping from planning to execution, so that the closed loop of image planning and physical execution is connected.

[0226] Furthermore, the robot control device according to embodiments of this disclosure can support rapid intraoperative planning without preoperative images. Specifically, it can determine the true three-dimensional target points without preoperative images by intersecting the pixel back projection of the monitoring camera with the three-dimensional model, thereby improving the system's applicability.

[0227] Furthermore, the robot control device according to the embodiments of this disclosure makes the hole-opening process safer and more controllable. Specifically, the device can achieve visual verification, intermittent scanning, status judgment and termination control during the micro-hole opening process through coordinated control of the slide and processing components and coaxial visual feedback, thereby reducing the risk of insufficient opening or excessive ablation.

[0228] Furthermore, the robot control device according to embodiments of this disclosure can reduce the risk of cable twisting. Specifically, the device's minimum rotational attitude planning can align the implantation tool axis with the implantation direction while reducing rotation around the implantation direction, which helps to avoid cable tangling, bending, or deflection.

[0229] Furthermore, the robot control device according to the embodiments of this disclosure can be adapted to different types of structures (such as different types of robotic arms, structured light cameras, ToF cameras, binocular cameras, lasers, galvanometers, and implanted actuators, etc.). It can be migrated to different experimental platforms simply by recalibrating accordingly, demonstrating strong adaptability.

[0230] According to a second aspect of the embodiments of the present disclosure, a robot is provided, which may include control devices for the robot according to the embodiments of the present disclosure.

[0231] In addition, the robot may also include the navigation system, processing system and implantation system mentioned above. The specific implementation of each system has been described in detail in the embodiments of the relevant control device, and will not be elaborated here.

[0232] According to a third aspect of the embodiments of this disclosure, a robot control method is provided, such as... Figure 7 As shown, the control method may include: step S710, using the robot's navigation system to collect shape data of the target object; step S720, using the robot's processing system to process the target object according to the shape data and the position of the hole to be opened on the target object, so as to form an implantation hole on the target object; step S730, using the robot's implantation system to implant the implantation tool into the target object through the implantation hole.

[0233] As an example, the control method further includes: using a machining system to process a calibration part of the robot to mark feature markers on the calibration part, wherein the calibration part is fixed to an object fixing platform of the robot for fixing a target object; and calibrating the positions of the navigation system, implantation system and machining system based on the feature markers.

[0234] As an example, the navigation system includes a navigation robotic arm and a navigation camera mounted on the navigation robotic arm. The navigation camera is used to collect shape data of the target object, and the navigation robotic arm is used to drive the navigation camera to move. The object fixing platform is movable. The position calibration of the navigation system, implantation system, and processing system based on feature markers includes: controlling the object fixing platform to move to a navigation station for the navigation camera to collect data; using the navigation camera to collect first shape data of feature markers on the calibration component; and calibrating the positional relationship between the navigation robotic arm and the robot's reference position based on the first shape data.

[0235] As an example, the calibration of the positional relationship between the navigation robot arm and the robot's reference position based on the first shape data includes: determining a first position of the feature marker relative to the navigation robot arm when the object fixing platform is in the navigation station based on the first shape data; determining a second position of the feature marker relative to the reference position when the object fixing platform is in the navigation station based on the positional relationship between the navigation station and the reference position; and calibrating the positional relationship between the navigation robot arm and the reference position based on the first and second positions.

[0236] As an example, the implantation system includes an implantation robotic arm, an implantation tool disposed at the end of the implantation robotic arm, the implantation robotic arm being used to drive the implantation tool to move, and an object fixing platform being movable. The step of calibrating the positions of the navigation system, implantation system, and processing system based on feature markers includes: controlling the object fixing platform to move to an implantation station for the implantation robotic arm to perform implantation operations; controlling the implantation robotic arm to drive the implantation tool to move so that the application end of the implantation tool contacts the feature marker, and recording the pose data of the end of the implantation robotic arm when the application end contacts the feature marker; and calibrating the positional relationship between the implantation robotic arm and the robot's reference position based on the pose data.

[0237] As an example, the calibration of the positional relationship between the implanted robotic arm and the robot's reference position based on pose data includes: determining the third position of the feature marker relative to the implanted robotic arm when the application end contacts the feature marker based on pose data; determining the fourth position of the feature marker relative to the reference position when the application end contacts the feature marker based on the positional relationship between the implantation station and the reference position; and calibrating the positional relationship between the implanted robotic arm and the reference position based on the third and fourth positions.

[0238] As an example, the processing system includes a monitoring camera and a processing component. The processing component is used to perform processing operations at a processing station, and the monitoring camera is used to collect shape data of the processing station. The object fixing platform is capable of moving in a first direction, and the processing component is capable of moving upward in a second direction and a third direction, which are perpendicular to the first direction, respectively. The second direction and the third direction are perpendicular to each other. The position calibration of the navigation system, the implantation system, and the processing system based on feature markers includes: controlling the object fixing platform to move to the processing station; using the monitoring camera to collect second shape data of the feature markers; and calibrating the positional relationship between the monitoring camera and the processing component based on the second shape data.

[0239] As an example, the calibration of the positional relationship between the monitoring camera and the processing component based on the second shape data includes: determining the position of the feature marker relative to the processing component; and calibrating the positional relationship between the monitoring camera and the processing component based on the position of the feature marker relative to the processing component and the second shape data.

[0240] As an example, the method further includes: determining the location of the hole to be opened based on pre-acquired image data and shape data; and / or, determining the location of the hole to be opened based on received user instructions.

[0241] As an example, determining the location of the hole to be opened based on the pre-acquired image data and shape data includes: registering the pre-acquired image data and shape data; and determining the location of the hole to be opened based on the registration result.

[0242] As an example, the processing system includes a monitoring camera and a processing component. The processing component is used to perform processing operations on a processing station, and the monitoring camera is used to acquire image data of the processing station. The step of determining the location of the hole to be drilled according to the received user instruction includes: acquiring target image data of the target object using the monitoring camera; and determining the location of the hole to be drilled according to the received user instruction, wherein the user instruction indicates the target location selected by the user in the target image data.

[0243] As an example, determining the location of the hole to be opened according to the received user instruction includes: determining the processing direction from the processing component toward the target object based on the target location; and determining the location of the hole to be opened according to the shape data and the processing direction.

[0244] As an example, the processing system includes a processing component for performing processing operations. The processing of the target object using the robot's processing system based on shape data and the location of the hole to be opened on the target object includes: determining the processing position for the processing component to perform the processing operation at the location of the hole to be opened based on the shape data; controlling the slide to move to the processing position; and processing the target object using the processing component to form an implantation hole.

[0245] As an example, the processing system includes a monitoring camera and a processing component. The processing component is used to perform processing operations. The monitoring camera is used to acquire image data. The processing component uses a laser to perform processing operations, wherein the target object is processed by executing at least one of the following processing strategies: in response to the location of the hole to be opened being a single point, the laser focus position of the processing component is kept constant, and the target object is processed by adjusting the position of the processing component; or, in response to the location of the hole to be opened including multiple points or continuous patterns within the same field of view of the monitoring camera, the position of the processing component is kept constant, and the target object is processed by adjusting the laser focus position of the processing component. While processing the target object by adjusting the laser focus position of the processing component, in response to the laser focus position of the processing component deflecting beyond a predetermined range, the laser focus position of the processing component is kept constant, and the target object is processed by adjusting the position of the processing component.

[0246] As an example, the processing system includes a monitoring camera and a processing component. The processing component is used to perform processing operations, and the monitoring camera is used to acquire image data. The target object is processed by iteratively performing the following operations: using the processing component, performing the current processing on the processing location of the target object; using the monitoring camera to acquire an image of the processing location of the target object; determining whether the current state of the processing location meets the hole-opening termination condition based on the image of the processing location; stopping the iterative operation in response to the current state of the processing location meeting the hole-opening termination condition; and performing the next processing on the processing location using the processing component in response to the current state of the processing location not meeting the hole-opening termination condition.

[0247] As an example, the implantation system includes an implantation robotic arm, with an implantation tool disposed at the end of the implantation robotic arm. The implantation robotic arm is used to move the implantation tool. The implantation system utilizing the robot to implant the implantation tool into a target object through an implantation hole includes: determining a target posture of the implantation tool based on its current posture, such that the optical axis of the implantation tool is aligned with the implantation direction and the rotation angle between the target posture and the current posture is minimized; determining a target pose of the implantation tool based on the position of the implantation hole and the target posture; and controlling the implantation robotic arm to move the implantation tool to the target pose, thereby implanting the implantation tool into the target object through the implantation hole.

[0248] The specific implementation of the control method described herein has been described in detail in the embodiments of the relevant control device, and will not be elaborated here.

[0249] According to embodiments of this disclosure, the control method described above can be implemented using an electronic device such as a robot controller. The electronic device may include a processor and a memory for storing processor-executable instructions. Here, when executed by the processor, the processor-executable instructions cause the processor to perform the operations described in the exemplary embodiments above.

[0250] As an example, an electronic device does not necessarily have to be a single device; it can also be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. An electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a server, either locally or remotely (e.g., via wireless transmission).

[0251] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0252] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0253] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0254] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device can be interconnected via buses and / or networks.

[0255] In an exemplary embodiment, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the robot control method as described in the exemplary embodiment above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, or a BD-R... LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0256] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer-executable instructions that, when executed by at least one processor, implement a robot control method according to exemplary embodiments of the present disclosure.

[0257] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0258] Furthermore, it should be noted that although several examples of each step have been described above with reference to the specific accompanying drawings, it should be understood that the embodiments of this disclosure are not limited to the combinations given in the examples. The steps appearing in different drawings can be combined, and the execution order of each step can be changed, which will not be exhaustive here.

[0259] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A control device for a robot, characterized in that, The control device includes: The acquisition module is used to acquire shape data of the target object using the robot's navigation system; The processing module is used to process the target object according to the shape data and the position of the hole to be opened on the target object using the processing system of the robot, so as to form an implantation hole on the target object; An implantation module is used to implant an implantation tool into the target object through the implantation hole using the robot's implantation system.

2. The control device according to claim 1, characterized in that, The control device further includes a calibration module, which is used for: Using the processing system, the calibration part of the robot is processed to mark feature marks on the calibration part, wherein the calibration part is fixed to the robot's object fixing platform for fixing the target object; Based on the feature markers, the navigation system, the implantation system, and the processing system are calibrated in terms of location.

3. The control device according to claim 2, characterized in that, The navigation system includes a navigation robotic arm and a navigation camera mounted on the robotic arm. The navigation camera is used to collect shape data of the target object, and the navigation robotic arm is used to move the navigation camera. The object fixing platform is movable. The calibration module is used for: Control the object fixing platform to move to the navigation station where the navigation camera can collect data; Using the navigation camera, first shape data of the feature mark on the calibration component is acquired; Based on the first shape data, the positional relationship between the navigation robotic arm and the robot's reference position is calibrated.

4. The control device according to claim 3, characterized in that, The calibration module is used for: Based on the first shape data, determine the first position of the feature marker relative to the navigation robot arm when the object fixing platform is in the navigation station; Based on the positional relationship between the navigation station and the reference position, a second position of the feature marker relative to the reference position is determined when the object fixing platform is at the navigation station; Based on the first position and the second position, the positional relationship between the navigation robot arm and the reference position is calibrated.

5. The control device according to claim 2, characterized in that, The implantation system includes an implantation robotic arm, with the implantation tool disposed at the end of the robotic arm. The robotic arm is used to move the implantation tool, and the object fixing platform is movable. The calibration module is used for: Control the object fixing platform to move to the implantation station for the implantation robotic arm to perform the implantation operation; The implantation robotic arm is controlled to move the implantation tool so that the application end of the implantation tool contacts the feature marker, and the pose data of the end of the implantation robotic arm is recorded when the application end contacts the feature marker; Based on the pose data, the positional relationship between the implanted robotic arm and the robot's reference position is calibrated.

6. The control device according to claim 5, characterized in that, The calibration module is used for: Based on the pose data, a third position of the feature marker relative to the implanted robotic arm is determined when the application terminal comes into contact with the feature marker. Based on the positional relationship between the implantation station and the reference position, a fourth position of the feature marker relative to the reference position is determined when the application terminal comes into contact with the feature marker; Based on the third and fourth positions, the positional relationship between the implanted robotic arm and the reference position is calibrated.

7. The control device according to claim 2, characterized in that, The processing system includes a monitoring camera and a processing component. The processing component performs processing operations on a processing station. The monitoring camera acquires shape data of the processing station. The object fixing platform is movable in a first direction. The processing component is movable upwards in a second direction and a third direction, which are perpendicular to the first direction, respectively. The second direction and the third direction are perpendicular to each other. The calibration module is used for: Control the object fixing platform to move to the processing station; Using the surveillance camera, the second shape data of the feature marker is acquired; Based on the second shape data, the positional relationship between the monitoring camera and the processing component is calibrated.

8. The control device according to claim 7, characterized in that, The calibration module is used for: Determine the position of the feature marker relative to the processing component; The positional relationship between the monitoring camera and the processing component is calibrated based on the position of the feature marker relative to the processing component and the second shape data.

9. The control device according to claim 1, characterized in that, The processing module is used for: Based on the pre-acquired image data and the shape data, the location of the hole to be opened is determined; and / or, The location of the hole to be opened is determined based on the received user instruction.

10. The control device according to claim 9, characterized in that, The processing module is used for: The pre-acquired image data and the shape data are registered; Based on the registration results, the location of the hole to be opened is determined.

11. The control device according to claim 9, characterized in that, The processing system includes a monitoring camera and processing components. The processing components are used to perform processing operations at the processing station, and the monitoring camera is used to acquire image data of the processing station. The processing module is used for: Using the surveillance camera, target image data of the target object is acquired; The location of the hole to be opened is determined according to the received user instruction, wherein the user instruction indicates the target location selected by the user in the target image data.

12. The control device according to claim 11, characterized in that, The processing module is used for: Based on the target location, determine the processing direction from the processing component toward the target object; The location of the hole to be opened is determined based on the shape data and the processing direction.

13. The control device according to claim 1, characterized in that, The processing system includes processing components for performing processing operations. The processing module is used for: Based on the shape data, determine the processing position where the processing component performs the processing operation on the position of the hole to be opened; Control the movement of the processing component to the processing position; The target object is processed using the processing components to form the implantation hole.

14. The control device according to claim 1, characterized in that, The processing system includes a monitoring camera and a processing component. The processing component performs processing operations, the monitoring camera acquires image data, and the processing component utilizes lasers to perform the processing operations. The processing module is configured to process the target object by executing at least one of the following processing strategies: In response to the fact that the location to be drilled is a single point, the laser focus position of the processing component remains unchanged, and the target object is processed by adjusting the position of the processing component; or, In response to the location of the hole to be opened including multiple points or continuous patterns within the same field of view of the monitoring camera, the position of the processing component remains unchanged, and the target object is processed by adjusting the laser focus position of the processing component. Specifically, when processing the target object by adjusting the laser focus position of the processing component, in response to the laser focus position of the processing component deflecting beyond a predetermined range, the laser focus position of the processing component remains unchanged, and the target object is processed by adjusting the position of the processing component.

15. The control device according to claim 1, characterized in that, The processing system includes a monitoring camera and processing components. The processing components are used to perform processing operations, and the monitoring camera is used to acquire image data. The processing module is used to process the target object by iteratively performing the following operations: Using the processing components, the current processing is performed on the processing position of the target object; The monitoring camera is used to capture images of the processing location of the target object; Based on the image of the processed position, determine whether the current state of the processed position meets the hole termination condition; The iteration operation stops when the current state of the processed position satisfies the opening termination condition. In response to the current state of the processed position not satisfying the hole termination condition, the processing component is used to perform the next processing on the processed position.

16. The control device according to claim 1, characterized in that, The implantation system includes an implantation robotic arm, with the implantation tool disposed at the end of the implantation robotic arm. The implantation robotic arm is used to drive the implantation tool to move. The implantation module is used for: Based on the current posture of the implantation tool, a target posture of the implantation tool is determined, such that the optical axis of the implantation tool is aligned with the implantation direction in the target posture and the rotation angle between the target posture and the current posture is minimized; Based on the position of the implantation hole and the target orientation, the target orientation of the implantation tool is determined; The implantation robotic arm is controlled to move the implantation tool to the target position so as to implant the implantation tool into the target object through the implantation hole.

17. A robot, characterized in that, The robot includes a control device according to any one of claims 1 to 16.

18. A method for controlling a robot, characterized in that, The control method includes: The robot's navigation system is used to collect shape data of the target object; Based on the shape data and the location of the hole to be drilled on the target object, the robot's processing system is used to process the target object to form an implantation hole on the target object; Using the robot's implantation system, the implantation tool is inserted into the target object through the implantation hole.

19. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the robot control method according to claim 18.

20. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, the robot control method according to claim 18 is implemented.

Images