Dual-arm ultrasound-guided automatic puncture surgical robot and method

Abstract

The present invention discloses a dual-arm ultrasound-guided automatic puncture surgical robot and method. The robot includes a first and second robotic arms with multiple degrees of freedom, an ultrasonic probe connected to the execution end of the first robotic arm, and a puncture instrument connected to the execution end of the second robotic arm. It also includes a position tracking system and an image workstation. The ultrasonic probe transmits the collected ultrasonic image to the image workstation via an ultrasound machine. The position tracking system is used to collect position information and transmit it to the image workstation. The image workstation is used to control the first robotic arm to adjust the position of the ultrasonic probe in real time, and to control the second robotic arm to adjust the posture of the puncture instrument to automatically puncture the surgical target. By combining ultrasonic imaging technology, robotics technology, and artificial intelligence, the present invention introduces deep learning methods into the surgical robot. It can detect, locate, and track the surgical target and surgical instruments in real time, and improve the accuracy of puncture surgery through the two robotic arms.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a dual-robotic-arm ultrasound-guided automatic puncture surgical robot and method. Background Art

[0002] Currently, minimally invasive surgical procedures are gaining increasing recognition and application in clinical fields such as biopsy and interventional therapy due to their minimal trauma, minimal pain, few surgical complications, and rapid recovery. Ultrasound-guided puncture procedures (biopsy puncture, ablation puncture, etc.) are widely used in clinical practice. Traditional preoperative puncture procedures, due to insufficient pre-operative lesion position information and interference from human body movement, are highly dependent on the surgeon's clinical experience and surgical skills, resulting in low stability and accuracy of the puncture operation, which affects the success rate of the procedure. Summary of the Invention

[0003] The purpose of the present invention is to provide a dual-arm ultrasound-guided automatic puncture surgical robot and method, aiming to overcome the above-mentioned defects of the prior art.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0005] According to one aspect of the present invention, a dual-robotic-arm ultrasound-guided automatic puncture surgical robot is disclosed, wherein the robot comprises a first robotic arm and a second robotic arm with multiple degrees of freedom, an ultrasonic probe connected to the execution end of the first robotic arm, a puncture instrument connected to the execution end of the second robotic arm, and further comprises a position tracking system and an image workstation; the ultrasonic probe transmits the collected ultrasonic image to the image workstation via an ultrasound machine, and the position tracking system is used to collect position information of the ultrasonic probe, the puncture instrument, the first robotic arm and the second robotic arm and transmit it to the image workstation; the image workstation is used to control the first robotic arm to adjust the position of the ultrasonic probe in real time according to the real-time position information and the ultrasonic image, and to control the second robotic arm to adjust the posture of the puncture instrument by calculating the position and posture difference between the puncture instrument and the surgical target in the ultrasonic image so as to automatically puncture the surgical target.

[0006] Furthermore, the robot also includes a robotic arm control system, which is used to drive the first robotic arm and the second robotic arm based on the calculation results of the position information and the ultrasonic image by the image workstation.

[0007] Furthermore, the robot also includes a GPU accelerator, and the image workstation establishes data connections with the GPU accelerator, the robotic arm control system, the ultrasound machine and the position tracking system in a wired and / or wireless manner.

[0008] Furthermore, the GPU accelerator is used to accelerate the processing of the ultrasound image by the image workstation, and is built-in or external to the image workstation.

[0009] Furthermore, the puncture instrument is one or more pieces, and one or more pieces of the puncture instrument and the ultrasound probe are provided with a tracker that can be identified by the position tracking system.

[0010] Furthermore, the position tracking system identifies the tracker based on one or more of optical, electromagnetic, ultrasonic or mechanical positioning methods.

[0011] Furthermore, the puncture instrument is one of a puncture needle, a syringe, and an infusion set.

[0012] Furthermore, the image workstation acquires the ultrasonic image of the ultrasound machine through an image acquisition card.

[0013] According to a second aspect of the present disclosure, a dual-manipulator ultrasound-guided automatic puncture surgery method is provided, which is applied to the robot as described above, the method comprising: calibrating the puncture instrument and the ultrasound probe to obtain calibration matrices for the puncture instrument and the ultrasound probe, respectively; displaying the ultrasound image scanned by the ultrasound probe and a three-dimensional model of the puncture instrument on the image workstation based on the calibration matrices; setting initial positions of the first and second robotic arms, and planning a surgical path for the puncture instrument; driving the first robotic arm to drive the ultrasound probe to perform ultrasound scanning around a surgical target, and transmitting a plurality of ultrasound images containing the surgical target to the image workstation; tracking the surgical target in the ultrasound image in real time based on a deep learning algorithm, and marking the center of the surgical target; and calculating the relative positional relationship between the puncture instrument and the surgical target in real time as the second robotic arm drives the puncture instrument to move along the surgical path. Based on the relative positional relationship, driving the second robotic arm to adjust the position and angle of the puncture instrument in real time to perform puncture at the center of the marked surgical target.

[0014] Furthermore, the deep learning algorithm is based on which the surgical target in the ultrasound image is tracked in real time and the center of the surgical target is marked, including: inputting the currently acquired ultrasound image and freezing it; identifying the surgical target to be tracked and inputting the ultrasound image into a pre-learned 2D ultrasound landmark tracking network; acquiring the ultrasound image marked with the landmark points and inputting it into the current target tracking network to track the center of the surgical target; and determining whether the tracking is finished. If not, continue acquiring the ultrasound image and tracking the surgical target. If finished, stop target tracking.

[0015] Furthermore, the real-time calculation of the relative positional relationship between the puncture instrument and the surgical target includes: calculating the normal of the plane equation of the ultrasound image in the current frame; tracking the puncture target point and calculating its three-dimensional coordinates based on a target tracking algorithm; recording the plane equation of the ultrasound image in the current frame and the three-dimensional coordinates of the puncture target point therein; and retrieving the plane equation of the ultrasound image in the previous frame and the three-dimensional coordinates of the puncture target point therein to calculate a first distance from the puncture target point in the current frame to the puncture target point in the previous frame, a second distance from the puncture target point in the current frame to the normal line in the previous frame, and a deviation between the normal lines of the current frame and the previous frame. Using the first distance, the second distance, and the deviation, the three-dimensional spatial position of the current surgical target is adjusted and transmitted to the imaging workstation.

[0016] Furthermore, when the puncture instrument is performing a puncture, the apparatus includes: providing in real time the needle tip position of the puncture instrument calculated under the identification and calibration of the position tracking system; driving the second robotic arm to move the ultrasonic probe to ultrasonically scan the area of ​​the needle tip position of the puncture instrument to search for the needle tip of the puncture instrument; tracking the needle tip position of the puncture instrument based on the ultrasonic image target tracking algorithm; calculating the deviation of the needle tip position of the puncture instrument, and adjusting the posture of the second robotic arm when the deviation is greater than a threshold.

[0017] The technical solution disclosed in this disclosure has the following beneficial effects:

[0018] By combining the characteristics of ultrasound imaging technology, robotics technology and artificial intelligence, deep learning methods are introduced into surgical robots guided by ultrasound images. They can autonomously collect data and, through spatial coordinate alignment and online ultrasound image processing, perform real-time detection, positioning and tracking of surgical targets and surgical instruments during surgery. The stable support of two robotic arms can also improve the accuracy of puncture surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a structural block diagram of a dual-arm ultrasound-guided automatic puncture surgical robot in an embodiment of this specification;

[0020] Figure 2 This is a schematic structural diagram of an ultrasound probe with an optical tracker fixed thereto according to an embodiment of this specification;

[0021] Figure 3 This is a flow chart of the puncture surgical method in the embodiment of this specification;

[0022] Figure 4 This is a flow chart of a method for real-time tracking of a surgical target in an ultrasound image in an embodiment of this specification;

[0023] Figure 5 This is a flow chart of a method for calculating the relative position relationship between a puncture instrument and a surgical target in real time in an embodiment of this specification;

[0024] Figure 6 This is a flow chart of the puncture instrument position identification method in the embodiment of this specification. DETAILED DESCRIPTION

[0025] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0026] Reference Figure 1 The present embodiment provides a dual-arm ultrasound-guided automatic puncture surgical robot, the robot comprising a first robotic arm 101 and a second robotic arm 102 with multiple degrees of freedom, an ultrasonic probe 103 connected to the execution end of the first robotic arm 101, a puncture instrument 104 connected to the execution end of the second robotic arm 102, and a position tracking system 105 and an image workstation 106; the ultrasonic probe 103 transmits the collected ultrasonic image to the image workstation 106 via an ultrasonic machine 107, and the position tracking system 105 is used to collect the ultrasonic image. The position information of the acoustic probe 103, the puncture instrument 104, the first robotic arm 101 and the second robotic arm 102 is transmitted to the image workstation 106; the image workstation 106 is used to control the first robotic arm 101 to adjust the position of the ultrasonic probe 103 in real time according to the real-time position information and the ultrasonic image, and to control the second robotic arm 102 to adjust the posture of the puncture instrument 104 by calculating the position and posture difference between the puncture instrument 104 and the surgical target in the ultrasonic image to automatically puncture the surgical target.

[0027] The term "puncture procedure" as used in this disclosure includes, but is not limited to, vascular puncture, joint cavity puncture, lymph node puncture, tumor puncture, and the like. The term "puncture instrument 104" as used in this disclosure may refer to a thin, flexible instrument that can be inserted into a blood vessel, joint cavity, muscle, or the like. It may be a puncture needle, syringe, infusion set, guidewire, catheter, or the like. The "ultrasound probe 103" may be an array ultrasound probe 103, a dual-plane ultrasound probe 103, or a volumetric three-dimensional ultrasound probe 103 capable of scanning multiple planes or providing real-time three-dimensional volumetric ultrasound scanning. There may be one or multiple ultrasound probes 103 connected to the same first robotic arm 101, or multiple ultrasound probes 103 connected to different first robotic arms 101, as long as each ultrasound probe 103 has independent degrees of freedom of movement. The imaging workstation 106 may include a processor, memory, a display, and control signal input peripherals (such as a mouse and keyboard). Generally, the imaging workstation 106 is located to one side of the operating table, with the display directly in front of the surgeon. The first robotic arm 101 and the second robotic arm 102 can be multi-axis robotic arms with multiple motion joints. Then, the first robotic arm 101 and the second robotic arm 102 use the angular displacement of each joint to feedback the movement path, and the movement of the ultrasonic probe 103 and the puncture instrument 104 relative to the initial position can be known, and the relative position between the ultrasonic probe 103 and the puncture instrument 104 can also be known.

[0028] In one embodiment, the robot further includes a robotic arm control system 108, which is configured to drive the first robotic arm 101 and the second robotic arm 102 based on the calculation results of the position information and the ultrasound image by the imaging workstation 106. Specifically, the robotic arm control system 108 may be a motion manipulator. The robotic arm control system 108 may be one, with one robotic arm control system 108 being connected to both the first robotic arm 101 and the second robotic arm 102, or two, with two robotic arm control systems 108 being connected to the first robotic arm 101 and the second robotic arm 102, respectively.

[0029] In one embodiment, the robot further includes a GPU accelerator 109. The imaging workstation 106 establishes a data connection with the GPU accelerator 109, the robotic arm control system 108, the ultrasound machine 107, and the position tracking system 105 via wired and / or wireless means. The GPU accelerator 109 is configured to accelerate the processing of the ultrasound image by the imaging workstation 106 and is internally or externally located in the imaging workstation 106.

[0030] In one embodiment, the puncture instrument 104 is one or more pieces, and one or more of the puncture instruments 104 and the ultrasound probe 103 are provided with a tracker 111 that can be identified by the position tracking system 105. In one embodiment, the position tracking system 105 identifies the tracker 111 based on one or more optical, electromagnetic, ultrasonic, or mechanical positioning methods.

[0031] Specifically, such as Figure 2 As shown, taking the optical tracking system as an example, when the tracker 111 is installed on the ultrasound probe 103, it can be directly installed on the housing of the ultrasound probe 103 or installed on the positioning object 112 of the ultrasound probe 103. The fixing method can be permanent, semi-permanent, or removable. The optical tracker 111 can be a flat graphic sticker directly attached to or painted on the surface of the ultrasound probe 103 or puncture instrument 104.

[0032] In one embodiment, the image workstation 106 acquires the ultrasound image of the ultrasound machine 107 via an image acquisition card 110 to speed up the processing speed of the image workstation 106 .

[0033] Based on the above robot, the embodiment of this specification also provides a dual-arm ultrasound-guided automatic puncture surgery method, referring to Figure 3 , including steps S301-S306:

[0034] In step S301, the puncture instrument and the ultrasound probe are calibrated to obtain calibration matrices of the puncture instrument and the ultrasound probe respectively.

[0035] Among them, the calibration of the ultrasonic probe specifically includes: based on the position tracking system and the tracker set on the ultrasonic probe, obtaining the coordinates of the ultrasonic probe in different postures, and obtaining the coordinate system of the ultrasonic probe in the position tracking system; based on the ultrasonic phantom, obtaining the ultrasonic image corresponding to the ultrasonic phantom generated by the ultrasonic probe in different postures, and obtaining the ultrasonic coordinate system of the ultrasonic image; based on the execution path of the first manipulator during calibration, calculating the conversion relationship between the coordinate system of the ultrasonic probe in the position tracking system and the ultrasonic coordinate system of the ultrasonic image. Specifically, the ultrasonic phantom is as follows: Figure 4 As shown, the calibration phantom 113 includes a fixed nylon thread 114 or other materials that can be easily visualized in an ultrasound device, and other parts are filled with agar to simulate human tissue.

[0036] In step S302, based on the calibration matrix, the ultrasound image scanned by the ultrasound probe and the three-dimensional model of the puncture instrument are displayed on the image workstation.

[0037] In step S303 , the initial positions of the first robotic arm and the second robotic arm are set, and the surgical path of the puncture instrument is estimated.

[0038] In step S304 , the first robotic arm is driven to drive the ultrasound probe to perform ultrasound scanning around the surgical target, and transmit a plurality of ultrasound images containing the surgical target to the image workstation.

[0039] In step S305 , based on a deep learning algorithm, the surgical target in the ultrasound image is tracked in real time, and the center of the surgical target is marked.

[0040] In step S306, when the second robotic arm drives the puncture instrument to move along the surgical path, the relative position relationship between the puncture instrument and the surgical target is calculated in real time. Based on the relative position relationship, the second robotic arm is driven to adjust the position and angle of the puncture instrument in real time and then puncture the center of the marked surgical target.

[0041] In one embodiment, if Figure 4 As shown, step S305 also includes steps S401 to S404:

[0042] In step S401, the currently acquired ultrasound image is input and frozen.

[0043] In step S402, the surgical target to be tracked is identified, and the ultrasound image is input into a pre-learned 2D ultrasound landmark tracking network;

[0044] In step S403, the ultrasound image marked with the landmarks is collected and input into the current target tracking network to track the center of the surgical target;

[0045] In step S404, it is determined whether the tracking is completed. If not, the ultrasound image is continuously acquired and the surgical target is tracked. If it is completed, the target tracking is stopped.

[0046] In one embodiment, step S306 specifically includes steps S501 to S503:

[0047] In step S501, the normal N_Pn of the plane equation Pn of the ultrasound image In of the current frame is calculated, and based on the target tracking algorithm, the puncture target point Obj_n is tracked and its three-dimensional coordinates are calculated, and the plane equation Pn of the ultrasound image In of the current frame and the three-dimensional coordinates of the puncture target point Obj_n therein are recorded.

[0048] In step S502, the plane equation Pn-1 of the previous ultrasound image In-1 and the three-dimensional coordinates of the puncture target point Obj_n-1 are used. The first distance Dobj from the puncture target point Obj_n in the current ultrasound image In to the puncture target point Obj_n-1 in the previous frame is calculated. The second distance Dobj-Pn-1 from the puncture target point Obj_n in the current frame to the normal line Pn-1 in the previous frame is calculated. Finally, the deviation N_Pn in the current frame and the normal line N_Pn-1 in the previous frame, NBiasobj-Pn-1 = N_Pn-N_Pn-1, is calculated.

[0049] In step S503, the three-dimensional spatial position of the current surgical target is adjusted using the first distance Dobj, the second distance Dobj-Pn-1, and the deviation NBiasobj-Pn-1, and the new coordinates are transferred to the imaging workstation.

[0050] In one embodiment, if Figure 5 As shown, step S306 also includes steps S601 to S604:

[0051] In step S601, the needle tip position P_pin_rob of the puncture instrument calculated by the position tracking system under identification and calibration is given in real time;

[0052] In step S602, the second robotic arm is driven to move the ultrasonic probe to ultrasonically scan the area where the needle tip of the puncture instrument is located, thereby searching for the needle tip of the puncture instrument.

[0053] In step S603, the needle tip position P_pin of the puncture instrument is tracked based on an ultrasound image target tracking algorithm;

[0054] In step S604, the deviation Pbias_pin=P_pin_rob-P_pin of the needle tip position of the puncture instrument is calculated. When the deviation Pbias_pin=P_pin_rob-P_pin is greater than a threshold, the posture of the second robotic arm is adjusted until the deviation Pbias_pin=P_pin_rob-P_pin is less than the threshold.

[0055] The above embodiments combine the characteristics of ultrasound imaging technology, robotics technology and artificial intelligence to introduce deep learning methods into surgical robots guided by ultrasound images. They can autonomously collect data and, through spatial coordinate alignment and online ultrasound image processing, can perform real-time detection, positioning and tracking of surgical targets and surgical instruments during surgery. The stable support of two robotic arms can also improve the accuracy of puncture surgery.

[0056] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will appreciate that the technical solutions described in the aforementioned embodiments may be modified, or some or all of the technical features therein may be replaced with equivalents. Such modifications or replacements do not deviate from the essence of the corresponding technical solutions within the scope of the technical solutions of the various embodiments of the present invention. Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not others, the combination of features from different embodiments is intended to be within the scope of the present invention and to form different embodiments. For example, in the claims above, any of the claimed embodiments may be used in any combination. The information disclosed in this background section is intended solely to enhance understanding of the general background technology of the present invention and should not be construed as an admission or any form of suggestion that such information constitutes prior art known to those skilled in the art.

Claims

1. A dual-arm ultrasound-guided automatic puncture surgical robot, characterized in that: The robot includes a first robotic arm and a second robotic arm with multiple degrees of freedom, an ultrasonic probe connected to the execution end of the first robotic arm, and a puncture instrument connected to the execution end of the second robotic arm, and also includes a position tracking system and an image workstation; the ultrasonic probe transmits the collected ultrasonic image to the image workstation through the ultrasound machine, and the position tracking system is used to collect position information of the ultrasonic probe, the puncture instrument, the first robotic arm and the second robotic arm and transmit it to the image workstation; the image workstation is used to control the first robotic arm to adjust the position of the ultrasonic probe in real time according to the real-time position information and the ultrasonic image, and to control the second robotic arm to adjust the posture of the puncture instrument by calculating the difference in position and posture between the puncture instrument and the surgical target in the ultrasonic image so as to automatically puncture the surgical target; When puncturing, the robot includes: Calibrate the puncture instrument and the ultrasound probe to obtain calibration matrices of the puncture instrument and the ultrasound probe respectively; Based on the calibration matrix, the ultrasound image scanned by the ultrasound probe and the three-dimensional model of the puncture instrument are displayed on the image workstation; Setting the initial positions of the first robotic arm and the second robotic arm, and planning the surgical path of the puncture instrument; driving the first robotic arm to drive the ultrasound probe to perform ultrasound scanning around the surgical target, and transmitting a plurality of ultrasound images containing the surgical target to the image workstation; Based on a deep learning algorithm, the surgical target in the ultrasound image is tracked in real time, and the center of the surgical target is marked; When the second robotic arm drives the puncture instrument to move along the surgical path, the relative position relationship between the puncture instrument and the surgical target is calculated in real time. Based on the relative position relationship, the second robotic arm is driven to adjust the position and angle of the puncture instrument in real time and then puncture the center of the marked surgical target.

2. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The robot further includes a robotic arm control system, which is configured to drive the first robotic arm and the second robotic arm according to calculation results of the position information and the ultrasonic image by the image workstation.

3. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 2, characterized in that: The robot further includes a GPU accelerator, and the image workstation establishes data connections with the GPU accelerator, the robotic arm control system, the ultrasound machine, and the position tracking system in a wired and / or wireless manner.

4. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 3, characterized in that: The GPU accelerator is used to accelerate the processing of the ultrasound image by the image workstation, and is built-in or external to the image workstation.

5. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The puncture instrument is one or more pieces, and one or more of the puncture instruments and the ultrasound probe are provided with a tracker that can be identified by the position tracking system. The position tracking system identifies the tracker based on one or more optical, electromagnetic, ultrasonic or mechanical positioning methods.

6. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The image workstation acquires the ultrasonic image of the ultrasound machine through an image acquisition card.

7. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The method of tracking the surgical target in the ultrasound image in real time based on a deep learning algorithm and marking the center of the surgical target includes: Inputting the currently acquired ultrasound image and freezing it; Identifying a surgical target to be tracked, and inputting the ultrasound image into a pre-learned 2D ultrasound landmark tracking network; Acquire the ultrasound image marked with landmarks and input it into the current target tracking network to track the center of the surgical target; Determine whether tracking is completed. If not, continue to acquire the ultrasound image and track the surgical target. If it is completed, stop target tracking.

8. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The real-time calculation of the relative position relationship between the puncture instrument and the surgical target includes: Calculating the normal of the plane equation of the ultrasound image of the current frame, tracking the puncture target and calculating its three-dimensional coordinates based on the target tracking algorithm, and recording the plane equation of the ultrasound image of the current frame and the three-dimensional coordinates of the puncture target therein; Recalling the plane equation of the ultrasound image of the previous frame and the three-dimensional coordinates of the puncture target point therein, respectively calculating a first distance from the puncture target point in the current frame to the puncture target point in the previous frame, and calculating a second distance from the puncture target point in the current frame to the normal line of the previous frame; The deviation of the normal line between the current frame and the previous frame is calculated, and the three-dimensional spatial position of the current surgical target is adjusted according to the first distance, the second distance and the deviation, and then transmitted to the image workstation.

9. The dual-arm ultrasound-guided automatic puncture surgical robot according to claim 1, characterized in that: The puncture instrument comprises: Provide in real time the needle tip position of the puncture instrument calculated under the identification and calibration of the position tracking system; driving the second robotic arm to move the ultrasonic probe to ultrasonically scan an area where the needle tip of the puncture instrument is located, thereby searching for the needle tip of the puncture instrument; Tracking the needle tip position of the puncture instrument based on an ultrasound image target tracking algorithm; The deviation of the needle tip position of the puncture instrument is calculated, and when the deviation is greater than a threshold, the posture of the second robotic arm is adjusted.

Images