Automatic calibration of endoscope using pull wire
The endoscopic motion data is captured by the surgical robot system, the gain value is calculated and the pulling wire is manipulated, which solves the problem of insufficient endoscopic calibration accuracy, and achieves the precise navigation of the endoscopic and the improvement of surgical accuracy.
Patent Information
- Application Number
- CN202011084500.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-09-30
- Filing Date
- 2017-09-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2037-09-28
AI Technical Summary
The existing endoscopic calibration technology relies on limited endoscopic motion simulation, making it difficult to accurately calibrate, resulting in position deviation during the surgery and affecting surgical accuracy.
The surgical robot system uses space sensors to capture the actual position and motion data of the endoscope, calculate the gain value to compensate for its non-ideal behavior, and automatically calibrate the endoscope by manipulating the pull wires of the sheath and guides through the instrument device manipulator to ensure that Its precise navigation.
The precise motion simulation and navigation of the endoscopy is realized, the position accuracy during the surgery is improved, the dependence of manual calibration is reduced, and the reliability and efficiency of the surgery is improved.
Smart Images

Figure CN112472007B_ABST
Abstract
Description
[0001] This application is a divisional application of an application with an application date of September 28, 2017, application number 201780021357.X (international application number PCT / US2017 / 054127), and invention name “Automatic calibration of an endoscope using a pull wire”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] The subject matter of this application is related to U.S. application No. 14 / 523,760, filed on October 24, 2014, entitled “SYSTEM FOR ROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS”, the entire disclosure of which is incorporated herein by reference. Technical Field
[0004] The present specification relates generally to surgical robots, and specifically to an automated process for calibrating an endoscope with a pull wire. Background Art
[0005] Robotics has a wide range of applications. In particular, robotic arms help to complete tasks that humans normally perform. For example, factories use robotic arms to manufacture cars and consumer electronics. In addition, scientific facilities use robotic arms to automate laboratory procedures, such as transporting microplates. More recently, doctors have begun using robotic arms to help perform surgical procedures. For example, doctors use robotic arms to control surgical instruments, such as endoscopes.
[0006] Endoscopes with movable tips facilitate performing surgical procedures in a minimally invasive manner. The movable tip can be guided to a remote location of the patient, such as a lung or blood vessel. Deviations from the actual position of the tip from the target position may result in additional operations to correct the position of the tip. Existing techniques for manual calibration may rely on a limited amount of deflection of the tip of the endoscope that cannot accurately simulate the movement of the tip. Summary of the invention
[0007] A surgical robotic system automatically calibrates tubular and flexible surgical tools, such as endoscopes. By compensating for the non-ideal behavior of the endoscope, the surgical robotic system can accurately simulate the movement of the endoscope and navigate the endoscope while performing a surgical procedure on a patient. During calibration, the surgical robotic system moves the endoscope to a target position and receives calibration data describing the actual position of the endoscope. The surgical robotic system determines the actual position and / or orientation of the endoscope moving in response to a command based on the calibration data captured by a spatial sensor. Exemplary spatial sensors include accelerometers, gyroscopes, electromagnetic sensors, optical fibers, cameras, and fluoroscopic imaging systems. The surgical robotic system determines a gain value based at least on the difference between the target position and the actual position. The surgical robotic system can perform calibration before or during a surgical procedure.
[0008] In some embodiments, the endoscope includes tubular components referred to as a sheath and a guide. The gain value can also be based on the length of the guide extending outside the sheath or the relative roll angle of the guide relative to the sheath. The surgical robotic system uses an instrument device manipulator (IDM) to move the sheath and guide. For example, the IDM translates a pull line connected to the sheath or guide, which moves the endoscope along different axes, such as the pitch axis, the yaw axis, and the roll axis. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 A surgical robotic system according to one embodiment is shown.
[0010] Figure 2 A command console for a surgical robotic system is shown according to one embodiment.
[0011] Figure 3A Several degrees of motion of an endoscope are shown according to one embodiment.
[0012] Figure 3B is a top view of an endoscope including a sheath and a guide component according to one embodiment.
[0013] Figure 3C is a cross-sectional side view of a sheath of an endoscope according to one embodiment.
[0014] Figure 3D is an isometric view of a helical section of a sheath of an endoscope according to one embodiment.
[0015] Figure 3E is another isometric view of a helical section of a sheath of an endoscope according to one embodiment.
[0016] Figure 3F is a side view of a sheath with a helical section of an endoscope according to one embodiment.
[0017] Figure 3G According to one embodiment of the endoscope Figure 3F Another view of the sheath shown in .
[0018] Figure 3H is a cross-sectional side view of a guide of an endoscope according to one embodiment.
[0019] Fig. 3I According to one embodiment of the endoscope Figure 3H A cross-sectional isometric view of the distal tip of the guide shown in FIG.
[0020] Figure 4A is an isometric view of an instrument device manipulator of a surgical robotic system according to one embodiment.
[0021] Figure 4B According to one embodiment, Figure 4A Exploded isometric view of the instrument device manipulator shown in .
[0022] Figure 4C According to one embodiment, Figure 4A Isometric view of the independent drive mechanism of the instrument device manipulator shown in .
[0023] Figure 4D A conceptual diagram is shown, which shows how to Figure 4C The force is measured by strain gauges in the independent drive mechanism shown in FIG.
[0024] Figure 5A A pull wire is shown inside an endoscope according to one embodiment.
[0025] Figure 5B A rear view of the endoscope is shown in a rest position according to one embodiment.
[0026] Figure 5C According to one embodiment, Figure 5B A top view of the endoscope shown in FIG.
[0027] Figure 5D According to one embodiment, Figure 5B Side view of the endoscope shown in .
[0028] Figure 5E The diagram shows a deflected position according to one embodiment. Figure 5B Rear view of the endoscope shown in .
[0029] Fig. 5F According to one embodiment, Figure 5EA top view of the endoscope shown in FIG.
[0030] Figure 5G According to one embodiment, Figure 5E Side view of the endoscope shown in .
[0031] Figure 5H FIG. 1 shows a schematic diagram of a rotatable device in a deflected position with an additional non-ideal offset according to one embodiment. Figure 5B Rear view of the endoscope shown in .
[0032] Fig.5I According to one embodiment, Figure 5H A top view of the endoscope shown in FIG.
[0033] Figure 5J shows a static position according to one embodiment. Figure 5B Rear view of the endoscope shown in .
[0034] Figure 5K According to one embodiment, Figure 5J Side view of the endoscope shown in .
[0035] Figure 5L FIG. 1 shows a yaw position with an additional undesirable roll offset according to one embodiment. Figure 5J Rear view of the endoscope shown in .
[0036] Figure 5M According to one embodiment, Figure 5L Side view of the endoscope shown in .
[0037] Fig. 6A is a diagram of an electromagnetic tracking system according to one embodiment.
[0038] Figure 6B is an illustration of a camera positioned proximate to an endoscope, according to one embodiment.
[0039] Figure 6C is an illustration of a motion tracking camera including fiducial markers positioned proximate to an endoscope, according to one embodiment.
[0040] Fig.6D is an illustration of an endoscope with a shape sensing optical fiber according to one embodiment.
[0041] Fig. 6E is an illustration of a fluoroscopic imaging system positioned proximate to an endoscope, according to one embodiment.
[0042] Fig. 7AThe length of a guide of an endoscope extending outside of a sheath of the endoscope is shown according to one embodiment.
[0043] Figure 7B The relative roll angle of a guide of an endoscope relative to a sheath of the endoscope is shown according to one embodiment.
[0044] Fig. 8A is a flow chart of a process for automatic calibration of an endoscope according to one embodiment.
[0045] Figure 8B is a flow chart of a process for automatic calibration of an endoscope based on extended length and relative roll angle, according to one embodiment.
[0046] Fig. 9 is a flow chart of a process for intraoperative automatic calibration of an endoscope according to one embodiment.
[0047] The accompanying drawings depict various embodiments of the present invention for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present invention described herein. DETAILED DESCRIPTION
[0048] The methods and devices disclosed herein are well suited for use with one or more endoscopic components or steps as described in U.S. Application Serial No. 14 / 523,760, entitled “SYSTEM FOR ROBOTIC-ASSISTED ENDOLUMENAL SURGERY AND RELATED METHODS,” filed on October 24, 2014 and published as U.S. Patent Application Publication No. US2015 / 0119637, the entire disclosure of which has been previously incorporated herein by reference. The aforementioned application describes system components, endoluminal systems, virtual guide rail structures, mechanism changer joints, instrument device manipulators (IDMs), endoscopic tool designs, consoles, endoscopes, instrument device manipulators, endoluminal navigation, and endoluminal procedures suitable for combination with embodiments disclosed herein.
[0049] I. Surgical Robotic System
[0050] Figure 1 A surgical robotic system 100 is shown according to one embodiment. The surgical robotic system 100 includes a base 101 coupled to one or more robotic arms, such as a robotic arm 102. The base 101 is communicatively coupled to a command console, which is described in Part II. Command Console. Figure 2 The base 101 can be positioned so that the robotic arm 102 can be approached to perform a surgical procedure on a patient, and a user such as a physician can control the surgical robotic system 100 with the help of a command console. In some embodiments, the base 101 can be coupled to a surgical table or bed for supporting a patient. Although for clarity, the base 101 is not shown in FIG. Figure 1 1, but the base 101 may include subsystems such as control electronics, pneumatic elements, power supplies, light sources, etc. The robotic arm 102 includes a plurality of arm segments 110 coupled at joints 111, which provide the robotic arm 102 with a plurality of degrees of freedom, for example, seven degrees of freedom corresponding to seven arm segments. The base 101 may include a power supply 112, pneumatic pressure elements 113, and control and sensor electronics 114 - including components such as a central processing unit, a data bus, control circuits, and memory - and associated actuators such as motors to move the robotic arm 102. The electronics 114 in the base 101 may also process and transmit control signals transmitted from a command console.
[0051] In some embodiments, the base 101 includes wheels 115 to transport the surgical robotic system 100. The mobility of the surgical robotic system 100 helps accommodate space limitations in a surgical operating room and facilitates proper positioning and movement of surgical equipment. In addition, the mobility allows the robotic arm 102 to be configured so that the robotic arm 102 does not interfere with the patient, the physician, the anesthesiologist, or any other equipment. During surgery, the user can control the robotic arm 102 using a control device such as a command console.
[0052] In some embodiments, the robot arm 102 includes a setting joint that uses a combination of brakes and balancers to maintain the position of the robot arm 102. The balancer can include a gas spring or a disc spring. The brake, such as a fail-safe brake, can include mechanical and / or electrical components. In addition, the robot arm 102 can be a gravity-assisted passive support type robot arm.
[0053] Each robotic arm 102 can be coupled to an instrument device manipulator (IDM) 117 using a mechanism converter interface (MCI) 116. The IDM 117 can be removed and replaced with a different type of IDM, for example, a first type of IDM manipulates an endoscope while a second type of IDM manipulates a laparoscope. The MCI 116 includes connectors to transmit pneumatic pressure, power, electrical signals, and optical signals from the robotic arm 102 to the IDM 117. The MCI 116 can be a positioning screw or a baseplate connector. The IDM 117 manipulates surgical instruments such as an endoscope 118 using technologies including direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, etc. The MCI 116 is interchangeable based on the type of IDM 117 and can be customized for a certain type of surgical procedure. The robotic arm 102 can include a joint level torque sensing device and wrist at the distal end, such as a KUKA LBR5 robotic arm.
[0054] Endoscope 118 is a tubular and flexible surgical instrument that is inserted into a patient's anatomy to capture images of the anatomy (e.g., body tissue). In particular, endoscope 118 includes one or more imaging devices (e.g., cameras or sensors) that capture images. The imaging devices may include one or more optical components, such as optical fibers, optical fiber arrays, or lenses. The optical components move with the tip of endoscope 118 so that movement of the tip of endoscope 118 results in changes to the images captured by the imaging devices. Endoscope 118 is referred to in Part III. Endoscopes. FIG. 3A to FIG. 3I Further description.
[0055] The robotic arm 102 of the surgical robotic system 100 uses an elongated motion member to manipulate the endoscope 118. The elongated motion member may include a pull wire, which is also referred to as a pull wire or push wire, cable, fiber, or flexible shaft. For example, the robotic arm 102 actuates multiple pull wires connected to the endoscope 118 to deflect the tip of the endoscope 118. The pull wires may include both metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. The endoscope 118 may exhibit non-ideal behavior in response to the force applied by the elongated motion member. The non-ideal behavior may be due to defects or differences in the stiffness and compressibility of the endoscope 118, and variability in the slack or stiffness between different elongated motion members.
[0056] The surgical robot system 100 includes a computer system 120, such as a computer processor. The computer system 120 includes a calibration module 130, a calibration memory 140, a command module 150, and a data processing module 160. The data processing module 160 can process calibration data collected by the surgical robot system 100. The calibration module 130 can use a gain value based on the calibration data to characterize the non-ideal behavior of the endoscope 118. The computer system 120 and its modules are further described in Part VII. Calibration Processing Flow. The surgical robot system 100 can control the endoscope 118 more accurately by determining the exact value of the gain value. In some embodiments, some or all of the functions of the computer system 120 are performed outside the surgical robot system 100, such as on another computer system or server that is communicatively connected to the surgical robot system 100.
[0057] II. Command Console
[0058] Figure 2 A command console 200 for a surgical robotic system 100 is shown according to one embodiment. The command console 200 includes a console base 201, a display module 202 such as a monitor, and a control module such as a keyboard 203 and a joystick 204. In some embodiments, one or more of the command module 200 functions may be integrated into the base 101 of the surgical robotic system 100 or communicatively coupled to another system of the surgical robotic system 100. A user 205, such as a physician, uses the command console 200 to remotely control the surgical robotic system 100 from an ergonomic position.
[0059] The console base 201 may include a central processing unit, a memory unit, a data bus, and an associated data communication port responsible for interpreting and processing, for example, Figure 1 104. In some embodiments, both the console base 201 and the base 101 perform signal processing for load balancing. The console base 201 may also process commands and instructions provided by the user 205 through the control modules 203 and 204. Figure 2 In addition to the keyboard 203 and joystick 204 shown in FIG, the control module may include other devices that capture gestures and finger postures, such as a computer mouse, a tracking pad, a trackball, a control pad, a video game controller, and sensors (e.g., a motion sensor or a camera).
[0060] The user 205 can use the command console 200 to control a surgical instrument such as the endoscope 118 in a speed mode or a position control mode. In the speed mode, the user 205 directly controls the pitch and yaw motion of the distal end of the endoscope 118 based on direct manual control using the control module. For example, the motion on the joystick 204 can be mapped to the yaw and pitch motion in the distal end of the endoscope 118. The joystick 204 can provide tactile feedback to the user 205. For example, the joystick 204 vibrates to indicate that the endoscope 118 cannot be translated or rotated further in a certain direction. The command console 200 can also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate that the endoscope 118 has reached a maximum translation or rotation.
[0061] In the position control mode, the command console 200 uses a three-dimensional (3D) mapping of the patient and a predetermined computer model of the patient to control a surgical instrument, such as an endoscope 118. The command console 200 provides control signals to the robotic arm 102 of the surgical robotic system 100 to maneuver the endoscope 118 to a target position. Because of its reliance on 3D mapping, the position control mode requires an accurate mapping of the patient's anatomy.
[0062] In some embodiments, the user 205 can manually manipulate the robotic arm 102 of the surgical robotic system 100 without using the command console 200. During setup in a surgical operating room, the user 205 can move the robotic arm 102, endoscope 118, and other surgical equipment to access the patient. The surgical robotic system 100 can rely on force feedback and inertial control from the user 205 to determine the appropriate placement of the robotic arm 102 and equipment.
[0063] The display module 202 may include an electronic monitor, a virtual reality viewing device such as sunglasses or glasses, and / or other display devices. In some embodiments, the display module 202 is integrated with the control module as a tablet device such as a touch screen. In addition, the user 205 can use the integrated display module 202 and control module to view data and input commands to the surgical robot system 100.
[0064] The display module 202 can display the 3D image using a stereoscopic device, such as goggles or sunglasses. The 3D image provides an "inner view" (i.e., an endoscopic view), which is a computer 3D model showing the patient's anatomical structure. The "inner view" provides a virtual environment inside the patient's body and the desired position of the endoscope 118 inside the patient's body. The user 205 compares the "inner view" model with the actual image captured by the camera to help mentally locate and confirm that the endoscope 118 is in the correct position or approximately the correct position inside the patient's body. The "inner view" provides information about the anatomical structure around the distal end of the endoscope 118, such as the shape of the patient's intestines or colon. The display module 202 can simultaneously display the 3D model and the computed tomography (CT) scan of the anatomical structure around the distal end of the endoscope 118. In addition, the display module 202 can overlay the predetermined optimal navigation path of the endoscope 118 on the 3D model and the CT scan.
[0065] In some embodiments, a model of the endoscope 118 is displayed along with the 3D model to help indicate the status of the surgical procedure. For example, a CT scan identifies a lesion in an anatomical structure that may require a biopsy. During operation, the display module 202 can display a reference image corresponding to the current position of the endoscope 118 captured by the endoscope 118. The display module 202 can automatically display different views of the model of the endoscope 118 based on user settings and a particular surgical procedure. For example, when the endoscope 118 approaches the surgical area of the patient, the display module 202 displays a top-down fluorescent view of the endoscope 118 during the navigation step.
[0066] III. Endoscope
[0067] Figure 3A FIG. 1 shows multiple degrees of motion of endoscope 118 according to one embodiment. Endoscope 118 is Figure 1 An embodiment of the endoscope 118 is shown in FIG. Figure 3A , the tip 301 of the endoscope 118 is oriented with zero deflection relative to a longitudinal axis 306 (also referred to as a roll axis 306). To capture images at different orientations of the tip 301, the surgical robotic system 100 deflects the tip 301 about a positive yaw axis 302, a negative yaw axis 303, a positive pitch axis 304, a negative pitch axis 305, or a roll axis 306. The tip 301 or body 310 of the endoscope 118 can be extended or translated along the longitudinal axis 306, the x-axis 308, or the y-axis 309.
[0068] Figure 3Bis a top view of an endoscope 118 including a sheath component and a guide component according to one embodiment. The endoscope 118 includes a guide 315 tubular component, which is embedded or partially embedded inside a sheath 311 tubular component and is longitudinally aligned with the sheath 311 tubular component. The sheath 311 includes a proximal sheath segment 312 and a distal sheath segment 313. The outer diameter of the guide 315 is smaller than the outer diameter of the sheath 311, and the guide 315 includes a proximal guide segment 316 and a distal guide segment 317. The sheath base 314 and the guide base 318 actuate the distal sheath segment 313 and the distal guide segment 317, respectively, based on, for example, a control signal from a user of the surgical robot system 100. The sheath base 314 and the guide base 318 are, for example Figure 1 The construction, composition, properties and use of the distal guide segment 317, which may also be referred to as the flexure segment, are disclosed in U.S. Patent Application No. 14 / 201,610 filed on March 7, 2014 and U.S. Patent Application No. 14 / 479,095 filed on September 5, 2014, the entire contents of which are incorporated herein by reference.
[0069] Both the sheath base 314 and the guide base 318 include a drive mechanism (e.g., as described in Section III.D. Instrument Manipulators). FIG. 4A to FIG. 4D 314 and 315. The pull wires may be coupled to the sheath 311 and the guide 315 using an independent drive mechanism (described further below) to control the pull wires coupled to the sheath 311 and the guide 315. For example, the sheath base 314 generates a tensile load on the pull wire coupled to the sheath 311 to deflect the distal sheath segment 313. Similarly, the guide base 318 generates a tensile load on the pull wire coupled to the guide 315 to deflect the distal guide segment 317. Both the sheath base 314 and the guide base 318 may also include connectors for transmitting pneumatic pressure, electricity, electrical signals, or optical signals from the IDM to the sheath 311 and the guide 314, respectively. The pull wire may include a steel coiled tube within the sheath 311 or the guide 315 along the length of the pull wire that transmits axial compression back to the origin of the load, such as the sheath base 314 or the guide base 318, respectively.
[0070] Due to the multiple degrees of freedom provided by the pull wires connected to the sheath 311 and the guide 315, the endoscope 118 can easily navigate the patient's anatomical structure. For example, four or more pull wires can be used in the sheath 311 and / or the guide 315, providing eight or more degrees of freedom. In other embodiments, up to three pull wires can be used, providing up to six degrees of freedom. The sheath 311 and the guide 315 can be rotated up to 360 degrees along the longitudinal axis 306, providing more degrees of motion. The combination of rotation angles and multiple degrees of freedom provides the user of the surgical robotic system 100 with user-friendly and instinctive control of the endoscope 118.
[0071] III.A. Endoscope Sheath
[0072] Figure 3C is a cross-sectional side view of a sheath 311 of an endoscope 118 according to one embodiment. The sheath 311 includes a lumen 323 sized to accommodate a Figure 3B 315. The sheath 311 includes a wall 324 with pull wires 325 and 326 extending through conduits 327 and 328 located within the length of the wall 324. The conduits include a helical section 330 and a distal non-helical section 329. Proper tensioning of the pull wire 325 can compress the distal end 320 in the positive y-axis direction while minimizing bending of the helical section 330. Similarly, proper tensioning of the pull wire 326 can compress the distal end 320 in the negative y-axis direction. In some embodiments, the lumen 323 is non-concentric with the sheath 311.
[0073] The pull wires 325 and 326 do not necessarily extend in a straight line throughout the length of the sheath 311. Rather, the pull wires 325 and 326 spiral around the sheath 311 along the helical segment 330 and extend longitudinally in a straight line (i.e., generally parallel to the longitudinal axis 306) along the distal non-helical segment 329 and any other non-helical segments of the sheath 311. The helical segment 330 may start and end at any position along the length of the sheath 311. In addition, the length and pitch of the helical segment 330 may be determined based on the desired performance of the sheath 311, for example, the flexibility of the sheath 311 and the friction in the helical segment 330.
[0074] Despite Figure 3C The middle pull wires 325 and 326 are positioned 180 degrees relative to each other, but it should be noted that the pull wires of the sheath 311 can be positioned at different angles. For example, the three pull wires of the sheath can each be positioned 120 degrees relative to each other. In some embodiments, the pull wires are unequally spaced relative to each other, that is, there is no constant angular offset.
[0075] III.B. Helical Segment
[0076] Figure 3D is an isometric view of a helical section 330 of a sheath 311 of an endoscope 118 according to one embodiment. Figure 3D Only one puller wire 325 is shown for the purpose of distinguishing the distal non-helical section 329 from the helical section 330. In some embodiments, the helical section 330 has a variable pitch.
[0077] Figure 3E is another isometric view of the helical segment 330 of the sheath 311 of the endoscope 118 according to one embodiment. Figure 3E Four pull wires 325 , 326 , 351 , and 352 are shown extending along the distal non-helical section 329 and the variable pitch helical section 330 .
[0078] The helical section 330 in the sheath 311 and the guide 315 of the endoscope 118 assist the surgical robotic system 100 and / or the user in navigating the endoscope 118 through a non-linear path in the patient's anatomy, such as the intestine or colon. When navigating a non-linear path, it is useful for the endoscope 118 to remain flexible while still having a controllable distal end segment (in both the sheath 311 and the guide 315). Additionally, it is advantageous to reduce the amount of undesirable bending along the endoscope 118. In previous endoscope designs, tensioning the pull wires to manipulate the distal end segment produced undesirable bending and torque along the length of the endoscope, which may be referred to as forced alignment and curvilinear alignment, respectively.
[0079] Figure 3F is a side view of a sheath 311 with a helical section 330 of an endoscope 118 according to one embodiment. Figure 3F to Figure 3G 300 helps substantially mitigate forced alignment and curvilinear alignment. As the pull wire 325 spirals around the length of the spiral section 330, the pull wire 325 radially and symmetrically distributes the compressive load 335 in multiple directions around the longitudinal axis 306. In addition, the bending moment applied to the endoscope 118 is also symmetrically distributed around the longitudinal axis 306, which balances and cancels out the opposing compressive and tensile forces. The distribution of the bending moment results in minimal net bending and rotational forces, creating a low potential energy state for the endoscope 118, and thus eliminating or substantially mitigating forced alignment and curvilinear alignment.
[0080] The pitch of the helical segment 330 may affect the friction and stiffness of the helical segment 330. For example, the helical segment 330 may be shorter to allow for a longer distal non-helical segment 329, resulting in less friction and / or stiffness of the helical segment 330.
[0081] Figure 3G According to one embodiment, Figure 3FAnother view of the sheath 311 of the endoscope 118 is shown in FIG. Figure 3F Compared to the distal non-helical section 329 shown in FIG. Figure 3G The distal non-helical section 329 shown in FIG. 3 is deflected at a greater angle.
[0082] III.C. Endoscope Guide
[0083] Figure 3H 34 is a cross-sectional side view of a guide 315 of an endoscope 118 according to one embodiment. The guide 315 includes at least one working channel 343 and pull wires 344 and 345 extending through tubes 341 and 342, respectively, along the length of a wall 348. The pull wires 344 and 345 and the tubes 341 and 342 are respectively connected to the guide 315. Figure 3C The pull wires 325 and 326 and the tubes 327 and 328 in the embodiment are substantially the same. For example, similar to the sheath 311 described above, the pull wires 344 and 345 can have a helical section that helps to alleviate the forced alignment and curved alignment of the guide 315.
[0084] Fig. 3I According to one embodiment, Figure 3H A cross-sectional isometric view of the distal tip of a guide 315 of an endoscope 18 is shown in FIG. The guide 315 includes an imaging device 349 (e.g., a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) camera, an imaging fiber bundle, etc.), a light source 350 (e.g., a light emitting diode (LED), an optical fiber, etc.), at least two pull wires 344 and 345, and at least one working channel 343 for other components. For example, the other components include a camera wire, an insufflation device, a suction device, electrical wires, optical fibers, an ultrasonic transducer, an electromagnetic (EM) sensing component, and an optical coherence tomography (OCT) sensing component. In some embodiments, the guide 315 includes a pocket hole to accommodate insertion of components into the working channel 343. As shown in FIG. Fig. 3I As shown in FIG. 3 , pull wires 344 and 345 are not concentric with imaging device 349 or working channel 343 .
[0085] III.D. Instrument Manipulator
[0086] Figure 4A4 is an isometric view of an instrument device manipulator 117 of a surgical robotic system 100 according to one embodiment. The robotic arm 102 is coupled to the IDM 117 via an articulated joint 401. The IDM 117 is coupled to an endoscope 118. The articulated joint 401 can transmit pneumatic pressure, electrical power signals, control signals, and feedback signals from the robotic arm 102 to the IDM 117 and vice versa. The IDM 117 may include a gear head, a motor, a rotary encoder, a power circuit, and a control circuit. A tool base 403 for receiving control signals from the IDM 117 is coupled to the proximal end of the endoscope 118. Based on the control signals, the IDM 117 manipulates the endoscope 118 by actuating an output shaft, which will be described below with reference to FIG. Figure 4B is further described.
[0087] Figure 4B According to one embodiment, Figure 4A An exploded isometric view of the instrument device manipulator shown in FIG. Figure 4B As shown in FIG. 1 , endoscope 118 has been removed from IDM 117 to reveal output shafts 405 , 406 , 407 , and 408 .
[0088] Figure 4C According to one embodiment, Figure 4A 1 is an isometric view of the independent drive mechanism of the instrument device manipulator 117 shown in . The independent drive mechanism can tighten or loosen the (e.g., independent of each other) pull wires 421, 422, 423, and 424 of the endoscope by rotating the output shafts 405, 406, 407, and 408 of the IDM 117, respectively. Just as the output shafts 405, 406, 407, and 408 transmit force downward to the pull wires 421, 422, 423, and 424, respectively, through angular motion, the pull wires 421, 422, 423, and 424 transmit force back to the output shafts. The IDM 117 and / or the surgical robot system 100 can measure the transmitted force using sensors, such as strain gauges described further below.
[0089] Figure 4D A conceptual diagram is shown, which shows how the Figure 4C 431 to measure the force. The force 431 may be directed away from the output shaft 405 of the motor mount 433 coupled to the motor 437. Thus, the force 431 results in a horizontal displacement of the motor mount 433. Additionally, the strain gauge 434, which is horizontally coupled to the motor mount 433, experiences strain in the direction of the force 431. The strain may be measured as the ratio of the horizontal displacement of the tip 435 of the strain gauge 434 to the total horizontal width 436 of the strain gauge 434.
[0090] In some embodiments, the IDM 117 includes additional sensors, such as an inclinometer or an accelerometer, to determine the orientation of the IDM 117. Based on the measurements from the additional sensors and / or the strain gauge 434, the surgical robotic system 100 can calibrate the readings from the strain gauge 434 to account for gravity load effects. For example, if the IDM 117 is oriented on the horizontal side of the IDM 117, the weight of certain components of the IDM 117 may cause strain on the motor mount 433. Therefore, without accounting for gravity load effects, the strain gauge 434 can measure strain that is not caused by strain on the output shaft.
[0091] IV. Non-ideal endoscope motion
[0092] Figure 5A Pull wires inside the endoscope 118 are shown according to one embodiment. The endoscope 118 may include different numbers of pull wires depending on the configuration of the endoscope, but for the sake of example, the following description assumes that the endoscope 118 includes a configuration of four pull wires 421, 422, 423, and 423, and the four pull wires 421, 422, 423, and 423 each correspond to a direction of movement along the yaw axis 510 and the pitch axis 520. Specifically, pulling the pull wires 421, 422, 423, and 423 respectively moves the endoscope 118 in a positive pitch direction, a positive yaw direction, a negative pitch direction, and a negative yaw direction. Although Figure 5A The pull wires shown in the drawings are each aligned with either the yaw 510 or pitch 520 directions, but in other embodiments, the pull wires need not be aligned along these axes, and the above axes are arbitrarily selected for ease of explanation. For example, the pull wires can be aligned with (e.g., intersect) point 540 in the endoscope 118. Thus, translating the pull wires will cause the endoscope 118 to move in the yaw 510 and pitch 520 directions. Throughout the described exemplary embodiments, the pull wires are generally parallel to the roll 530 axis when the endoscope 118 is in a stationary position.
[0093] The endoscope 118 may include one or more spatial sensors 550 coupled toward the distal tip of the endoscope 118. The spatial sensor 550 may be, for example, an electromagnetic (EM) sensor, an accelerometer, a gyroscope, a fiducial marker, and / or other types of sensors. In one embodiment, the spatial sensor 550 is a shape sensing optical fiber embedded inside the endoscope 118 and extending along the length of the endoscope 118. The spatial sensor 550 may provide spatial data indicating the position and / or orientation of the endoscope 118, for example, in real time. The spatial data may also be used as calibration data to assist in the calibration of the endoscope 118.
[0094] In an ideal endoscope, the pullwires of a translating endoscope move the endoscope precisely to a target position or orientation, e.g., bending the tip of the endoscope 90 degrees in positive pitch. In reality, however, due to imperfections in the endoscope, the target motion does not necessarily match the actual motion of the endoscope, and the endoscope may exhibit nonlinear behavior. Imperfections may occur for a variety of reasons, examples of which may be defects in manufacturing (e.g., a pullwire is not properly aligned with the axis of motion), differences in pullwires (e.g., a pullwire is stiffer than another pullwire or is a different length), or differences in endoscope materials (e.g., pitch bends more easily than yaw).
[0095] IV.A. Non-ideal deviation in pitch direction
[0096] FIG. 5B to FIG. 5D Three views of the endoscope 118 are shown in a rest position. Figure 5E to Figure 5G The same three views are shown after the endoscope 118 has moved to the deflected position in response to a command to articulate to a target deflection of 90 degrees in the positive pitch 520 direction. Figure 5E to Figure 5G As shown in , the actual deflection of the endoscope 118 exhibits a non-ideal offset in the positive pitch 520 direction.
[0097] Figure 5B A rear view of an endoscope 118 in a resting position is shown according to one embodiment. In the rear view, the viewer is looking down from the proximal end of the endoscope, where the opposite distal end is to be inserted into the body of a patient. The cross section of the endoscope 118 is aligned with the origin of the yaw 510 and pitch 520 axes. The endoscope 118 is parallel to the roll axis 530, and the spatial sensor 550 is coupled toward the tip of the endoscope 118. Figure 5C According to one embodiment, Figure 5B 1. As an illustrative example, a patient lies flat on a table horizontally for a surgical procedure. Endoscope 118 is positioned parallel to the patient's body, and surgical robotic system 100 inserts endoscope 118 into the body while maintaining the parallel configuration. In the top view, an observer is looking down from above the patient's body. Figure 5D According to one embodiment, Figure 5B A side view of the endoscope 118 is shown in FIG.
[0098] Figure 5E The diagram shows a deflected position according to one embodiment. Figure 5B A rear view of the endoscope 118 is shown in FIG. Fig. 5F According to one embodiment, Figure 5E A top view of the endoscope 118 is shown in FIG. Figure 5G According to one embodiment, Figure 5E. The dashed outline of the endoscope 118 indicates the target deflection position to which the endoscope should move in response to the command, for example, the tip of the endoscope 118 is hypothetically deflected 90 degrees in the positive pitch direction to become parallel to the pitch 520 axis. However, the actual deflection position does not reach the 90 degree deflection and therefore exhibits an undesirable offset in the positive pitch 520 direction.
[0099] IV.B. Non-ideal deviation in yaw direction
[0100] Figure 5H FIG. 1 shows a schematic diagram of a rotatable device in a deflected position with an additional non-ideal offset according to one embodiment. Figure 5B . In particular, in addition to the non-ideal offset in the positive pitch 520 direction, Figure 5H The endoscope shown in also exhibits an additional non-ideal deviation in the positive yaw direction. Fig. 5F In contrast to the "straight" distal end of the endoscope shown in , the distal end (eg, tip) of endoscope 118 is "curved." Figure 5H The endoscope 118 shown in FIG. 1 has defects in two directions (positive pitch and positive yaw), however in other embodiments, the endoscope may exhibit defects in any number of directions (e.g., negative pitch and yaw, as well as roll).
[0101] Fig.5I According to one embodiment, Figure 5H A top view of the endoscope 118 is shown in FIG.
[0102] IV.C. Non-ideal deviation in the rolling direction
[0103] Figure 5J to Figure 5K Two views of the endoscope 118 in a resting position are shown. For the purpose of illustrating the alignment of the endoscope 118 with respect to the yaw 510 and pitch 520 directions, four markers 560 are shown on the endoscope 118. In the resting position, each of the markers is aligned with the yaw 510 and pitch 520 axes.
[0104] Figure 5L to Figure 5M The same two views are shown after the endoscope 118 has been moved to the deflected position in response to a command to articulate to a target deflection of 90 degrees in the positive pitch 520 direction. Figure 5L to Figure 5M, the actual deflection of the endoscope 118 exhibits a non-ideal offset in the roll direction 530 (and in this example, no other non-ideal offsets). The dashed outline of the endoscope 118 indicates the target deflection position to which the endoscope should move in response to the command, for example, the tip of the endoscope 118 hypothetically deflects 90 degrees to become parallel to the pitch axis 520. The actual deflection position has a 90 degree deflection, but also has a rotation along the roll 530 axis. Therefore, the four markers 560 are no longer aligned with the yaw 510 and pitch 520 axes. Similar to Figure 5E The endoscope shown in Figure 5L The distal end of the endoscope in the embodiment is "straight" rather than "curved". In some embodiments, a rotation of the proximal end of the endoscope is accompanied by a corresponding rotation of the distal end of the endoscope (and vice versa). The rotations may be the same or different, for example, a 10 degree roll offset at the proximal end results in a 20 degree roll offset at the distal end. As another example, there may be no roll offset at the proximal end and there may be a non-zero roll offset at the distal end.
[0105] Figure 5M According to one embodiment, Figure 5L Side view of the endoscope shown in .
[0106] In conclusion, Figures 5B to 5G A non-ideal offset in the positive pitch direction is shown. FIG. 5H to FIG. 5I shows non-ideal shifts in yaw directions other than the positive pitch direction, and Figure 5J to Figure 5M The non-ideal deviation in the roll direction is shown. In other embodiments, the endoscope can exhibit non-ideal deviation in any number of directions or combinations of directions. The magnitude of the deviation can vary between different directions.
[0107] V. Spatial Sensor
[0108] VA electromagnetic sensor
[0109] Fig. 6A is an illustration of an electromagnetic tracking system according to one embodiment. A spatial sensor 550 coupled to the tip of the endoscope 118 includes one or more EM sensors 550 that detect an electromagnetic field (EMF) generated by one or more EMF generators 600 near the endoscope 118. The intensity of the detected EMF is a function of the position and / or orientation of the endoscope 118. For example, if the endoscope 118 includes more than one EM sensor 550, a first EM sensor is coupled to a guide tubular member and a second EM sensor is coupled to a sheath tubular member of the endoscope.
[0110] One or more EMF generators 600 are located outside the patient's body. The EMF generators 600 emit EM fields that are picked up by the EM sensors 550.
[0111] If multiple EMF generators 600 and / or EM sensors 550 are used, they may be modulated in a variety of different ways so that when their transmit / receive fields are processed by the computer system 120 (or any computer system external to the surgical robotic system 100), the signals are separable. Thus, the computer system 120 may process multiple signals (sent and / or received), each as a separate input, to provide separate triangulated positions about the positions of the EM sensors 550 and, by extension, the positions of the endoscope 118. For example, multiple EMF generators 600 may be modulated in time or frequency, and orthogonal modulation may be used so that each signal is completely separable from each other signal (e.g., using signal processing techniques such as filtering and Fourier transforms) despite possible overlap in time. Furthermore, multiple EM sensors 550 and / or EMF generators 600 may be oriented at non-zero, non-orthogonal angles relative to each other in Cartesian space, such that changes in the orientation of the EM sensors 550 will result in at least one of the EM sensors 550 receiving at least some signals from one or more of the EMF generators 600 at any given moment. For example, each EMF generator 600 may be offset at a small angle (e.g., 7 degrees) along any axis from each of the two other EM generators 600 (and similarly to the plurality of EM sensors 550). As many EMF generators or EM sensors as desired may be used in this arrangement to ensure accurate EM sensor location information along all three axes, and if desired, at multiple points along the endoscope 118.
[0112] VB Camera Sensor
[0113] Figure 6B1 is an illustration of a camera near an endoscope 118 according to one embodiment. The camera may include any type of optical camera, such as a digital camera, a stereo camera, a high-speed camera, a light field camera, etc. The first camera 610 is parallel to the longitudinal axis of the endoscope 118. The second camera 620 is orthogonal to the first camera 610. Since each camera captures an image frame showing the position and / or orientation of the endoscope in at least two dimensions, aligning the two cameras orthogonally to each other enables the surgical robot system 100 to receive at least three-dimensional (e.g., corresponding to pitch, yaw, and roll axes) information about the endoscope. In other embodiments, three or more cameras may be used to capture images of the endoscope 118. The data processing module 160 may implement object tracking image processing techniques by determining the real-time 3D position of the endoscope 118 using the captured image frames. Exemplary techniques include correlation-based matching methods, feature-based methods, and optical flow.
[0114] Figure 6C is an illustration of motion cameras including fiducial markers near endoscope 118 according to one embodiment. Spatial sensor 550 coupled toward the distal end of endoscope 118 is a fiducial marker. Motion cameras 630 and 640 capture image frames tracking the position and motion of the fiducial markers. Figure 6C Two fiducial markers and two motion cameras are shown in FIG, but other embodiments may include any other number of fiducial markers coupled to the endoscope and / or the motion camera to track the fiducial markers.
[0115] and Figure 6B In contrast to the camera in Figure 6C The motion camera in captures data describing the motion of the fiducial markers. Therefore, in some embodiments, the data processing module 160 requires fewer computing resources to process the motion camera data compared to processing data captured from other optical cameras without using fiducial markers. For example, other optical cameras capture data describing the visual appearance (e.g., color and size) of the endoscope 118. However, the motion camera may only need to capture the real-time coordinate position of each fiducial marker, which is sufficient for the data processing module 160 to use to determine the overall movement of the endoscope 118 in different directions (e.g., pitch, yaw, and roll) in response to commands from the surgical robotic system 100.
[0116] Camera-based sensors may be more suitable for determining the position of an endoscope outside the patient's body, while EM sensors may be more suitable for situations where the endoscope is used inside the body. In some embodiments, image processing techniques using camera data provide, for example, more accurate or higher resolution position and motion data than EM sensor-based techniques because a user of the endoscope viewing the outside of the body can verify the results of the image processing techniques. In contrast, an advantage of EM sensors is that they can still detect EM fields generated by an EMF generator even when the endoscope is inside the patient's body.
[0117] VC Shape Sensing Fiber
[0118] Fig.6D is an illustration of an endoscope 118 with a shape sensing fiber according to one embodiment. The spatial sensor 550 is a shape sensing fiber embedded in the interior of the endoscope 118. A console 650 positioned near the endoscope 118 is coupled to the shape sensing fiber. The console 650 transmits light through the shape sensing fiber and receives light reflected from the shape sensing fiber. The shape sensing fiber may include a segment of a fiber Bragg grating (FBG). The FBG reflects light of a specific wavelength while transmitting light of other wavelengths. The console 650 generates reflection spectrum data based on the wavelength of light reflected by the FBG.
[0119] The data processing module 160 can analyze the reflected spectral data to generate position and orientation data of the endoscope 118 in two or three-dimensional space. In particular, as the endoscope 118 bends, the shape sensing fiber embedded inside also bends. The specific wavelength of light reflected by the FBG changes based on the shape of the shape sensing fiber (e.g., the shape of a "straight" endoscope is different from that of a "bent" endoscope). Therefore, the data processing module 160 can determine (e.g., in response to a command from the surgical robotic system 100) how many degrees the endoscope 118 has been bent in one or more directions, for example by identifying differences in the reflected spectral data. Similar to EM sensors, shape sensing fibers are suitable for data acquisition within a patient's body because no line of sight to the shape sensing fiber is required.
[0120] VD Fluoroscopic Imaging
[0121] Fig. 6E1 is an illustration of a fluoroscopic imaging system 660 in the vicinity of an endoscope 118 according to one embodiment. The endoscope 118 is inserted into a patient 670 who is undergoing a surgical procedure via the robotic arm 102. The fluoroscopic imaging system 660 is a C-arm that includes a generator, a detector, and an imaging system (not shown). The generator is coupled to the bottom end of the C-arm and faces upward toward the patient 670. The detector is coupled to the top end of the C-arm and faces downward toward the patient 670. The generator emits X-ray waves toward the patient 670. The X-ray waves penetrate the patient 670 and are received by the detector. Based on the received X-ray waves, the fluoroscopic imaging system 660 generates an image of a body part or other object, such as the endoscope 118, within the body of the patient 670. In contrast to the optical camera described in Section VB Camera Sensor that captures an image of the actual endoscope, the fluoroscopic imaging system 660 generates an image of an object within the body of the patient 670 that includes an outline representing, for example, the shape of the endoscope based on reflected X-rays. Thus, the data processing module 160 may use similar image processing techniques as previously described, such as optical flow, to determine the position and movement of the endoscope, for example, in response to commands from the surgical robotic system 100 .
[0122] VI. Insertion and Roll Offset
[0123] Fig. 7A A length 700 of the guide 315 of the endoscope 118 extending outside the sheath 311 of the endoscope 118 is shown according to one embodiment. As the length 700 increases, the flexibility of the distal end of the guide 315 increases because the length 700 is not significantly enclosed by the sheath 311. In contrast, the portion of the guide 315 radially enclosed by the sheath 311 has less flexibility because the material of the sheath 311 provides greater rigidity. In some embodiments, because the physical properties of the endoscope change based on the extended length, the surgical robotic system 100 needs to provide commands that move the endoscope that take into account the extension. For example, as the extension increases, the distal end of the endoscope may become heavier (and / or more flexible) because more of the length of the guide is outside the sheath. Therefore, in order to achieve the same bending motion, the surgical robotic system 100 may need to provide commands that translate the pull wire of the endoscope to a greater extent relative to commands that move an endoscope with a smaller extended length.
[0124] Figure 7BThe relative roll angle 710 of the guide 315 of the endoscope 118 relative to the sheath 311 of the endoscope 118 is shown according to one embodiment. For example, due to differences in the materials of the endoscope 118, the guide 315 and / or the sheath 311 can be more flexible in certain directions than in other directions. Therefore, based on the relative roll angle 710, the flexibility of the endoscope 118 can change in a particular direction. In addition to considering the length of the extension as described above, the surgical robotic system 100 may also need to consider the relative roll angle when providing commands to move the endoscope. For example, when the relative roll angle is 5 degrees, a command to bend the endoscope 90 degrees may result in an actual bend of 80 degrees, but when the relative roll angle is -5 degrees, it may result in an actual bend of 100 degrees.
[0125] VII. Calibration
[0126] VII.A. Overview
[0127] The surgical robot system 100 performs a calibration process to determine a gain value that compensates for defects in the behavior of the endoscope. During the calibration process, the surgical robot system 100 moves the endoscope to one or more target positions (or angles) by translating one or more pull wires according to one or more commands. The surgical robot system 100 receives spatial data indicating the actual position and orientation of the endoscope obtained in response to the command, wherein the actual position may be different from the target position due to defects in the endoscope. The surgical robot system 100 determines the gain value based on the command, the target position expected to be obtained, and the actual position obtained. The surgical robot system 100 can perform such a calibration process, for example, on a production line for quality assurance, or in a laboratory or clinical environment before a surgical procedure. In addition, the surgical robot system 100 can perform such a calibration process while performing a surgical procedure on a patient.
[0128] As a simple illustrative example, the material of a particular endoscope may be stiffer than expected. When performing the calibration process, the spatial data indicates that the endoscope is deflected to an actual position of 30 degrees in the pitch direction, while the target position is 60 degrees. As part of the exemplary calibration process, the surgical robotic system 100 determines that the corresponding gain value for the endoscope is a decimal value of 2.0 because the target position is twice the value of the actual position. In other embodiments, other formats, such as percentages, integers, in degrees, etc., may be used to express the gain value.
[0129] The gain value can be associated with a particular pullwire, an endoscope, a direction of motion (e.g., a positive pitch direction, a negative yaw direction, or a roll direction), and / or other types of factors. In particular, the above example is a simple scenario that assumes a constant gain value of 2.0 for all conditions of the endoscope. However, in practice, calibrating an endoscope is a more complex problem because the gain value depends on a variety of factors, either individually or in combination. For example, the gain value may be 2.0 in the positive pitch direction, 3.0 in the negative pitch direction, 1.5 in the positive yaw direction, etc. In addition, in the same endoscope, for the first pullwire, the gain value may be 2.0 in the positive pitch direction, while for the second pullwire, the gain value may be 2.2. In addition, the gain value may be 2.0 for the first pullwire of the first endoscope, while it may be 2.5 for the first pullwire of the second endoscope.
[0130] In some embodiments, the calibration module 130 receives a length of the guide of the endoscope that extends outside of the sheath of the endoscope (or is radially enclosed by the sheath of the endoscope), and / or a relative roll angle of the guide relative to the sheath. The calibration module 130 further determines a gain value based on the length and / or the relative roll angle. The gain value for a particular length or relative roll angle may be different than the gain value for another length or relative roll angle because the endoscope may be more flexible in a particular direction or section of the endoscope.
[0131] In one embodiment, the endoscope (eg, guide and / or sheath) includes a plurality of segments, each segment having a different stiffness level. The calibration module 130 receives the Young's modulus of at least one segment and further determines the gain value based on the Young's modulus.
[0132] In one embodiment, the complete calibration process involves several sub-calibration processes. For example, the surgical robotic system 100 provides a command to move the endoscope to a target position in a first direction. The calibration module 130 receives calibration data indicating the actual position of the endoscope, which may be different from the target position. The surgical robotic system 100 releases the endoscope back to a stationary position and repeats the data collection process for many other directions. The surgical robotic system 100 may also provide a command to extend the guide to a greater length outside the sheath and repeat the calibration data collection process for many different extension lengths. Similarly, the surgical robotic system 100 may provide a command to rotate the guide to a relative roll angle relative to the sheath and repeat the calibration data collection process for many different relative roll angles.
[0133] The calibration module 130 determines the gain values based on the aggregate calibration data set from each sub-calibration process. As can be seen from the number of potential combinations of factors to be considered during calibration, the calibration process can become a more complex process with many nested loops of different tests. Therefore, it is advantageous to automate the calibration using the surgical robotic system 100, for example, to help keep track of all the factors that need to be tested, reduce the chance of calibration errors or oversights, and eliminate the need for the user to manually perform mechanical tasks for each test.
[0134] In some embodiments, the calibration module 130 stores the calibration data and associated gain values along with one or more other factors (e.g., information about a corresponding command to move the endoscope, a direction of movement, an identifier of a pull wire, a length of the guide relative to the sheath, and / or a relative roll angle, or a unique identifier of the endoscope) in the calibration memory 140. The calibration module 130 can upload the calibration data, gain values, and / or factors to a global calibration database that includes information from multiple endoscopes.
[0135] VII.B. Calibration Model
[0136] The surgical robot system 100 can use one or more types of models to generate commands to appropriately move the endoscope based on the calibration data. Specifically, the command module 150 generates commands for each pull wire of the endoscope based on the parameters of one of the models, wherein the parameters and associated gain values are determined based on the calibration data. For some models, the parameters may be the same as the gain values, while for other models, the surgical robot system 100 may determine the gain values based on the parameters. The model may be associated with a guide, a sheath, or both a guide and a sheath of the endoscope. Implementations of models associated with both a guide and a sheath take into account parameters that describe the interaction between the guide and the sheath, for example, the length and relative roll angle of the extension of the guide relative to the sheath.
[0137] In one embodiment, the calibration module 130 uses an empirical model implemented using a gain value matrix. The gain values are determined empirically by solving a set of linear equations based on calibration data from a previously completed calibration process. The calibration module 130 can multiply a vector representing an input command (e.g., including the target translation of each pull line of the endoscope and the extension and relative roll values) by a gain value matrix, and an output vector representing the adjusted command (e.g., including the modified translation value of one or more pull lines) can be generated. The empirical model gain value can compensate for a specific compression or expansion of the pull line based on the bending of the endoscope. In particular, the distance that a certain pull line travels in the endoscope can be contracted or extended based on the curvature of the endoscope. In some embodiments, the empirical model takes into account the dependency between lines in opposite directions. For example, the first line corresponds to the positive pitch direction and the second line corresponds to the negative pitch direction. Providing slack on the first line while pulling the second line both contributes to the same movement of bending the endoscope in the negative pitch direction.
[0138] In one embodiment, the calibration module 130 uses a physics-based model to determine the effective physical properties of the endoscope when it is bent. A physics-based model has the potential to capture the behavior of the endoscope more comprehensively. In comparison, a simple model may assume that a bent endoscope bends uniformly over a specific length of the endoscope according to a given bending stiffness in that specific length and remains straight over the rest of the length of the endoscope. In addition, a physics-based model may decompose the guide and sheath of the endoscope into segments, each with an associated bending stiffness. The physics-based model also takes into account the stiffness of the pull wire and the effects of the sheath and guide interactions (e.g., extension length and relative roll angle) on the stiffness of any particular segment.
[0139] Using the physics-based model, the computer system 120 can use inverse solid mechanics to convert commands to move the endoscope (e.g., indicating an angle of bending in the pitch and / or yaw directions) into the distance that the surgical robotic system 100 should translate one or more pull wires to obtain the desired motion. In addition, by using the physics-based model, the robotic system 100 can move one or more IDMs to compensate for any unwanted motion of the distal tip of the endoscope due to axial deformation associated with the bending motion.
[0140] In one embodiment, the calibration module 130 uses a model-free inversion process to generate commands to move the endoscope. For example, the calibration module 130 performs a lookup table that does not require the use of gain values and / or parameters. Instead, the lookup table maps previously recorded input values (e.g., commands to move the endoscope in pitch and yaw) to output values (e.g., translation of each pull line of the endoscope) based on the calibration data. If the exact input-to-output mapping is unknown, the lookup table can interpolate (e.g., solved by Delaunay triangulation or other multidimensional triangulation techniques) or extrapolate between data points, for example, a bend of 42 degrees can be interpolated using data points for 40 degrees and 45 degrees. In order to reduce the amount of computing resources required by the computer system 120 to execute the lookup table, the calibration module 130 can minimize the size of the mapped data set by using techniques such as Taylor decomposition or approximating the data set using a Fourier representation.
[0141] VII.C. Exemplary Process Flow
[0142] Fig. 8A is a flow chart of a process 800 for automatic calibration of an endoscope according to one embodiment. Process 800 may include in some embodiments in conjunction with Fig. 8A Steps different from or additional to the steps described, or in combination with Fig. 8A The steps may be performed in a different order than described. Process 800 is particularly useful for calibrating an embodiment of an endoscope that includes four pull wires, for example, each pull wire is 90 degrees apart and corresponds to a positive or negative pitch or yaw direction. However, process 800 may be generalized to any number of pull wires, and reference may be made to the embodiment of the endoscope that includes four pull wires, for example, each pull wire is 90 degrees apart and corresponds to a positive or negative pitch or yaw direction. Figure 8B Because the computer system 120 can automate the process 800, the user does not have to manually perform the calibration process to use the surgical robotic system 100. Automated calibration is advantageous, for example, because the process reduces the time required to calibrate an endoscope.
[0143] The command module 150 provides 804 a command to move the endoscope to a target position in a first direction. The calibration module 130 receives 806 spatial data indicating the actual position and orientation of the endoscope moved in response to the command. The spatial data can be received from a spatial sensor (e.g., coupled to or positioned near the endoscope) such as an accelerometer, gyroscope, fiducial marker, fiber optic cable, camera, or imaging system, as previously described in Section V Spatial Sensors. The spatial data describes the position and / or orientation of the endoscope or a portion of the endoscope in one or more directions of movement. The command module 150 provides 808 a command to release the endoscope to a stationary position.
[0144] The command module 150 provides 810 a command to move the endoscope to the target position in the second direction. The calibration module 130 receives 812 the spatial data. The command module 150 provides 814 a command to release the endoscope to the rest position.
[0145] The command module 150 provides 816 a command to move the endoscope to the target position in the third direction. The calibration module 130 receives 818 the spatial data. The command module 150 provides 820 a command to release the endoscope to the rest position.
[0146] The command module 150 provides 822 a command to move the endoscope to the target position in the fourth direction. The calibration module 130 receives 824 the spatial data. The command module 150 provides 826 a command to release the endoscope to the rest position.
[0147] For each of the four directions, the target position can remain constant. In some embodiments, the target position changes between different directions. For example, the target position is 90 degrees for the first and third directions and 45 degrees for the second and fourth directions. The first, second, third, and fourth directions can be positive pitch, positive yaw, negative pitch, and negative yaw directions in any particular order. In other embodiments, the command moves the endoscope toward the target position in two or more directions simultaneously, for example, 60 degrees in both the positive pitch direction and the positive yaw direction. Although process 800 involves four directions, in other embodiments, the computer system 120 can repeat steps 804 to 808 for any other number of directions (more or less).
[0148] The calibration module 130 determines 828 gain values for the pull wires of the endoscope based on the spatial data for one or more directions. The calibration module 130 can determine a gain value associated with each pull wire. At least one of the gain values can have a value different from uniform. A uniform gain value indicates that the corresponding pull wire exhibits an ideal behavior, for example, the actual movement of the endoscope matches the target movement based on the translation of the corresponding pull wire. In some embodiments, the calibration module 130 retrieves a default gain value for the pull wire (e.g., determined in a previous calibration process) and further determines a gain value based on the default gain value.
[0149] The calibration module 130 stores 830 the gain value in the calibration memory 140. The endoscope may include a computer readable tangible medium, such as a flash memory or a database, to store the gain value. In some embodiments, the command module 150 provides a command to modify the length and / or relative roll angle of the guide relative to the sheath, and the surgical robotic system 100 repeats the steps of process 800 to determine the gain value associated with the modified length and / or relative roll angle.
[0150] Figure 8Bis a flow chart of a process 840 for automatic calibration of an endoscope based on extended length and relative roll angle according to one embodiment. Process 840 may include in some embodiments in conjunction with Figure 8B Steps different from or additional to the steps described, or in combination with Figure 8B The steps are performed in a different order than described. In contrast to process 800, process 840 is generalized to any number of orientations, and any number of extended lengths and relative roll angles of the guide relative to the sheath of the endoscope. For example, instead of an endoscope having four pull wires offset 90 degrees relative to each other, the endoscope may include three pull wires offset 120 degrees relative to each other, or any other configuration with different offset angles (e.g., at pointer positions at 11 o'clock, 2 o'clock, and 6 o'clock).
[0151] The surgical robot system 100 provides 850 a command to move the endoscope to a length or extension and / or a relative roll angle. The surgical robot system 100 performs 860 a calibration with a length or extension and / or a relative roll angle. In step 860, the command module 150 provides 862 a command to move the endoscope to a target position in one direction. The calibration module 130 receives 864 spatial data indicating the actual position and orientation of the endoscope. The command module 150 provides a command 866 to release the endoscope to a static position. The surgical robot system 100 repeats steps 862 to 866 for each direction in a set of directions. In addition, the surgical robot system 100 repeats steps 850 to 860 for each extended length and / or relative roll angle (or a combination of extended length and relative roll angle) in a set of different extended lengths and / or relative roll angles. The calibration module 130 determines 870 a gain value based on the spatial data received from each calibration.
[0152] Fig. 9 is a flow chart of a process 900 for intraoperative automatic calibration of an endoscope according to one embodiment. Process 900 may include in some embodiments in conjunction with Fig. 9 Steps different from or additional to the steps described, or in combination with Fig. 9 The steps may be performed in a different order than described. In some implementations, command console 200 may use process 900 in either the velocity mode or the position control mode as previously described in Section II. Command Console.
[0153] The calibration module 130 retrieves 910 default gain values for an endoscope including a pull wire, a guide, and a sheath. Each pull wire can be associated with one of the default gain values. The surgical robotic system 100 inserts 920 an endoscope into a patient undergoing a surgical procedure, such as ureteroscopy, percutaneous nephrolithotomy (PCNL), colonoscopy, fluoroscopy, prostatectomy, colectomy, cholecystectomy, inguinal hernia, and bronchoscopy. The calibration module 130 receives 930 information about the relative roll angle of the guide relative to the sheath and the length of the guide radially enclosed by the sheath. The information about the relative roll angle and length can be based on a previous command provided to move the endoscope, a default relative roll angle and length value, or data generated by a sensor (e.g., an accelerometer and gyroscope connected to the endoscope). The command module 150 provides a command 940 to move the endoscope by translating at least one of the pull wires.
[0154] The calibration module 130 receives 950 spatial data of the endoscope that has been moved in response to the command. In one embodiment, the spatial data is received from a fluoroscopic imaging system. The fluoroscopic imaging system can capture an image of the endoscope in the patient's body, which enables the surgical robot system 100 to perform process 900 during the surgical procedure. The calibration module 130 determines 960 a new gain value based on the spatial data, the corresponding default gain value, the extended length, the relative roll angle, and / or the command. The calibration module 130 stores 970 the new gain value in the calibration memory 140. The surgical robot system 100 can generate additional commands based on the new gain value determined by process 900. For example, the endoscope moves to an actual position of 80 degrees in response to the first command, where the target position is actually 90 degrees. The command module 150 generates a new command based on the new gain value and provides the new command to move the endoscope using the surgical robot system 100. Since the new command compensates for the angle difference (i.e., 80 degrees is 10 degrees less than 90 degrees), the endoscope moves to an actual position of 90 degrees in response to the new command.
[0155] VIII. Alternative Considerations
[0156] After reading this disclosure, those skilled in the art will understand additional alternative structural and functional designs through the principles disclosed herein. Therefore, although specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise configurations and components disclosed herein. Various modifications, changes and variations that are obvious to those skilled in the art may be made in the arrangement, operation and details of the methods and devices disclosed herein without departing from the spirit and scope defined by the appended claims.
[0157] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.
[0158] The expressions "coupled" and "connected" and their derivatives may be used to describe some embodiments. For example, the term "coupled" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. Unless otherwise expressly stated, the embodiments are not limited to this context.
[0159] As used herein, the terms "comprises," "including," "comprising," "containing," "having," "with," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, unless expressly stated to the contrary, "or" refers to inclusive or non-exclusive. For example, a condition A or B is satisfied by any of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).
[0160] In addition, "a" or "an" is used to describe elements and components of the embodiments herein. This is merely for convenience and to give a general sense of the invention. The description should be understood to include one or at least one, and the singular also includes the plural, unless it is obvious otherwise.
[0161] Some portions of this specification describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to effectively communicate the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent circuits, microcode, etc. In addition, it sometimes proves advantageous to refer to the arrangement of these operations as modules without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.
[0162] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product that includes a computer-readable non-transitory medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described.
[0163] Embodiments of the present invention may also relate to products produced by the computing process described herein. Such products may include information generated by the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of the computer program product described herein or other data combination.
Claims
1. A method for performing a calibration procedure when a surgical procedure is not being performed on a patient, comprising: retrieving a plurality of gain values from a computer readable medium storing calibration data, the plurality of gain values each being associated with a pull wire of a plurality of pull wires of a surgical instrument and including a first gain value of the plurality of gain values associated with a first pull wire of the plurality of pull wires and a second gain value associated with a second pull wire of the plurality of pull wires, wherein the first gain value is different from the second gain value; providing a first command derived at least in part from one or more of the plurality of gain values to move the surgical instrument by translating at least one of the plurality of pull wires using a surgical robotic system; receiving spatial data indicating an actual position of the surgical instrument that has been moved in response to the first command; For at least one of the pullwires, determining a new gain value based on the spatial data; and The new gain value is stored in the computer readable medium.
2. The method according to claim 1, further comprising: generating a second command based on the new gain value; as well as The second command is provided to move the surgical instrument using the surgical robotic system.
3. The method according to claim 1, wherein: Receiving the spatial data includes: positioning a fluoroscopic imaging system proximate to the surgical instrument; and A plurality of fluoroscopic images of the surgical instrument are captured by the fluoroscopic imaging system.
4. The method according to claim 1, wherein: The surgical robot system deflects the surgical instrument to a certain angle in a yaw direction and a pitch direction in response to the first command.
5. The method according to claim 1, wherein: The surgical instrument comprises at least one electromagnetic (EM) sensor coupled to a distal end of the surgical instrument, and wherein the method further comprises: positioning at least one EM field generator proximate to the electromagnetic (EM) sensor; and Wherein receiving the spatial data comprises detecting an electromagnetic (EM) field at the EM sensor, a strength of the EM field being a function of an actual position of the distal end of the surgical instrument comprising the EM sensor.
6. The method according to claim 1, wherein: The surgical instrument includes one or more spatial sensors connected to the distal end of the surgical instrument, the one or more spatial sensors including at least one of an accelerometer or a gyroscope, and wherein receiving the spatial data includes detecting movement in at least one direction through the one or more spatial sensors.
7. The method according to claim 1, wherein: The surgical instrument includes an optical fiber embedded in an interior of the surgical instrument, wherein the method further includes: positioning a console proximate to the surgical instrument, the console coupled to the optical fiber and configured to generate reflectance spectral data based on light reflected by the optical fiber; and Wherein, receiving the spatial data includes analyzing the reflection spectrum data.
8. The method according to claim 1, wherein: The surgical instrument includes a camera lens and a working channel, and wherein the camera lens and the working channel are each non-concentric with each of the plurality of pull wires.
9. The method according to claim 1, wherein: The surgical instrument includes a sheath tubular member including a first puller wire of the plurality of puller wires and a guide tubular member including a second puller wire of the plurality of puller wires.
10. The method according to claim 9, wherein: The guide tubular member and the sheath tubular member each include a plurality of segments.
11. The method according to claim 10, wherein: The first of the plurality of puller wires winds along a first of the plurality of sections of the sheath tubular member.
12. The method according to claim 11, wherein: The second one of the plurality of puller wires winds along a second one of the plurality of sections of the guide tubular member.
13. The method according to claim 9, further comprising: receiving information indicative of a roll angle of the guide tubular member relative to the sheath tubular member and a length of the guide tubular member radially enclosed by the sheath tubular member; and Wherein, the new gain value is determined based on at least one of the roll angle and the length.
14. The method according to claim 1, wherein: Moving the surgical instrument by translating at least one of the plurality of pull wires using the surgical robotic system comprises: Commands are provided to a plurality of robotic arms coupled to the surgical instrument.
15. The method according to claim 1, wherein: At least one of the plurality of pull wires includes a first pull wire and a second pull wire, and wherein the first command includes translating the first pull wire a first distance, the method further comprising providing a second command to translate the second pull wire a second distance.
16. A surgical robot system, comprising: one or more robotic arms; A surgical instrument comprising a plurality of pull wires; A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the following steps, the steps comprising: retrieving a plurality of gain values, the plurality of gain values each being associated with a pull line of the plurality of pull lines, the plurality of pull lines comprising a first gain value of the plurality of gain values associated with a first pull line of the plurality of pull lines and a second gain value associated with a second pull line of the plurality of pull lines, wherein the first gain value is different from the second gain value; providing a first command derived from one or more of the plurality of gain values to move the surgical instrument by translating at least one of the plurality of pull wires using the one or more robotic arms; receiving spatial data indicating an actual position of the surgical instrument that has been moved in response to the first command; For at least one pullwire of the plurality of pullwires, determining a new gain value based on the spatial data; and The new gain value is stored.
17. The surgical robot system according to claim 16, wherein: The steps also include: generating a second command based on the new gain value; and The second command is provided to move the surgical instrument using the one or more robotic arms.
18. The surgical robot system according to claim 16, wherein: Receiving the spatial data includes: A plurality of fluoroscopic images of the surgical instrument captured by a fluoroscopic imaging system are retrieved.
19. The surgical robot system according to claim 16, wherein: The one or more robotic arms deflect the surgical instrument to a certain angle in a yaw direction and a pitch direction in response to the first command.
20. The surgical robot system according to claim 16, wherein: The surgical instrument includes at least one electromagnetic (EM) sensor coupled to a distal end of the surgical instrument, and wherein receiving the spatial data includes: An electromagnetic (EM) field is detected at the EM sensor, the strength of the EM field being a function of the actual position of the distal end of the surgical instrument including the EM sensor relative to an EM field generator.
21. The surgical robot system according to claim 16, wherein: The surgical instrument includes one or more spatial sensors connected to the distal end of the surgical instrument, the one or more spatial sensors including at least one of an accelerometer or a gyroscope, and wherein receiving the spatial data includes detecting movement in at least one direction through the one or more spatial sensors.
22. The surgical robot system according to claim 16, wherein: The surgical instrument includes an optical fiber embedded in an interior of the surgical instrument, and wherein receiving the spatial data includes: The analysis is based on reflectance spectral data generated by light reflected from the optical fiber.
23. The surgical robot system according to claim 16, wherein: The surgical instrument includes a camera lens and a working channel, and wherein the camera lens and the working channel are each non-concentric with each of the plurality of pull wires.
24. The surgical robot system according to claim 16, wherein: The surgical instrument includes a sheath tubular member including a first puller wire of the plurality of puller wires and a guide tubular member including a second puller wire of the plurality of puller wires.
25. The surgical robot system according to claim 24, wherein: The guide tubular member and the sheath tubular member each include a plurality of segments.
26. The surgical robot system according to claim 25, wherein: The first of the plurality of puller wires winds along a first of the plurality of sections of the sheath tubular member.
27. The surgical robot system according to claim 26, wherein: The second one of the plurality of puller wires winds along a second one of the plurality of sections of the guide tubular member.
28. The surgical robot system according to claim 24, further comprising: receiving information indicative of a roll angle of the guide tubular member relative to the sheath tubular member and a length of the guide tubular member radially enclosed by the sheath tubular member; and Wherein, the new gain value is determined based on at least one of the roll angle and the length.
29. The surgical robot system according to claim 16, wherein: The one or more robotic arms include at least a first robotic arm and a second robotic arm, and wherein a first command is provided to the first robotic arm; and wherein the non-transitory computer-readable storage medium also stores instructions that, when executed by the processor, cause the processor to provide a second command different from the first command to the second robotic arm.
30. The surgical robot system according to claim 16, wherein: At least one of the plurality of pull wires includes a first pull wire and a second pull wire, and wherein the first command includes The first puller wire is caused to translate a first distance, and wherein the non-transitory computer readable storage medium further stores instructions that, when executed by the processor, cause the processor to provide a second command to translate the second puller wire a second distance.
