Systems and methods for orientation detection and tool installation

Abstract

Systems and methods for orientation detection and tool installation. A system includes a tool and a conduit sized to receive the tool. The system also includes a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions cause the system to receive a plurality of images from within a lumen of the catheter via an image sensor of the tool. Each image includes a plurality of pixels. For each image, a subset of the plurality of pixels associated with the observable feature is identified. For each image, a possible orientation of the observable feature is determined based on the identified subset of the plurality of pixels. A composite orientation of the observable feature is determined based on a combination of possible orientations of the observable feature for each image. A rotational offset of the tool relative to the catheter is determined based on the determined composite orientation.

Description

[0001] This application is a divisional application of Chinese patent application 201980055906.4, filed on July 10, 2019, entitled “System and method for orientation detection and tool installation”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 696,178, filed on July 10, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to systems and methods for detecting and identifying the orientation of a tool in a catheter and, in various embodiments, may include determining proper installation of a tool in a catheter. Background Art

[0005] Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort and harmful side effects. This minimally invasive technique can be performed through the natural orifices of the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, a physician can insert a minimally invasive medical device (including surgical instruments, diagnostic instruments, therapeutic instruments or biopsy instruments) to reach the target tissue position. One such minimally invasive technique is to use a flexible and / or steerable elongated device (such as a flexible catheter) that can be inserted into an anatomical passage and directed to the region of interest in the patient's anatomical structure. Controlling this elongated device by medical staff involves the management of several degrees of freedom, including at least the insertion and retraction of the management elongated device and the steering of the device.

[0006] Therefore, it would be advantageous to develop improved systems and methods for use during minimally invasive medical techniques. Summary of the invention

[0007] Some embodiments of the invention are best summarized in the claims appended to the description.

[0008] In accordance with some embodiments, a method for determining the orientation of a tool at least partially mounted within a catheter is provided. The method may include capturing a first image within the lumen of the catheter via an image sensor of the tool, the first image comprising a plurality of pixels. The method may further include identifying a first subset of the plurality of pixels comprising observable features and a second subset of the plurality of pixels comprising a background color. The method may further include adjusting the second subset of the plurality of pixels to a neutral color. The method may further include generating a modified image by filtering the plurality of pixels to remove the second subset. The method may further include determining the angular orientation of the observable feature in the modified image. The method may further include determining a rotational offset of the tool relative to the catheter based on the angular orientation of the observable feature.

[0009] In accordance with some embodiments, a system for determining the orientation of a tool in a catheter is provided. The system may include a tool shaped to be positioned in a lumen of the catheter. The system may further include an imaging sensor positioned in the tool. The system may further include an observable feature positioned in the catheter within a field of view of the imaging sensor, and the imaging sensor may be configured to capture an image of the field of view including a first image, the first image including the observable feature. The orientation of the observable feature in the first image may indicate a rotational offset that the tool has been rotated within the catheter.

[0010] In accordance with some embodiments, a method for determining the orientation of a tool at least partially mounted within a catheter is provided. The method may include capturing a first image from within the lumen of the catheter via an image sensor of the tool, the first image comprising a plurality of pixels. The method may further include identifying a first subset of the plurality of pixels comprising a first observable feature, a second subset of pixels comprising a second observable feature, and a third subset of the plurality of pixels comprising a background color. The method may further include adjusting the third subset to a neutral color. The method may further include generating a modified image by filtering the plurality of pixels to remove the third subset. The method may further include determining an angular orientation of each of the first and second observable features in the modified image. The method may further include determining a rotational offset of the tool relative to the catheter based on the angular orientation of the first and second observable features.

[0011] It should be understood that both the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure rather than to limit the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a simplified schematic diagram of a teleoperated medical system according to some embodiments.

[0013] Figure 2A is a simplified partial schematic diagram of a medical device system according to some embodiments.

[0014] Figure 2B is a simplified schematic diagram of a medical device with an expanded medical tool according to some embodiments.

[0015] Figure 3A and Figure 3B is a simplified diagrammatic side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments.

[0016] Figure 4 is a simplified partial schematic diagram of a distal portion of a catheter assembly according to some embodiments.

[0017] Figure 5 is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments.

[0018] Figure 6 is a simplified partial cross-sectional view of a distal portion of a catheter assembly with longitudinal markings according to some embodiments.

[0019] Figure 7 According to some embodiments Figure 6 Simplified cross-sectional end view of the distal portion of the catheter assembly.

[0020] Fig. 8A is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments.

[0021] Figure 8B According to some embodiments Fig. 8A Simplified cross-sectional end view of a distal portion of a catheter assembly with a tool rotationally aligned with the catheter.

[0022] Figure 8C According to some embodiments Fig. 8A Simplified cross-sectional end view of the distal portion of a catheter assembly with the tool rotationally offset from the catheter.

[0023] Fig. 9A According to some embodiments Figure 8C Representative images of a tool acquisition of a catheter assembly where the tool is rotationally offset from the catheter.

[0024] Fig. 9B According to some embodiments Fig. 9A Representative images of images with the rotation offset removed.

[0025] Fig. 10A is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein the distal portion of the tool has a square cross-section.

[0026] Fig. 10B According to some embodiments Fig. 10A Simplified cross-sectional view of a catheter assembly.

[0027] Fig. 10C According to some embodiments Fig. 10A Another simplified cross-sectional view of a catheter assembly.

[0028] Fig. 10D According to some embodiments Fig. 10A Simplified cross-sectional end view of a distal portion of a catheter assembly with a tool rotationally aligned with the catheter.

[0029] Fig. 10E According to some embodiments Fig. 10A Simplified cross-sectional end view of the distal portion of a catheter assembly with the tool rotationally offset from the catheter.

[0030] Fig.10F is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein the distal portion of the tool has a square cross-section.

[0031] Figure 10G According to some embodiments Figure 8B Representative images of a tool of a catheter assembly having one or more longitudinal markings acquired with the tool rotationally aligned with the catheter.

[0032] Fig. 10H According to some embodiments Figure 10G Representative plots of saturation versus hue for pixels in a representative image.

[0033] Fig.10I According to some embodiments Figure 10G Representative images where a circular filter is applied to the image.

[0034] Fig.10J According to some embodiments Figure 10G Representative images where custom-shaped filters are applied to images.

[0035] Figure 10K and Fig.10L According to some embodiments Figure 8B Representative images acquired of a tool of a catheter assembly, with one or more longitudinal markers in each image, with the tool rotationally aligned with the catheter, and a custom shaped filter applied to each image.

[0036] Figure 10M According to some embodiments Figure 8B A representative image with one or more longitudinal markers acquired with a tool of a catheter assembly, wherein the tool is not rotationally aligned with the catheter, and a custom shaped filter is applied to the image.

[0037] Fig.11 is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein an observable feature protrudes from the distal end of the catheter and a light source is at the distal end of a tool within the catheter.

[0038] Fig.12 is a simplified perspective view of a distal portion of a catheter assembly according to some embodiments, with observable features protruding from the distal end of the catheter.

[0039] Fig.13 is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein an observable feature protrudes from the distal end of the catheter and a light source is at the distal end of the catheter.

[0040] Fig.14A is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein two light sources are oppositely disposed at the distal end of the catheter.

[0041] Fig. 14B According to some embodiments Fig.14A A simplified partially cut-away end view of a distal portion of a catheter assembly with a tool rotationally offset from the catheter.

[0042] Fig. 14C is a simplified partial cross-sectional view of a distal portion of a catheter assembly according to some embodiments, wherein a fluid jet nozzle is disposed at the distal end of the catheter.

[0043] Fig.14D According to some embodiments Fig. 14C A simplified partially cut-away end view of a distal portion of a catheter assembly with a tool rotationally offset from the catheter.

[0044] Figures 15A-15B According to some embodiments, when a light source is used from Fig. 14B Representative images acquired while observing the distal end of the tool.

[0045] Fig. 15C According to some embodiments Fig. 15B Representative images of images with the rotation offset removed.

[0046] Figures 16A-16B According to some embodiments, when another light source is used from Fig. 14B Representative images acquired while observing the distal end of the tool.

[0047] Fig. 16C According to some embodiments Fig. 16B Representative images of images with the rotation offset removed.

[0048] Figures 17A-17B According to some embodiments, when two light sources are used from Fig. 14B Representative images acquired while observing the distal end of the tool.

[0049] Fig. 17C According to some embodiments Fig. 17B Representative images of images with the rotation offset removed.

[0050] Fig.18A According to some embodiments, when a light source having a structured light pattern is used from Fig. 14B Representative images acquired while observing the distal end of the tool.

[0051] Fig.18B According to some embodiments Fig.18A Representative images of images with the rotation offset removed.

[0052] Fig.19A is a representative flow chart for controlling the collection and adjustment of images during a procedure, according to some embodiments.

[0053] Fig.19B is a representative flow chart for controlling when rotational offsets of images are collected and reported to a control system during a procedure in accordance with some embodiments.

[0054] Fig.19C is a representative flow chart for detecting observable features in a captured image and determining a rotational offset of the observable features in accordance with some embodiments.

[0055] Embodiments of the present disclosure and their advantages can be best understood by referring to the detailed description below. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the accompanying drawings, wherein the illustrations herein are for the purpose of illustrating embodiments of the present disclosure rather than for limiting embodiments of the present disclosure. DETAILED DESCRIPTION

[0056] In the following description, the specific details of some embodiments consistent with the present disclosure are described. A lot of specific details are described to provide a full understanding of the embodiments. However, it is apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. Those skilled in the art may realize other elements within the scope and spirit of the present disclosure although not specifically described herein. In addition, in order to avoid unnecessary repetition, one or more features shown and described in conjunction with an embodiment may be incorporated into other embodiments, unless specifically described otherwise or if one or more features will make the embodiment inoperable.

[0057] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0058] The present disclosure describes various apparatuses and apparatus parts from the perspective of their states in three-dimensional space. As used herein, the term "position" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational arrangement of an object or a portion of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "posture" refers to the position of an object or a portion of an object on at least one translational degree of freedom and the orientation of the object or some portions of the object on at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of postures, positions, or orientations measured along an object.

[0059] Figure 1 is a simplified schematic diagram of a teleoperated medical system 100 according to some embodiments. In some embodiments, the teleoperated medical system 100 may be suitable for use in, for example, surgical procedures, diagnostic procedures, therapeutic procedures, or biopsy procedures. Although some embodiments of such procedures are provided herein, any reference to medical instruments or surgical instruments and medical methods or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnostics, and for industrial systems and general robotic systems or teleoperated medical systems.

[0060] like Figure 1As shown, medical system 100 generally includes a manipulator assembly 102 for operating a medical device 104 to perform various procedures on a patient P. Manipulator assembly 102 can be a teleoperated assembly, a non-teleoperated assembly, or a hybrid teleoperated and non-teleoperated assembly having selected degrees of freedom of movement that can be motorized and / or teleoperated and having selected degrees of freedom of movement that can be non-motorized and / or non-teleoperated. Manipulator assembly 102 is mounted to or near an operating table T. Main assembly 106 allows an operator O (e.g., Figure 1 The surgeon, clinician, or physician (shown) views the intervention site and controls the manipulator assembly 102.

[0061] The main assembly 106 can be located at an operator console, which is usually located in the same room as the surgical table T, such as on the side of the surgical table on which the patient P is located. However, it should be understood that the operator O can be located in a different room or a completely different building from the patient P. The main assembly 106 generally includes one or more control devices for controlling the manipulator assembly 102. The control device can include any number of various input devices, such as a joystick, a tracking ball, a data glove, a trigger gun, a manually operated controller, a voice recognition device, a body movement or a presence sensor and / or the like. In order to provide the operator O with a strong sense of directly controlling the instrument 104, the control device can be provided with the same degree of freedom as the associated medical device 104. In this way, the control device provides the operator O with a remote presentation / telepresence or perception of the integrated control device and the medical device 104.

[0062] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104 and still provide telepresence / telepresence to the operator O. In some embodiments, the control device may optionally be a manual input device that moves with six degrees of freedom and may also include an actuatable handle for actuating the device (e.g., for closing a grasping jaw, applying an electrical potential to an electrode, delivering a medication, and / or the like).

[0063] The manipulator assembly 102 supports the medical device 104 and may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in an appropriate orientation, generally referred to as a mounting structure) and / or one or more servo-controlled links (e.g., one or more links that can be controlled in response to commands from a control system), and a manipulator. The manipulator assembly 102 may optionally include a plurality of actuators or motors that drive inputs on the medical device 104 in response to commands from a control system (e.g., the control system 112). The actuator may optionally include a drive system that, when coupled to the medical device 104, can advance the medical device 104 into a natural orifice or an anatomical orifice created by a surgical procedure. Other drive systems can move the distal end of the medical device 104 in multiple degrees of freedom, which may include three degrees of freedom of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of freedom of rotational motion (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Additionally, the actuator can be used to actuate an articulatable portion of the medical device 104 for grasping tissue in the jaws of a biopsy device and / or the like. Actuator positioning sensors such as rotary transformers, encoders, potentiometers, and other mechanisms can provide sensor data describing the rotation and orientation of the motor shaft to the medical system 100. This positioning sensor data can be used to determine the motion of an object manipulated by the actuator.

[0064] The teleoperated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instrument of the manipulator assembly 102. These subsystems may include: a position / location sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, speed, velocity, posture, and / or shape along the distal end and / or one or more segments of the flexible body that may comprise the medical instrument 104; and / or a visualization system for acquiring images from the distal end of the medical instrument 104.

[0065] The teleoperated medical system 100 also includes a display system 110 for displaying images or diagrams of the surgical site and medical instrument 104 generated by the subsystems of the sensor system 108. The display system 110 and the main assembly 106 can be oriented so that the operator O can control the medical instrument 104 and the main assembly 106 with the perception of telepresence / telepresence.

[0066] In some embodiments, the medical device 104 can have a visualization system (discussed in more detail below) that can include a viewing mirror assembly that records concurrent or real-time images of the surgical site and provides the images to the operator or operator O through one or more displays of the medical system 100 (such as one or more displays of the display system 110). The concurrent images can be, for example, two-dimensional or three-dimensional images acquired by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that can be integrally or removably coupled to the medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly can be used with the medical device 104 to image the surgical site. The visualization system can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors (which can include a processor of the control system 112).

[0067] The display system 110 can also display images of the surgical site and medical instruments acquired by the visualization system. In some examples, the teleoperation medical system 100 can configure the controls of the medical instruments 104 and the main assembly 106 so that the relative orientation of the medical instruments is similar to the relative orientation of the eyes and hands of the operator O. In this way, the operator O can operate the medical instruments 104 and the hand controls as if observing the workspace in a substantially telepresence situation. By telepresence, it means that the presentation of the image is a true perspective image that simulates the perspective of the physician who physically manipulates the medical instrument 104.

[0068] In some examples, display system 110 can present images of a surgical site recorded preoperatively or intraoperatively using image data from an imaging technique such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermography, impedance imaging, laser imaging, nanotube x-ray imaging, or the like. The preoperative or intraoperative image data can be presented as a two-dimensional (2D), three-dimensional (3D), or four-dimensional (2D) (including, for example, time-based or velocity-based information) image, and / or as an image from a model created from a preoperative or intraoperative image data set.

[0069] In some embodiments, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered (i.e., dynamically referenced) with the preoperative or immediate image / model, typically for the purpose of an imaging-guided surgical procedure. Doing so may present the operator O with a virtual image of the internal surgical site from the perspective of the medical device 104. In some examples, the perspective may be from the tip of the medical device 104. An image of the tip of the medical device 104 and / or other graphical or alphanumeric indicators may be superimposed on the virtual image to assist the operator O in controlling the medical device 104. In some examples, the medical device 104 may not be visible in the virtual image.

[0070] In some embodiments, the display system 110 can display a virtual navigation image in which the actual position of the medical device 104 is registered with the preoperative image or the real-time image to present a virtual image of the medical device 104 in the surgical site to the operator O from an external perspective. An image of a portion of the medical device 104 or other graphic or alphanumeric indicator can be superimposed on the virtual image to help the operator O control the medical device 104. As described herein, a visual diagram of the data points can be rendered to the display system 110. For example, measured data points, moved data points, registered data points, and other data points described herein can be displayed on the display system 110 in a visual diagram. The data points can be visually represented on the user interface by a plurality of points or dots on the display system 110 or as a rendered model (such as a grid or line model created based on a set of data points). In some examples, the data points can be color-coded according to the data they represent. In some embodiments, after each processing operation has been implemented to change the data points, the visual diagram can be refreshed in the display system 110.

[0071] The remotely operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory (not shown) and at least one computer processor (not shown) for implementing control between the medical device 104, the main component 106, the sensor system 108, and the display system 110. The control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing these instructions) for implementing some or all of the methods described in accordance with the various aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112 is Figure 1100 is shown as a single box in the simplified schematic diagram of the remote medical system 100, but the control system 112 may include two or more data processing circuits distributed throughout the remote medical system 100 to perform distributed data processing. For example, a portion of the data processing performed by the distributed control system 112 can be optionally performed on or adjacent to the manipulator assembly 102, another portion of the data processing can be optionally performed at the main assembly 106, and the other portion of the data processing can be optionally performed at other data processing circuits. At least one computer processor or two or more data processing circuits of the control system 112 can execute instructions corresponding to the processing disclosed herein and described in more detail below. Any variety of centralized or distributed data processing architectures can be utilized. Similarly, the programmed instructions can be implemented as several independent programs or subroutines, or they can be integrated into several other aspects of the remotely operated medical system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (infrared data communication), HomeRF (home radio frequency), IEEE802.11, DECT (digital enhanced wireless communication) and wireless telemetry.

[0072] In some embodiments, the control system 112 can receive force and / or torque feedback from the medical device 104. In response to the feedback, the control system 112 can transmit a signal to the main assembly 106. In some examples, the control system 112 can transmit a signal instructing one or more actuators of the manipulator assembly 102 to move the medical device 104. The medical device 104 can extend into an internal surgical site within the body of the patient P via one or more openings in the body of the patient P. Any suitable conventional and / or dedicated actuator can be used. In some examples, one or more actuators can be separate from the manipulator assembly 102, or integrated with the manipulator assembly 102. In some embodiments, one or more actuators and the manipulator assembly 102 are provided as part of a remotely operated cart positioned adjacent to the patient P and the operating table T.

[0073] The control system 112 may optionally further include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on reference to a preoperative or intraoperative data set of an acquired anatomical pathway. The virtual visualization system processes images of surgical sites imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques. Software that can be used in conjunction with manual input is used to convert the recorded images into a two-dimensional or three-dimensional composite view of the segmentation of a local or global anatomical organ or anatomical region. The image data set is associated with the composite view. The composite view and the image data set describe the various positions and shapes of the pathway and their connectivity. The images used to generate the composite view can be recorded preoperatively or intraoperatively during a clinical procedure. In some embodiments, the virtual visualization system can use a standard view (i.e., not patient-specific) or a mixture of a standard view and patient-specific data. The composite view and any virtual images generated from the composite view may represent the static pose of the deformable anatomical region during one or more phases of motion (eg, during the inspiration / expiration cycle of the lungs).

[0074] During the virtual navigation procedure, the sensor system 108 can be used to calculate the approximate position of the medical device 104 relative to the anatomy of the patient P. This position can be used to generate a macroscopic (external) tracking image of the anatomy of the patient P and a virtual internal image of the anatomy of the patient P. The sensor system 108 can implement one or more electromagnetic (EM) sensors, fiber optic sensors and / or other sensors to align and display the medical device and the surgical image recorded before the operation. For example, U.S. patent application US13 / 107,562 (filed on May 13, 2011) (disclosing "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery") discloses such a sensor system, which is incorporated herein by reference in its entirety. The remotely operated medical system 100 may further include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems and / or suction systems. In some embodiments, the remotely operated medical system 100 may include more than one manipulator assembly and / or more than one main assembly. The total number of teleoperated manipulator assemblies included in the teleoperated medical system will depend on a number of factors including the surgical procedure and the space constraints within the operating room. When implemented as multiple units, the main assembly 106 can be collocated or positioned in independent locations. Multiple main assemblies allow more than one operator to control one or more teleoperated manipulator assemblies in various combinations.

[0075] Figure 2A is a simplified schematic diagram of a medical device system 200 according to some embodiments. In some embodiments, the medical device system 200 can be used as a medical device 104 in an image-guided medical procedure performed using a teleoperated medical system 100. In some examples, the medical device system 200 can be used in a non-teleoperated exploratory procedure or in a procedure involving a traditional manually operated medical device (such as an endoscope). Optionally, the medical device system 200 can be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical pathway of a patient (such as patient P).

[0076] The medical device system 200 includes an elongated device 202 (such as a flexible catheter) coupled to a drive unit 204. The elongated device 202 includes a flexible body 216 having a proximal end 217 and a distal end 218 (in some embodiments, the distal end may be a tip portion). In some embodiments, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible body outer diameters may be larger or smaller.

[0077] The medical device system 200 further includes a tracking system 230 for determining the position, orientation, speed, velocity, posture, and / or shape along the distal end 218 of the flexible body 216 and / or along one or more segments 224 of the flexible body 216 using one or more sensors and / or imaging devices, as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into a plurality of segments 224. The tracking system 230 can optionally be implemented as hardware, firmware, software, or a combination thereof interacting with or otherwise executed by one or more computer processors, which can include Figure 1 At least one processor or two or more data processing circuits of the control system 112 in.

[0078] The tracking system 230 may optionally use a shape sensor 222 to track the distal end 218 and / or one or more segments 224. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size of the optical fiber may be larger or smaller. The optical fiber of the shape sensor 222 forms a fiber optic bend sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. patent application Ser. No. 11 / 180,389, filed Jul. 13, 2005 (disclosing “Fiberoptic position and shape sensing device and method relating thereto”); U.S. patent application Ser. No. 12 / 047,056, filed Jul. 16, 2004 (disclosing “Fiber-optic shape and relative position sensing”); and U.S. Pat. No. 6,389,187, filed Jun. 17, 1998 (disclosing “Optical Fibre Bend Sensor”), which are incorporated herein by reference in their entirety.

[0079] In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the elongated device may be determined using other techniques. For example, the history of the distal end posture of the flexible body 216 can be used to reconstruct the shape of the flexible body 216 within a given time interval. In some embodiments, the tracking system 230 may optionally and / or additionally use an orientation sensor system 220 to track the distal end 218. The orientation sensor system 220 may be a component of an EM sensor system, wherein the orientation sensor system 220 includes one or more conductive coils that can be subjected to an externally generated electromagnetic field. Each coil of the EM sensor system then generates an induced electrical signal having characteristics that depend on the orientation and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the orientation sensor system 220 may be configured and positioned to measure six degrees of freedom (e.g., three orientation coordinates X, Y, Z and three orientation angles of pitch, yaw, and roll indicating a base point) or five degrees of freedom (e.g., three orientation coordinates X, Y, Z and two orientation angles of pitch and yaw indicating a base point). A further description of the orientation sensor system is provided in U.S. Pat. No. 6,380,732 (filed Aug. 11, 1999) (disclosing “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated herein by reference in its entirety.

[0080] In some embodiments, tracking system 230 may alternatively and / or additionally rely on historical posture, position, or orientation data stored for known points of the instrument system along a cycle of alternating motion (such as breathing). This stored data may be used to develop shape information about flexible body 216. In some examples, a series of position sensors (not shown) (such as electromagnetic (EM) sensors similar to the sensors of position sensor system 220) may be positioned along flexible body 216 and then used for shape sensing. In some examples, a history of data from one or more of these sensors acquired during a procedure may be used to represent the shape of elongated device 202, particularly if the anatomical pathway is substantially static.

[0081] Flexible body 216 includes a channel 221 sized and shaped to receive a medical tool 226 . Figure 2B2 is a simplified schematic diagram of a flexible body 216 with an extended medical tool 226 according to some embodiments. In some embodiments, the medical tool 226 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation or suction. The medical tool 226 can be deployed through the channel 221 of the flexible body 216 and used at a target location in the anatomy. The medical tool 226 may include, for example, an image acquisition probe, a biopsy instrument, a laser ablation fiber and / or other surgical, diagnostic or therapeutic tools. The medical tool may include a single working member, such as a surgical scalpel, a blunt blade, an optical fiber, an electrode and / or the like. Other medical tools may include, for example, tweezers, a grasper, scissors, a clip applier and / or the like. Other medical tools may further include electrically activated tools, such as electrosurgical electrodes, transducers, sensors and / or the like. In various embodiments, the medical tool 226 is a biopsy instrument that can be used to remove sample tissue or cell sampling from a target anatomical location.

[0082] The medical tool 226 can be used with an image acquisition probe that is also within the flexible body 216. In various embodiments, the medical tool 226 can be an image acquisition probe that includes a distal portion having a stereoscopic camera or a monoscopic camera at or near the distal end 218 of the flexible body 216 for acquiring images (including video images), which are processed by the visualization system 231 for display and / or provided to the tracking system 230 to support tracking of the distal end 218 and / or one or more segments 224. The image acquisition probe can include a cable coupled to the camera for transmitting the acquired image data. In some examples, the image acquisition instrument can be a fiber optic bundle coupled to the visualization system 231, such as a fiberscope. The image acquisition instrument can be single-spectrum or multi-spectrum, for example, acquiring image data in one or more of the visible light spectrum, infrared spectrum, and / or ultraviolet spectrum. Alternatively, the medical tool 226 itself can be an image acquisition probe. The medical tool 226 can be advanced from the opening of the channel 221 to perform a procedure, and then retracted into the channel when the procedure is completed. The medical tool 226 may be removed from the proximal end 217 of the flexible body 216 or from another optional instrument port (not shown) along the flexible body 216 .

[0083] The medical tool 226 may additionally house a cable, linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of the medical tool 226. Steerable instruments are described in detail in U.S. Pat. No. 7,316,681 (filed Oct. 4, 2005) (disclosing “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent Application No. 12 / 286,644 (filed Sept. 30, 2008) (disclosing “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated herein by reference in their entireties.

[0084] The flexible body 216 may also house cables, linkages, or other steering controls (not shown) extending between the drive unit 204 and the distal end 218 to controllably bend the distal end 218 as shown in different orientations 219, as depicted by dashed lines of the distal end 218, for example. In some examples, at least four cables are used to provide independent "up and down" steering to control the pitch of the distal end 218 and "side to side" steering to control the yaw of the distal end 218. Steerable elongated devices are described in detail in U.S. patent application US13 / 274,208 (filed on October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. In embodiments where the medical device system 200 is actuated by a remotely operated assembly, the drive unit 204 may include a drive input device that is removably coupled to a drive element (such as an actuator) of the remotely operated assembly and receives power from the drive element. In some embodiments, the medical device system 200 may include clamping features, manual actuators, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator-controlled bending of the distal end 218. In some examples, one or more lumens are defined in the wall of the flexible body 216 through which a medical device can be deployed and used at a target surgical site.

[0085] In some embodiments, the medical device system 200 may include a flexible bronchial device, such as a bronchoscope or bronchial catheter for use in inspection, diagnosis, biopsy, or treatment of the lung. The medical device system 200 is also suitable for navigating and treating other tissues via natural or surgically established connections in any of a variety of anatomical systems, including the colon, intestines, kidneys and calyces, brain, heart, circulatory system including vasculature, and / or the like.

[0086] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or preoperatively acquired models to provide real-time position information to the physician or other operator. In some examples, the real-time position information can be displayed on a Figure 1 on the display system 110 for controlling the medical device system 200. In some examples, Figure 1 The control system 112 can use the orientation information as feedback to position the medical instrument system 200. U.S. patent application US13 / 107,562, filed on May 13, 2011, disclosing "Medical System Providing Dynamic Registration ofa Model of an Anatomic Structure for Image-Guided Surgery," provides various systems for utilizing fiber optic sensors to align and display surgical instruments and surgical images, which is incorporated herein by reference in its entirety.

[0087] In some examples, the medical device system 200 may be Figure 1 In some embodiments, Figure 1 The manipulator assembly 102 of the apparatus may be replaced by direct operator controls. In some examples, the direct operator controls may include various handles and operator interfaces for handheld operation of the apparatus.

[0088] Figure 3A and Figure 3B is a simplified schematic diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments. Figure 3A and Figure 3B As shown, surgical environment 300 includes a patient P positioned in Figure 1The patient P may be stationary within the surgical environment in the sense that overall patient movement is limited by sedation, restraints, and / or other means. Periodic anatomical motion (including breathing and cardiac motion of the patient P) may continue unless the patient is told to hold his or her breath to temporarily suspend respiratory motion. Thus, in some embodiments, data may be collected at a specific stage in breathing and the stage may be used to mark and identify. In some embodiments, the stage during which data is collected may be inferred based on physiological information collected from the patient P. Within the surgical environment 300, a point collection instrument 304 is coupled to an instrument bracket 306. In some embodiments, the point collection instrument 304 may use an EM sensor, a shape sensor, and / or other sensor modalities. The instrument bracket 306 is mounted to an insertion table 308 fixed within the surgical environment 300. Alternatively, the insertion table 308 may be movable but have a known position within the surgical environment 300 (e.g., known via a tracking sensor or other tracking device). The instrument bracket 306 can be a component of a manipulator assembly (e.g., manipulator assembly 102) that is coupled to the point collection instrument 304 to control the insertion motion (i.e., motion along the A axis) and optionally control the motion of the distal end 318 of the elongated device 310 in multiple directions (including yaw, pitch, and roll). The instrument bracket 306 or insertion stage 308 can include an actuator, such as a servo motor (not shown), that controls the motion of the instrument bracket 306 along the insertion stage 308.

[0089] An elongated device 310 (e.g., a medical device) can be coupled to a device body 312. The device body 312 is coupled and fixed relative to the device bracket 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the device body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 can be movable along with the device body 312, but the position of the proximal point 316 can be known (e.g., via a tracking sensor or other tracking device). The fiber optic shape sensor 314 measures the shape from the proximal point 316 to another point (such as a distal end 318 of the elongated device 310). The point collection device 304 can be substantially similar to the medical device system 200.

[0090] The position measurement device 320 provides information about the position of the instrument body 312 as the instrument body 312 moves along the insertion axis A on the insertion stage 308. The position measurement device 320 may include a decoder, an encoder, a potentiometer, and / or other sensor that determines the rotation and / or orientation of the actuator that controls the movement of the instrument bracket 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be arcuate, or have a combination of arcuate segments and linear segments.

[0091] Figure 3A The instrument body 312 and instrument bracket 306 are shown in a retracted orientation along the insertion stage 308. In this retracted orientation, the proximal point 316 is at an orientation L0 on the axis A. In this orientation along the insertion stage 308, components of the position of the proximal point 316 can be set to zero and / or another reference value to provide a basic reference describing the orientation of the instrument bracket 306 and, therefore, the proximal point 316 on the insertion stage 308. With this retracted orientation of the instrument body 312 and instrument bracket 306, the distal end 318 of the elongated device 310 can be positioned just inside the access orifice of the patient P. Also in this orientation, the orientation measurement device 320 can be set to zero and / or another reference value (e.g., I=0). Figure 3B 308, the instrument body 312 and the instrument cradle 306 have been advanced along the linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advanced orientation, the proximal point 316 is at the orientation L1 on the axis A. In some examples, encoders and / or other orientation data from one or more actuators that control the movement of the instrument cradle 306 along the insertion stage 308 and / or one or more orientation sensors associated with the instrument cradle 306 and / or the insertion stage 308 are used to determine the orientation Lx of the proximal point 316 relative to the orientation L0. In some examples, the orientation Lx can further be used as an indication of the distance or insertion depth of the distal end 318 of the elongated device 310 inserted into the passage of the anatomical structure of the patient P.

[0092] In order to operate the medical device system safely and effectively, the medical tool may need to be properly installed, located, identified, authorized and / or otherwise received, and identified when installed into a system (such as manipulator assembly 102) or inserted into a receiving member (such as medical device system 200). As disclosed herein, the tool identification assembly at the receiving member can be used to detect the presence or absence of a target on the tool to detect and develop an insertion mark for each inserted tool. Based on the insertion mark detected and developed, various options for operating the tool or medical device system can be enabled or disabled. Although many embodiments described herein describe the receiving member as a catheter, the tool identification system and method described are suitable for use with any type of tool and receiving member. In one embodiment described in detail below, the tool identification assembly can be used to determine the operating mode based on whether the medical tool is fully inserted into the catheter assembly. If, for example, the tool is a camera probe, the tool identification assembly can be used to determine whether the probe is properly located in the delivery catheter before the catheter can be operated in a drive mode and advanced into the patient. Allowing the catheter to be blindly advanced without ensuring that the camera probe is properly positioned can cause damage to the patient that can be prevented by the use of the tool recognition assembly. Once at the destination, the camera probe can be withdrawn from the catheter to make room for a different medical tool. Withdrawal of the camera probe can prevent the physician from observing the internal body structures to be treated or evaluated. Consistent with the teachings of the present disclosure, the tool recognition assembly can detect that the camera has been removed and can respond by entering a safe mode. When in the safe mode, the control system (e.g., Figure 1 One or more functions of the control system 112 in the tool identification assembly can be limited or disabled. For example, the speed at which catheter flexibility and / or adjustments to the catheter orientation can be made can be limited. It is expected that such limitations reduce the possibility of patient injury caused by blind adjustments to the instrument that remains inserted in the patient after the withdrawal of the camera. Therefore, embodiments of the teachings of the present disclosure are expected to improve the safety of minimally invasive procedures. The tool identification assembly can also be used to identify counterfeit or otherwise unauthorized devices (such as devices manufactured by unauthorized manufacturers). The tool identification assembly can also be used to identify tool types (e.g., needles, ablation tools, cutters, grabbers, etc.), and based on the identification of the tool type, control mode alternation or tool behavior modification can be implemented.

[0093] In some embodiments, the tool can be installed in the catheter, and the tool can include an imaging sensor for collecting images, for example, during the procedure. If the tool inserted through the catheter lumen is not rigidly coupled to the catheter, the tool can rotate relative to the catheter. In order to prevent this rotation, a physical key system can be provided in combination on the catheter and the tool. For example, a physical groove extending longitudinally with a rectangular cross section can be provided for the inner wall of the catheter. A physical key with a similar but slightly smaller cross section than the groove can protrude from the outside of the distal portion of the tool. In order to install the tool in the catheter, the physical key must be aligned with the groove. Therefore, when the physical key is aligned with the groove, the tool is oriented together with the catheter so that the tool rotates with the catheter during the installation of the catheter in the patient's anatomical structure. This physical key system can help prevent the tool from rotating relative to the catheter, but there are also some disadvantages.

[0094] Since the key on the tool is aligned with the catheter at only one orientation, the tool must be positioned so as to properly align the physical key with the matching groove. This can sometimes be challenging and at least a hassle. In addition, since the catheter may have a certain amount of flexibility and a certain degree of imperfect fit (e.g., due to manufacturing tolerances) for the physical key system, when a sufficient torque difference occurs between the tool and the catheter, the tool may be able to overcome the constraints of the physical key system and rotate or flip in the catheter. The rotation of the tool in the catheter can cause the physical key to be stuck in the catheter, which may render the system unusable. In addition, providing a physical key on the tool and providing a groove in the catheter will add cost to the product.

[0095] Eliminating the protruding physical key from the tool and the groove from the interior of the catheter can eliminate the insertion alignment, jamming and cost issues described above. In place of the protruding physical key, a real-time control system based on visual feedback can be used to measure the rotational offset of the tool relative to the catheter. Images captured during the procedure can be adjusted to remove the rotational offset, so that the resulting adjusted image produces the appearance that the tool is rotationally aligned with the catheter.

[0096] Reference now Figure 4, a simplified local schematic diagram of the distal portion of catheter assembly 500 is shown.Catheter assembly 500 can be similar to the medical device system 200 as described before.Instrument 520 is installed in catheter 522, wherein the distal end 516 of catheter 522 is aligned with the distal end 512 of instrument 520.Catheter 522 can be a steerable catheter.Tension force can be selectively applied to control cables (also referred to as traction cables) 630,632,634,636 so as to provide the articulation of distal end 218 with 2 degrees of freedom (i.e. front and back, left and right).Four control cables can provide cable pairs 630 / 634 and cable pairs 632 / 636, wherein each cable of a pair is arranged relatively to each other around the circumference of catheter (i.e. about 180 degrees).Each pair can be separated 90 degrees circumferentially. Thus, the four control cables 630, 632, 634, 636 are circumferentially spaced apart around the conduit at approximately 90 degree spacings from adjacent control cables (e.g., cable 630 can have a 90 degree spacing between it and two adjacent cables 632, 636 and a 180 degree spacing from cable 634).

[0097] The cable pairs 630 / 634, 632 / 636 can bend the distal end 516 of the catheter 522 both forward and backward and left and right (e.g., some articulation positions 219 of the distal end 516 are shown). The combination of left and right and forward and backward bending can cause the catheter to rotate relative to the tool 520 about the central axis 502. The rotation of the catheter 522 relative to the tool 520 can cause the imaging sensor in the tool 520 to capture an image that is not aligned with the orientation of the catheter 522 and its cable pair. If the orientation of the image is separated from the orientation of the catheter and the cable pair, then the image becomes ineffective as an aid to the user when steering the catheter via the cable. Images that are rotationally offset from the catheter control cable can be misinterpreted, causing incorrect manipulation of the control cable and undesired articulation of the distal end 516 of the catheter. The present disclosure eliminates (or at least minimizes) these problems by providing the user with an image that has been adjusted to remove the rotational offset.

[0098] An algorithm as described below can determine the rotational offset of the tool 520 relative to the catheter 522. Once the rotational offset is known, the control system 112 can rotate the image for display on the user screen to appropriately present visual feedback to the user, and the control system 112 can map user commands to the pull cables to modify the movement of the catheter 522 to correlate with the displayed image. The amount of tension in the pull cables can be controlled by the control system 112. The tension on the pull cables can be determined by always maintaining a minimum tension in all the pull cables and then adjusting the tension to react to input from the input device commanding the catheter in a specific direction.

[0099] refer to Figure 5, a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500 is provided. The XYZ coordinate reference is given for the purpose of discussion and does not limit the present disclosure. In general, the X-axis can be parallel to the central axis 502, wherein the Y-axis and the Z-axis are perpendicular to the central axis 502. The tool 520 is partially mounted in the lumen 524 and is spaced apart from the distal end 516 of the catheter 522. The imaging sensor 510 can be positioned in the distal portion 514 of the tool 520 and has an observable area 540 extending from the distal end 512. However, it should be understood that the imaging sensor 510 can be positioned at other locations in the tool 520. For example, the imaging sensor 510 can be positioned at the proximal portion of the tool 520 using an optical transmission system (e.g., an optical waveguide) that transmits light from the distal end 516 to the imaging sensor 510. The lens 534 can be positioned at the distal end 512 and has an optical center of the lens 528. Lens 534 can provide an observable area 540 in front of imaging sensor 510, wherein near point 536 and far point 538 are the extreme ends of observable area 540. It should be understood that observable area 540 is not necessarily to scale, and far point 538 of area 540 can be larger than Figure 5 Furthermore, near point 536 can extend farther or closer than indicated in FIG. Figure 5 The imaging sensor 510 can capture both high frequency light (such as light within the human visible spectrum) and low frequency light (such as infrared light). The optical source 530 can be disposed at the distal end 512 of the tool 520 and / or at the distal end 516 of the catheter 522. The optical source 530 can generate emitted light (or illumination light) 532 to illuminate the lumen 524 when the tool 520 is being inserted into the catheter 522, and to illuminate objects outside the lumen 524 when the tool 520 is fully inserted (i.e., inserted into the catheter 522 to allow full functionality of the tool 520). In some embodiments, the illumination light 532 can be red, blue, green, yellow, or any other suitable color of light.

[0100] Reference now Figure 6, a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500 is shown. In this embodiment, an observable feature 470 in the form of a longitudinal mark extending parallel to the central axis 502 is formed on the inner surface of the lumen 524, and can be a contrast feature with the rest of the inner surface of the lumen, such as different colors, different textures, etc. The rest of the surface can be referred to as the background layer of the lumen. The mark (or observable feature) 470 provides sufficient contrast between the background layer (such as background color) so that the mark can be seen in the image captured by the imaging sensor 510 in the catheter 522. The observable area 540 gives the imaging sensor 510 a 360-degree viewing angle of the inner surface of the lumen 524 in front of the distal end 512 of the tool 520. The longitudinal mark 470 is formed at a predetermined angle position in the lumen 524. Therefore, the angle relationship between the mark 470 and the catheter 522 is fixed. When the imaging sensor 510 captures an image within the lumen 524, the longitudinal marking 470 will appear in the image at an angular position in the 360-degree viewing angle of the imaging sensor 510. The longitudinal marking 470 can be any feature that is visible on the inner surface of the lumen 524 and distinguishable from the rest of the inner surface of the lumen 524. Moreover, the longitudinal marking 470 can occupy a small circumferential distance of the inner surface when compared to the entire circumferential distance around the inner surface. Therefore, the longitudinal marking can be referred to as a longitudinal stripe that extends along at least a portion of the inner surface of the lumen.

[0101] When the tool 520 is being inserted into the catheter 522, the imaging sensor 510 can continuously capture images from the lumen 524 to be used by the control system 112 to determine the orientation of the (one or more) longitudinal markers in the lumen 524 (and therefore the orientation of the tool 520). At any time, the orientation of the (one or more) longitudinal markers can be determined by analyzing multiple (e.g., about 50) recently captured images. The recently captured images include the latest orientation information of the (one or more) longitudinal markers, and multiple images can be processed to provide an increased confidence that the result of the tool orientation determination is accurate. When the tool 520 is fully inserted into the catheter 522 and placed together with the catheter 522, the tool 520 can no longer observe the (one or more) longitudinal markers 470 on the interior of the catheter 522. Therefore, further collection of images for orientation determination is no longer beneficial and can be stopped. The tool 520 can then be used for other tasks, such as collecting images of the patient's anatomical structure observed from the distal end of the catheter 522.

[0102] Figure 7A simplified cross-sectional end view at the distal portion 518 of the catheter assembly 500 is illustrated. The tool 520 can be rotated relative to the catheter 522 and the catheter cables 630-636, which in some embodiments can be control cables (e.g., during installation of the tool 520 and / or due to other factors, such as manufacturing tolerances). The angular orientation 610 indicates the angular orientation of the catheter 522 at 0 (zero) degrees relative to the catheter, and the angular orientation 612 indicates the angular orientation of the tool 520 at 0 (zero) degrees relative to the tool. The resulting rotational offset A1 (or angular offset) can be seen in the image captured from within the lumen 524 by the angular orientation of the longitudinal marker 470 within the 360 ​​degree viewing angle of the imaging sensor 510. It should be understood that the rotation can be achieved in the opposite direction (i.e., clockwise) to Figure 7 Angular orientation 612 is shown. Arrow 620 indicates the angular orientation of the top center of the imaging sensor, and therefore the top center of the image captured by imaging sensor 510.

[0103] Reference now Fig. 8A , showing a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500. The catheter assembly 500 is similar to Figure 5 504, except that the distal end 512 of the tool 520 engages the distal end 516 of the catheter 522. The light source 530 provides illumination light 532 for an object observable through the opening 504.

[0104] Reference now Figures 8B-8C , showing Fig. 8A A simplified cross-sectional view of the distal portion 518 of the catheter assembly 500. In this example, the tool 520 and the catheter 522 have a Figures 8B-8C The cross-sectional view is viewed in the direction of the observable region 540 (ie, looking distally from the distal portion of the tool 520 toward the region 540, as shown by the cross-sectional arrows 8B-8B, 8C-8C). Figure 8B Tool 520 is shown rotationally aligned with catheter 522. Angular orientation 612 of tool 520 is aligned with angular orientation 610 of catheter 522. Arrow 620 again indicates the top middle orientation of imaging sensor 510 relative to central axis 502. Figure 8C Angular rotation 622 is shown to have rotated tool 520 relative to catheter 522 by a rotational offset A1.

[0105] Reference now Fig. 9A , showing that through Figure 8C Representative images of tool 520 of catheter assembly 500. Tool 520 is offset from catheter 522 by a rotational offset A1, which can result in Fig. 9AA captured image 700 of the patient's anatomy is shown. Image 700 is acquired with imaging sensor 510 at a rotational offset A1 relative to catheter 522. Angular orientation 610 again indicates an angular orientation of 0 (zero) degrees for catheter 522, wherein angular orientation 612 again indicates an angular orientation of 0 (zero) degrees for imaging sensor 510. Because the imaging sensor is rotationally offset, captured image 700 is not aligned with the control cables, thus resulting in inaccurate steering of catheter 522 if the image is not corrected to produce an aligned image of the anatomy.

[0106] Reference now Fig. 9B , a representative image 710 representing the image 700 with the rotational offset A1 removed is shown. The control system 112 can receive the captured image 700 and perform a rotation 624 of the image 700 to remove the rotational offset A1. The rotation 624 can rotate the angular orientation 610 of the image 700 by the angular distance of A1 so that the angular orientation 610 and the angular orientation 612 are aligned, thereby producing a modified image 710, which can then be displayed on the user display. With the images properly aligned, the user can use the control cables 630-636 to perform the procedure as if the imaging sensor 510 is aligned with the catheter 522.

[0107] Reference now Fig. 10A , showing a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500. The catheter assembly 500 is similar to Fig. 8A In the catheter assembly of the present invention, except that the distal portion 550 of the tool 520 has a square cross section and the rest of the tool 520 has a circular cross section. The square cross section of the distal portion 550 matches the square cross section of the lumen 524 of the catheter 522. When the distal portion 550 is inserted into the matching square cross section of the lumen 524, the square cross section of the distal portion 550 and the catheter 522 prevents the rotation of the tool 520 relative to the catheter 522. Some rotation can be allowed due to the gap between the tool 520 and the catheter 522, but this is small, and even this small relative rotation can be compensated by using the principles of the present disclosure. In various embodiments, the square cross section of the distal portion 550 can be incorporated into the portion of the tool 520 with a circular cross section. In one example, the distal portion 550 can be incorporated into the circular portion of the tool 520 proximal to the distal end 516 30-40mm.

[0108] Reference now Figures 10B-10C , showing Fig. 10A A simplified cross-sectional view of the distal portion 518 of the catheter assembly 500. Fig. 10B is a cross-section of the proximal portion of the tool 520 having a circular cross-section, and the lumen 524 of the catheter 522 has a square cross-section. Fig. 10Cis a cross-section of the catheter assembly 500 at the distal portion 550, showing the distal portion 550 having a square cross-section and the lumen 524. The cross-sectional view is viewed in the direction of the observable area 540 (ie, looking toward the distal end of the catheter 522). Fig. 10D Tool 520 is shown rotationally aligned with catheter 522. Angular orientation 612 of tool 520 is aligned with angular orientation 610 of catheter 522. Arrow 620 again indicates the top middle orientation of imaging sensor 510 relative to central axis 502. Fig. 10E The tool 520 is shown offset from the catheter 522 by a rotational offset A1. The angular rotation 622 has rotated the distal portion 550 relative to the catheter 522 by a rotational offset A1.

[0109] Because the square cross-section of the distal portion 550 is being inserted into a matching (however, slightly larger) square cross-section of the catheter 522, there are four possible rotational offset values, namely 0 (zero) degrees, 90 degrees, 180 degrees, and 270 degrees. The first quadrant Q1 of the tool 520 can be defined as a 0 (zero) degree rotational offset. The second quadrant Q2 of the tool 520 can be defined as a 90 degree rotational offset. The third quadrant Q3 of the tool 520 can be defined as a 180 degree rotational offset. The fourth quadrant Q4 of the tool 520 can be defined as a 270 degree rotational offset. The four quadrants can be viewed as being divided by lines 614 and 616 (e.g., Fig. 10D and Fig. 10E ) and can be considered to be the quadrant of the image captured by tool 520. Therefore, a quadrant in the reference image is also referred to as the same quadrant of tool 520. When tool 520 is inserted into catheter 522, angular rotation 622 indicates that tool 520 is rotated 90 degrees relative to catheter 522. Catheter 522 can include observable features 470 (as well as additional observable features, such as longitudinal markers 471 (see FIG. 4 ) positioned 180° offset from observable features 470) observable by imaging sensor 510 (not shown) of tool 520. Figure 10G )). When tool 520 is inserted into catheter 522, the captured image from imaging sensor 510 can indicate in which quadrant Q1-Q4 of the captured image observable feature 470 appears, and the indicated quadrant can be used to determine the rotational offset. Removal of rotational offset A1 can be performed as described above to produce a modified image that is rotationally aligned with catheter 522. If observable feature 471 is also included (e.g. Figure 10G), the captured image can be used to determine the quadrant (Q1-Q4) in which the observable feature 471 appears, and the additional observable feature can be used to determine the rotational offset or at least support the determination of the rotational offset A1. However, the observable features 470, 471 can be positioned at any angular orientation relative to each other around the lumen 524, such as positioning the observable features 471 at + / - 30 degrees, 45 degrees, 60 degrees, 90 degrees, 120 degrees, 135 degrees, 150 degrees, and 180 degrees from each other and other orientations within a range from 10 degrees to 350 degrees relative to each other. One or more additional observable features (such as more longitudinal markings 470, 471) can also be used and positioned at desired angular orientations around the lumen, such as at the angles described for the observable features 470, 471.

[0110] In order to correct an image that has been captured at a rotational offset, the rotational offset A1 must be determined. Figure 10G-18B 520 relative to the catheter 522 (except as described herein). Figure 7 The angular orientation can then be used by the control system 112 to determine the rotational offset A1 at which the image is captured, and the captured image can be corrected by rotating the image in a direction that removes the rotational offset A1. Any of these various embodiments can be used to determine the rotational offset A1. After the rotational offset A1 is determined, the captured image can be corrected to a modified image that is rotationally aligned with the catheter 522.

[0111] Reference now Fig.10F , showing a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500. The catheter assembly 500 is similar to Fig. 10AIn the catheter assembly of the present invention, except that the distal portion 550 of the tool 520 is spaced apart from the distal end of the catheter assembly by a distance L5 and is positioned between two other parts of the tool 520 with a circular cross section. The distance L5 can be in the range between 50mm and 200mm. The distance L5 can be, for example, 50mm, 60mm, 70mm, 80mm, 90mm, 100mm or 200mm. The square cross section of the distal portion 550 matches the square cross section of the lumen 524 of the catheter 522. When the distal portion 550 is inserted into the matching square cross section of the lumen 524, the square cross section of the distal portion 550 and the catheter 522 prevents the rotation of the tool 520 relative to the catheter 522. Certain rotation can occur due to the gap between the tool 520 and the catheter 522, but this is smaller, and even this smaller relative rotation can be compensated by using the principle of the present disclosure. Because the distal portion of tool 520 is able to enter lumen 524 prior to engaging the square cross-section of distal portion 550, observable feature 470 may not be included in the proximal portion of catheter 522 to ensure that the orientation of observable feature 470 in the captured image remains consistent with the square lumen of the catheter.

[0112] Reference now Figure 10G , the representative image 750 can be obtained by Fig. 10A and / or Fig.10FThe image sensor 510 is captured and acquired when the light source 530 is turned on. Since the angular orientations 610 and 612 are aligned, the image 750 indicates that the tool 520 is rotationally aligned with the catheter 522. However, if the angular orientation is offset by a rotation offset A1, the observable features 470, 471 in the image 750 can be used to determine the offset. In this embodiment, the observable features 470, 471 are longitudinal marks positioned on the inner surface of the catheter lumen at 180° relative to each other. In addition to the observable features 470, 471, the image 750 can also include several observable objects 472, 473, 474 and 475. The object can interfere with the offset determination, or at least require additional processing to minimize (or minimize) interference. For example, the observable objects 472, 473 can be blood, other body fluids or tissues located in the lumen 524 or deposited on the surface of the lumen. Observable objects 474 can be longitudinal marks that can be produced by longitudinal abrasion along lumen 524 or possibly produced during manufacture of catheter 522. Observable objects can also be areas of saturated illumination, such as area 475 at the corner of image 750. Near the center of image 750 can be seen as the distal end 516 of catheter 522 at the distal end of lumen 524. Central axis 502 indicates the central axis of the lumen, but it should be understood that central axis 502 can be offset from the center of image 750 due to changes in the imaging direction along tool 520 as the tool is being inserted into lumen 524.

[0113] In order to detect and process observable features (in this example, which are longitudinal marks 470, 471), it may be beneficial to remove observable objects 473-475 in image 750 (or minimize the impact of observable objects 473-475 in image 750). This can be performed by various techniques, such as filtering, intensity threshold detection, automatic balancing of color distribution, etc. Bright portions of image 750 that exceed a brightness threshold can be removed by changes in pixel color and intensity values, or brightness can be reduced by changes in pixel color and intensity values. The color distribution and saturation of observable objects in image 750 can be balanced to form a more uniform color distribution or saturation. Corners of image 750 (e.g., observable object 475) that are saturated under lighting conditions can be filtered out using a mask shaped to remove corners (e.g., a circular mask, a custom form mask, etc.). As described in more detail below, after removing or at least minimizing the effects of observable objects 472, 473, 474 and 475 on image processing, the control system is able to process (one or more) modified images to determine the orientation of observable features 470, 471 with reduced processing time and increased accuracy.

[0114] The visibility and detection of observable features 470, 471 can be improved using color filtering techniques that remove or minimize colors associated with the natural anatomical environment (such as red) so that non-naturally occurring colors (e.g., blue and yellow) that can be used to form observable features are more visible in the modified image. For example, the pixels of image 750 can be separated into hue saturation values ​​(HSV). Fig. 10H A representative diagram 751 of pixels 752 of an image 750 grouped by hue and measured by saturation is illustrated. Pixels in region 490 may correspond to yellow pixels (e.g., the color of observable feature 471) that exceed a predetermined saturation value. Pixels in region 491 may correspond to blue pixels (e.g., the color of observable feature 470) that exceed a predetermined saturation value. Pixels in region 494 may correspond to hues and saturation levels outside of the hue-saturation values ​​associated with observable features 470, 471. For example, pixels in region 491 may correspond to red pixels or other hues associated with a natural anatomical environment (e.g., the color of blood and other fluids or tissues in a natural environment). To enhance detection of yellow observable feature 471 and blue observable feature 470, pixels in region 494 may be filtered out, neutralized, or otherwise modified so that the processed image removes the color of blood, allowing blue and yellow observable features to be more visible in the processed image.

[0115] The visibility and detection of observable features 470, 471 may also or alternatively be improved using masking techniques. Fig.10I , a representative image 752 indicating the modified image 750 is provided. The circular filter 480 can be used to filter out the saturated corner 475. The radius R1 of the circular filter can be adjusted as needed to remove the object 475 and to remove additional observable objects (e.g., 472, 473). The observable objects 472, 473, 474 that are not filtered by the circular filter 480 can be filtered using other known image processing techniques (such as color filtering), thereby leaving the observable features 470, 471 for determining the rotational offset A1. It should be noted that the rotational offset A1 of the image 752 is zero because the angular orientations 610 and 612 are aligned.

[0116] After removing observable objects 472, 473, 474 and 475 (or minimizing the influence of observable objects 472, 473, 474 and 475), the control system 112 can determine in which of the quadrants Q1-Q4 each observable feature 470, 471 is located. In an alternative embodiment, the image can be divided into segments other than quadrants. The position of each observable feature 470, 471 can be determined by detecting the concentration of the HSV of each observable feature 470, 471 and determining the quadrant in which the HSV concentration is located. The HSV is different for each observable feature 470, 471, so that each observable feature 470, 471 can be identified independently of another. The expected position of the HSV concentration can also be used to detect observable features. For example, in this embodiment, it is known that the blue visible feature 470 is located directly across the lumen from the yellow visible feature 471, so the hue saturation values ​​of each color will be expected to be relatively concentrated to each other. The control system 112 can also calculate the center of mass (e.g., center of mass 496, 498) of each observable feature 470, 471, and determine the quadrant in which each center of mass is located. Determining the quadrants Q1-Q4 shown as being separated by lines 616, 614 assumes that the tool 520 and lumen 524 have a square cross-section, and only four possible orientations are possible. However, these techniques can also be used to determine the angular orientation of each observable feature 470, 471 in the captured image from 0 degrees to 360 degrees. The rotational offset A1 can be calculated based on the angular orientation of one or more observable features 470, 471.

[0117] Reference now Fig.10J, a representative image 754 indicating a modified image 750 is provided. A custom formed filter region 484 is provided to filter out most, if not all, observable objects 472, 473, 474, and 475 from the image 750. The filter region 484 is shown as being slightly larger than the observable features 470, 471 and the distal end 516. The filter region 484 can be larger or smaller than the filter region shown in the figure, and can have various shapes and sizes as needed to provide the desired filtering effect on the captured image. The shape can be determined by the known shape of the combination of the observable features 470, 471 and the distal end 516. Such a filter region 484 can be translated and rotated up / down and left / right as indicated by arrows 626 to align with the observable features 470, 471. When the HSV of the pixels in the filter region 484 exceeds the saturation threshold, the alignment of the filter region 484 with the observable features 470, 471 can be determined. The angular orientation of each observable feature 470, 471 can then be determined as described above. It should be understood that more or fewer observable features 470, 471 can be used to determine the orientation of the tool 520 relative to the catheter 522. Moreover, as described above, the angular offset A2 from one observable feature (e.g., 470) to a second observable feature (e.g., 471) can be an angular offset other than 180 degrees.

[0118] Reference now Figure 10K , a representative image 762 indicating the modified image 760 is provided. The image 760 can be Fig. 10A and / or Fig.10F 510 and is acquired with the light source 530 turned on. Since the angular orientations 610 and 612 are aligned, the image 760 indicates that the tool 520 is rotationally aligned with the catheter 522. However, if the angular orientation is offset by a rotational offset A1, the observable features 470, 471 in the image 760 can be used to determine the offset. The image 760 can include one or more of the observable features 470, 471. In this example, the observable features 470, 471 are offset by an angular offset A2 of 90 degrees relative to each other. A custom formed filter area 484 is provided to filter out most, if not all, observable objects 472, 473, 474, and 475 from the image 760. The filter area 484 is shown as being slightly larger than the observable features 470, 471 and the distal end 516. The shape can be determined by the known shape of the combination of the observable features 470, 471 and the distal end 516. The filter area 484 can be as shown by arrow 626 (see Figure 10K) are translated up / down and left / right and rotated to align with observable features 470, 471. When the HSV of pixels in filter region 484 exceeds a saturation threshold, the alignment of filter region 484 with observable features 470, 471 can be determined. The angular orientation of each observable feature 470, 471 can then be determined as described above.

[0119] Reference now Fig.10L , a representative image 772 indicating the modified image 770 is provided. The image 770 can be Fig. 10A and / or Fig.10F 510 is captured by the image sensor 510 and is acquired with the light source 530 turned on. Since the angular orientations 610 and 612 are aligned, the image 770 indicates that the tool 520 is rotationally aligned with the catheter 522. However, if the angular orientation is offset by a rotational offset A1, the observable features 470, 471 in the image 770 can be used to determine the offset. The image 770 can include one or more of the observable features 470, 471. In this example, the observable features 470, 471 are offset from each other by an angular offset A2 greater than 90 degrees but less than 180 degrees. A custom-formed filter area 484 is provided to filter out most (if not all) observable objects 472, 473, 474, and 475 from the image 770. The filter area 484 is shown to be slightly larger than the observable features 470, 471 and the distal end 516. The shape can be determined by the known shape of the combination of the observable features 470, 471 and the distal end 516. The filter area 484 can be as shown by arrow 626 (see Figure 10K ) are translated up / down and left / right and rotated to align with observable features 470, 471. When the HSV of pixels in filter region 484 exceeds a saturation threshold, the alignment of filter region 484 with observable features 470, 471 can be determined. The angular orientation of each observable feature 470, 471 can then be determined as described above. Figure 10M It is illustrated that observable features 470 , 471 are not limited to quadrants of lumen 524 .

[0120] Reference now Figure 10M , a representative image 782 indicating the modified image 760 is provided. Image 780 can be Fig. 10A and / or Fig.10F530 is captured by the image sensor 510 and is acquired with the light source 530 turned on. Because the angular orientations 610 and 612 are rotationally offset from each other, the image 780 indicates that the tool 520 is rotationally offset relative to the catheter 522. The image 780 can include one or more of the observable features 470, 471. In this example, the observable features 470, 471 are offset from each other by an angular offset A2 of 180 degrees. A custom formed filter region 484 is provided to filter out most, if not all, observable objects 472, 473, 474, and 475 from the image 750. The filter region 484 is shown as being slightly larger than the observable features 470, 471 and the distal end 516. The shape can be determined by the known shape of the combination of the observable features 470, 471 and the distal end 516. The filter region 484 can be as shown by arrow 626 (see Figure 10K ) is translated up / down and left / right and rotated to align with observable features 470, 471. In this example, tool 520 has been rotated 90 degrees within catheter 522, which can be the case when lumen 524 and tool 520 have a square cross-section. When the HSV of the pixels in filter region 484 exceeds a saturation threshold, the alignment of filter region 484 with observable features 470, 471 can be determined. The angular orientation of each observable feature 470, 471 and the rotational offset A1 of the captured image can then be determined as described in the present disclosure. It should be understood that multiple observable features can be used in all other embodiments described in the present disclosure.

[0121] Reference now Fig.11 , a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500 is shown. The observable feature 560 is generated at a distance L2 from the distal end 516 of the catheter 522 when the light source 530 is at the distal end 512 of the tool 520. The optical waveguide 552 can be used to supply optical energy to the light source 530. The light 532 emitted from the light source 530 can illuminate the observable feature 560, wherein at least a portion 562 of the observable feature 560 is observed by the image sensor 510. Therefore, the angular orientation of the observable feature portion 562 in the captured image can be determined as previously described, and since the angular orientation of the observable feature 560 on the catheter 522 is known, the angular orientation of the observable feature 560 in the captured image can be used to determine the rotational offset A1. After the rotational offset A1 is determined, the captured image can be corrected to a modified image that is rotationally aligned with the catheter 522.

[0122] Reference now Fig.12, a representative perspective view of the catheter assembly 500 is shown. The observable feature 560 is generated at a distance L3 from the distal end 516 of the catheter 522. At least a portion 562 of the observable feature 560 can be observed by the image sensor 510. Therefore, the angular orientation of the observable feature portion 562 in the captured image can be determined as previously described, and since the angular orientation of the observable feature 560 on the catheter 522 is known, the angular orientation of the observable feature 560 in the captured image can be used to determine the rotational offset A1. After the rotational offset A1 is determined, the captured image can be corrected to a modified image that is rotationally aligned with the catheter 522.

[0123] Reference now Fig.13 , a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500 is shown. The observable feature 560 produces a distance L4 from the distal end 516 of the catheter 522 when the light source 530 is at the distal end 516. The optical waveguide 552 can be used to supply optical energy to the light source 530. The light 532 emitted from the light source 530 can illuminate the observable feature 560, wherein at least a portion of the light 566 is reflected by the reflector 564 (e.g., mirror) into the observable area 540, wherein the reflected light 566 can be captured by the image sensor 510. The reflector 564 can be any surface of the reflector surface and / or diffusely reflected light. Therefore, the angular position of the observable feature (in this example, i.e., the reflected light 566) in the captured image can be determined, and since the angular position of the observable feature 560 on the catheter 522 is known, the angular position of the observable feature 560 in the captured image can be used to determine the rotational offset A1. After the rotational offset A1 is determined, the captured image can be corrected to a modified image that is rotationally aligned with the catheter 522 .

[0124] Reference now Fig.14A , a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500 is shown. Two light sources 530a, 530b can be arranged at the distal end 516 of the catheter 522 oppositely (i.e., approximately 180 degrees apart from each other) around the circumference of the catheter 522. Optical waveguides 552a, 552b can supply optical energy to the light sources 530a, 530b, respectively. As described below with respect to Figures 15A-17B As described, these light sources 530a, 530b can be used together or individually to determine the rotational offset of the tool 520 relative to the catheter 522.

[0125] Reference now Fig. 14B, two light sources 530a, 530b are relatively arranged around the distal end 516 of the catheter 522, wherein the light source 530a is at an angular orientation 610, and the light source 530b is generally at an angular orientation of 180 degrees around the catheter 522 relative to the light source 530a. It should be understood that these light sources 530a, 530b can be arranged at angular orientations other than 180 degrees apart. Additional light sources can also be used to determine the rotational offset A1 between the tool 520 and the catheter 522. The angular orientation 612 of the tool 520 is shown as being rotated by a rotational offset A1 relative to the angular orientation 610. Figures 15A-17B is rotated relative to the catheter 522 by a rotational offset A1 Fig. 14B Representative images captured by the image sensor 510 of the present invention. The two light sources 530a, 530b can be controlled in various ways to facilitate the determination of the rotational offset A1. The tool 520 can include additional light sources, for example, to illuminate an area of ​​the patient's anatomical structure during the procedure. Therefore, the imaging sensor 510 can capture images using the illumination provided by the light source on the tool 520 and the illumination light 532a, 532b provided by the corresponding light sources 530a, 530b. The light sources 530a, 530b are generally used to produce an uneven distribution of light in the observable area 540 (which can be viewed as an observable feature), and by processing images with uneven light distribution using one or more processors in the control system 112, the rotational offset A1 of the tool 520 within the catheter 522 can be determined, and future image capture can be corrected to remove the rotational offset.

[0126] Reference now Fig. 14C , shows a simplified partial cross-sectional view of the distal portion 518 of the catheter assembly 500. A fluid jet nozzle 570 can be disposed on the catheter 522 at the distal end 516 of the catheter 522. A fluid flow channel 574 can supply the fluid jet nozzle 570 with fluid.

[0127] Reference now Fig.14D , the fluid injection nozzle 570 is positioned on the catheter 522 at an angular orientation 610. The angular orientation 612 of the tool 520 is shown rotated relative to the angular orientation 610 by a rotational offset A1. Fig.14DThe image captured by the image sensor 510 can be rotated relative to the conduit 522 by a rotational offset A1. The fluid jet nozzle 570 can be controlled to facilitate determination of the rotational offset A1 by jetting a fluid 572 at an angular orientation 610 in front of the imaging sensor 510. The tool 520 can include additional fluid sources, if desired. The imaging sensor 510 can capture images using illumination light 532 provided by a light source 530 on the tool 520 and / or illumination from other light sources mounted to the conduit 522. The fluid nozzle can produce a fluid jet pattern (which can be viewed as an observable feature) in the observable area 540, and by processing the image in which the fluid jet pattern appears using one or more processors in the control system 112, the rotational offset of the tool 520 within the conduit 522 can be determined, and future image capture can be corrected to remove the rotational offset.

[0128] Reference now Fig.15A , the representative image 720 is Fig. 14B The image 720 is captured by the image sensor 510 and is acquired without turning on either of the light sources 530a, 530b. The image 720 can be used as a reference image to compare with another captured image (e.g. Fig. 15B The image 730) is compared to determine the difference between the image with one or more of the light sources 530a, 530b turned on and the reference image with both light sources 530a, 530b turned off.

[0129] Reference now Fig. 15B , the representative image 730 can be represented by Fig. 14BThe image sensor 510 is captured by the image sensor 510 and is acquired with the light source 530a turned on and the light source 530b turned off. The uneven light distribution 810 can be generated when the light source 530a is turned on. The uneven light distribution 810 can have a boundary 830 between the uneven light and the light distribution in the image 720 that is not changed by turning on the light source 530a. The light distribution 810 can show a gradient of the light distribution from the boundary 830 to the high intensity area 820. The light distribution 810 can be a varying intensity of light, a color gradient, a pattern of light, etc. The control system 112 can compare the images 720, 730 and adjust a subset of pixels from the area of ​​the image 730 that is unchanged when the light source 530a is turned on to 0 (zero, i.e., red=blue=green=0). In the case where the unchanged subset of pixels is set to zero, the modified image 730 can only show a subset of pixels associated with the light distribution 810. The visible subset of pixels can be processed to determine the centroid 800 of the light distribution 810, thereby identifying the angular position of the light source 530a on the catheter, since the highest concentration of the light distribution 810 will be at the light source, and thereby identifying the angular position 610 in the modified image 730 of the light source 530a on the catheter 522. By comparing the angular position 610 in the modified image 730 of the light source 530a to the expected angular position 612 of the imaging sensor 510, the rotational offset A1 can be determined.

[0130] Reference now Fig. 15C , a representative image 740 is shown and represents the image 730 modified to remove the rotational offset A1. The control system 112 can rotate 624 the image 730 to compensate for the rotational offset A1 and produce a modified image 740 that is rotationally aligned with the tool 520 and can be displayed to the user. Other images captured by the imaging sensor 510 at this rotational offset A1 can be automatically adjusted before being displayed to the user. The rotational offset A1 can also be used to adjust the steering of the control cables for the distal portion 518 of the articulating catheter.

[0131] Reference now Fig.16A , the representative image 720 is Fig. 14B The image 720 is captured by the image sensor 510 and is acquired without turning on either of the light sources 530a, 530b. The image 720 can be used as a reference image to compare with another captured image (e.g. Fig. 16B The image 730) is compared to determine the difference between the image with one or more of the light sources 530a, 530b turned on and the reference image with both light sources 530a, 530b turned off.

[0132] Reference now Fig. 16B , the representative image 730 can be represented by Fig. 14BThe image sensor 510 is captured and acquired with the light source 530b turned on and the light source 530a turned off. The uneven light distribution 860 can be generated when the light source 530b is turned on. The uneven light distribution 860 can have a boundary 880 between the uneven light and the light distribution of the image 720 that is unchanged when the light source 530b is turned on. The light distribution 860 can show a gradient of the light distribution from the boundary 880 to the high intensity area 870. The light distribution 860 can be a varying intensity of light, a color gradient, a pattern of light, etc. The control system 112 can compare the images 720, 730 and adjust a subset of pixels that are unchanged when the light source 530b is turned on to zero. In the case where the subset of pixels is set to zero, the modified image 730 can only show a subset of pixels associated with the light distribution 860. The visible subset of pixels can be processed to determine the centroid 850 of the light distribution 860, thereby identifying the angular position 610 in the modified image 730 of the light source 530b (and thus the conduit 522) relative to the angular position 612 of the imaging sensor 510 (i.e., the top center of the image), because the highest concentration of the light distribution 860 will be at the light source, and thus the angular position 610 in the modified image 730 of the light source 530b can be identified. By comparing the angular position 610 of the light source 530b to the angular position 612 of the imaging sensor 510, the rotational offset A1 can be determined.

[0133] Reference now Fig. 16C , a representative image 740 is shown and represents the image 730 modified to remove the rotational offset A1. The control system 112 can rotate 624 the image 730 to compensate for the rotational offset A1 and produce a modified image 740 that is rotationally aligned with the tool 520 and can be displayed to the user. Future images captured by the imaging sensor 510 at this rotational offset A1 can be automatically adjusted to remove the rotational offset A1 before being displayed to the user. The rotational offset A1 can also be used to adjust the steering of the control cables for the distal portion 518 of the articulating catheter.

[0134] Reference now Fig.17A , the representative image 720 is Fig. 14B The image 720 is captured by the image sensor 510 and is acquired without turning on either of the light sources 530a, 530b. The image 720 can be used as a reference image to compare with another captured image (e.g. Fig. 17B The image 730) is compared to determine the difference between the image with one or more of the light sources 530a, 530b turned on and the reference image with both light sources 530a, 530b turned off.

[0135] Reference now Fig. 17B , the representative image 730 can be represented by Fig. 14BThe image sensor 510 is captured by the image sensor 510 and is acquired with the light sources 530a, 530b turned on. The uneven light distribution 810, 860 can be generated when the light sources 530a, 530b are turned on. The uneven light distribution 810, 860 can have a corresponding boundary 830, 880 between the uneven light distribution 810, 860 and the light distribution of the image 720 that is unchanged when the light sources 530a, 530b are turned on. The light distribution 810, 860 can show a gradient of the light distribution from the corresponding boundary 830, 880 to the corresponding high intensity area 820, 870. The light distribution 810, 860 can be a varying intensity of light, a color gradient, a pattern of light, etc. The control system 112 can compare the images 720, 730 and adjust a subset of pixels from the area of ​​the image 730 that is unchanged when the light sources 530a, 530b are turned on. With the unchanged subset of pixels set to zero, the modified image 730 may only show a subset of pixels associated with the light distributions 810, 860. The remaining subset of pixels can be processed to determine the centroids 800, 850 of the respective light distributions 810, 860, thereby identifying the angular position 610 in the image 730 of the light sources 530a, 530b on the conduit, since the highest concentration of the light distributions 810, 860 can be at the respective light sources, and thereby identifying the angular position 610 of the light sources 530a, 530b on the conduit 522. By comparing the angular position 610 of the light sources 530a, 530b with the angular position 612 of the imaging sensor 510, the rotational offset A1 can be determined.

[0136] Reference now Fig. 17C , a representative image 740 is shown and represents the image 730 modified to remove the rotational offset A1. The control system 112 can rotate 624 the image 730 to compensate for the rotational offset A1 and produce a modified image 740 that is rotationally aligned with the tool 520 and can be displayed to the user. Future images captured by the imaging sensor 510 at this rotational offset A1 can be automatically adjusted to remove the rotational offset A1. The rotational offset A1 can also be used to adjust the steering of the control cable for the distal portion 518 of the articulating catheter.

[0137] Reference now Fig.18A , the representative image 730 can be represented by Fig. 14BThe image sensor 510 is captured at a rotational offset A1 and is acquired with one or more of the light sources 530a, 530b turned on. For this example, at least the light source 530a is turned on and illuminates one or more objects in the observable area 540 of the imaging sensor 510. The light source 530a can be a waveguide having a manufacturing feature at its distal portion that projects a pattern of light onto the (one or more) objects in the observable area 540. The pattern 890 can be any pattern of light that can be detected by the imaging sensor 510 and used to determine the rotational offset of the imaging sensor 510. Therefore, the pattern 890 can be a grid pattern as shown, or a unique arrangement of illumination spots (e.g., similar to a constellation diagram). The spots can be uniform sizes with a unique arrangement, or the spots can be uniform shapes or patterns of light that illuminate (one or more) objects with non-uniform colors, or a combination thereof. The non-uniform color can be a varying color temperature, a color gradient, a color pattern, or a combination thereof. The light source 530a (or 530b) can generate light 532a (or 532b) that illuminates the object with high-frequency light (e.g., IR light) that is not in the human visible spectrum but can be detected and captured by the imaging sensor 510. In the case of low-frequency light, the light sources 530a, 530b can be periodically turned on and off to limit the time that the user sees the illumination from the light sources 530a, 530b in the captured image. In the case of high-frequency light, the light sources 530a, 530b can remain on during the procedure to maintain the offset correction integrity because the user cannot see the high-frequency light.

[0138] One or more of the light sources 530a, 530b can project structured light onto the object(s) in the observable area 540. In this example, the structured light 890 is a unique grid pattern. Fig. 14B The imaging sensor 510 is at a rotational offset A1 within the catheter 522. The imaging sensor 510 is capable of capturing Fig.18A The control system 112 can process the image 730 to adjust the subset of pixels not associated with the structured light 890 to zero, and then process the visible subset of pixels to determine the angular orientation of the structured light 890 in the image 730. The angular orientation of the structured light 890 can be used to determine the angular position 610 of the catheter 522 relative to the tool 520, and thereby determine the orientation of the tool 520 within the catheter 522. The control system 112 can determine the rotational offset A1 by comparing the angular position 610 of the catheter 522 in the image 730 with the angular position 612 of the imaging sensor 510.

[0139] Reference now Fig.18B, a representative image 740 is shown and represents the image 730 modified to remove the rotational offset A1. The control system 112 can rotate the image 730 (e.g., as shown by arrow 624) to compensate for the rotational offset A1 and produce a modified image 740 that can be displayed to the user. Future images captured by the imaging sensor 510 at this rotational offset A1 can be automatically adjusted to remove the rotational offset A1 before being displayed on the user display. The rotational offset A1 can also be used to adjust the steering of the control cable for the distal portion 518 of the articulating catheter.

[0140] Thus, observable features can include longitudinal marks 470, protruding observable features 560, non-uniform light distributions 810, 860, and structured light pattern 890. Rotational offset A1 can be continuously and / or periodically updated, and corrections to the rotated image can be updated throughout the procedure with the new offset A1 if necessary.

[0141] Method 1900 Figures 19A-19C 1902 to 1978. Not all of the illustrated processes 1902 to 1978 will be performed in all embodiments of the method. In addition, operations not illustrated in the flowchart may be included before, after, between, or as part of the processes 1902 to 1978. Figures 19A-19C In some embodiments, one or more of the processes may be at least partially implemented in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of the control system 112), may cause the one or more processors to perform one or more of the processes.

[0142] Reference now Figures 19A-19C , a representative flow chart of a method 1900 for controlling the collection of images during a procedure, determining a rotational offset of an imaging sensor, and correcting a captured image by removing the rotational offset before displaying the captured image to a user. Each of the illustrated methods includes a set of operations or processes. Not all of the illustrated processes will be performed in all embodiments of the method. In addition, one or more processes not explicitly illustrated may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more of the processes may be at least partially implemented in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of the control system 112), may cause the one or more processors to perform one or more of the processes.

[0143] Fig.19Ais a representative flow chart of a supervisory process 1901 capable of controlling the collection and modification of images during a procedure. Process 1901 can initiate and control process 1950 (see Fig.19B ) to determine the rotational offset of tool 520 in catheter 522 and receive the rotational offset and quality metrics of the collected and calculated data. Process 1950 can start process 1958 (see Fig.19C ) to analyze the captured images to determine the rotational offset. The following paragraphs describe these processes in more detail.

[0144] The supervision process 1901 begins at operation 1902. In some embodiments, an indication of the start of the process can be received from a user at 1902. Once the process 1901 is started at operation 1902, operation 1904 determines whether the tool 520 is present in the catheter 522. The presence indicator can be provided by operation 1918, which monitors the presence sensor to determine whether the tool 520 is partially or completely inserted into the catheter 522 or not inserted at all based on the insertion mark detected by one or more target readers (or sensors). When it is determined that the tool 520 is partially inserted, the supervision process 1901 can detect longitudinal markings within the lumen 524 of the catheter 522 to determine the rotational offset A1 of the tool 520 in the catheter 522. However, when it is determined that the tool 520 is fully inserted in the catheter 522, the longitudinal markings within the lumen 524 will not be observable. Thus, when the tool 520 is fully inserted in the catheter 522, the control system 112 may disable image collection that would be used for purposes of determining the rotational offset A1 if only the longitudinal marker(s) 470, 471 were being used.

[0145] The supervisory process 1901 can also detect observable features (e.g., 560, 810, 860, and 890) other than the longitudinal markers that can be used to determine the rotational offset A1 of the tool 520 in the catheter 522. When other observable features (e.g., 560, 810, 860, and 890) are being used, then image capture can be disabled during insertion of the tool 520 and enabled when the tool 520 is fully inserted. If one or more of the longitudinal markers (one or more) 470, 471 and other observable features (e.g., 560, 810, 860, and 890) are being used, then the image collection rate can be slower before the tool 520 enters the catheter 522, and then increase the image collection rate when the tool 520 is partially or fully inserted. The control system 112 can enable image collection after the tool 520 enters the catheter 522 so as to capture images with one or more of the longitudinal markers (one or more) 470, 471 in the image. After the tool 520 is fully inserted, the control system 112 can continue image collection to capture images with one or more of the observable features 560, 810, 860, and 890 in the image (or stop image collection if none of the other observable features 560, 810, 860, and 890 are being used). When it is determined that the tool 520 is not inserted into the catheter 522, then the supervisory process 1901 can remain in operation 1904 waiting for a clear / positive indication that the tool is at least partially inserted into the catheter.

[0146] When an explicit indication is received, the supervisory process 1901 can proceed to operation 1906, which can request a quality score for the orientation (or angular position) of the observable features 470, 471, 560, 810, 860, 890 captured in the image and the information collected and calculated (such as the angular position of the observable features). Process 1950 collects images and determines the rotational offset A1 of the observable features 470, 471, 560, 810, 860, 890 in the image(s), as well as a weighted quality metric that can provide confidence in the information provided back to the supervisory process 1901. If the supervisory process 1901 determines that the quality of the information from operation 1950 is below a threshold level, then operation 1910 can output an error message to the control system 112, which can alert a user and / or log the error message for later review.

[0147] If the quality of the information from operation 1950 is above a threshold level, then operation 1912 can use that information to rotate the operating image to remove the rotation offset A1 (such as by Fig. 9A , Fig. 9B) and / or adjust the manipulation of the control cables 630, 632, 634, 636. Operation 1912 can continue while operation 1914 continues to monitor the tool presence signal supplied by operation 1918. If the tool presence signal remains activated, indicating no change, then operation 1912 can continue. If the tool presence signal is no longer activated (i.e., if the tool 520 has been moved from being fully inserted, or removed entirely from the catheter 522), then process 1901 can proceed to operation 1916, which can again request the initiation of the observable feature process 1950. If the tool 520 remains partially inserted in the catheter 522, this would be desirable, and the observable feature process 1950 can again determine the orientation of the tool 520 in the catheter by detecting observable features, in which case the observable features can be (one or more) longitudinal markings 470, 471. Process 1901 can also proceed from operation 1916 to operation 1902 to restart the process, and proceed to operation 1904 to again wait for a clear / positive tool presence indicator before continuing operations in process 1901.

[0148] refer to Fig.19B , process 1950 can start at operation 1952 and proceed to operation 1954, which can wait for an indication from supervisory process 1901. When rotational excursion of tool 520 within catheter 522 is expected, supervisory process 1901 can send an explicit signal to process 1950 to proceed to the next operation. In operation 1956, an image (or images) can be captured by an imaging sensor, wherein the captured image(s) can include observable feature(s) 470, 471, 560, 810, 860, 890.

[0149] In operation 1958, the image(s) are processed to identify observable feature(s) 470, 471, 560, 810, 860, 890 and their rotational offset A1 relative to an expected orientation in the image (e.g., angular orientation 612 as a top middle orientation in the image).

[0150] In operation 1960, the information determined in operation 1958 can be stored in a memory buffer for later analysis and calculation. Operation 1962 checks whether the supervisory process 1901 has requested data. If a data request has not been indicated, the process 1950 continues to repeat processes 1956, 1958, 1960 and 1962 until a data request is indicated. Since the process 1950 can be started while the tool 520 is being installed in the catheter 522, the memory buffer can store images captured within the catheter 522 and / or (one or more) calculated rotational offsets A1. In one embodiment, the memory buffer can continue to store the most recent 50 images, pushing out the oldest image and inputting the newest image.

[0151] A data request may occur when an explicit / positive tool presence signal is received by the supervisory process 1901, which can mean that multiple images have been stored in the memory buffer because images were collected during installation of the tool 520. If a data request is indicated, then the process 1950 can proceed to operation 1964, which can analyze the data in the memory buffer, calculate a rotational offset for each image, and calculate a statistical pattern across multiple images that best describes the rotational orientation of the tool 520 within the catheter 522. Operation 1966 can calculate a quality metric indicating the confidence of the calculated rotational offset of the tool 520. The quality metric can include a determination of the confidence that the image contains an observable feature (e.g., a longitudinal marker) and the confidence of the angular orientation of the observable feature in the image for each processed image. Operation 1968 reports the rotational offset and quality metric to the supervisory process 1901, and returns the process 1950 to operation 1954 to again wait for a start algorithm indication from the supervisory process 1901.

[0152] In general, when the tool presence indication is negative, the supervisory process 1901 can start the algorithm, but not request data. When the tool presence indication is positive, the start algorithm indication can be negated, with the data request indication being positive, causing the calculated data to be transmitted to the supervisory process 1901. The supervisory process 1901 can involve a system that detects rotational offset of the tool 520 in the catheter as the tool is being inserted in the catheter 522. However, the process 1901 can also support embodiments in which the tool 520 is fully installed (i.e., inserted) in the catheter 522 and the images captured by the imaging sensor 510 include observable features 560, 810, 860, 890. The start algorithm indication and data request indication can be supplied to the process 1950 at times other than those mentioned above, and can cause multiple images to be collected when the tool 520 is fully installed, and in some embodiments, data requests are indicated at various times during the procedure as long as the tool 520 remains fully installed in the catheter 522. The process 1950 can provide analysis of multiple (e.g., 20, 30, 40, 50, 60, 70, 80, 100, etc.) recently captured images when a data request (e.g., requesting the orientation of the tool) is received (such as when the tool 520 is fully inserted). In addition, the process 1950 can continuously analyze multiple images (e.g., 20, 30, 40, 50, 60, 70, 80, 100, etc.) as they are captured and stored in the buffer. Then when a data request (e.g., requesting the orientation of the tool) is received, the process can immediately respond with the requested data for the multiple images without further analysis. Alternatively or in addition, the process 1950 can provide analysis of multiple (e.g., 20, 30, 40, 50, 60, 70, 80, 100, etc.) captured images after the tool 520 is fully inserted into the catheter 522. For observable features 560, 810, 860, and 890, these features may not be visible when tool 520 is not fully inserted. Therefore, process 1950 may wait for an indication that the tool is fully inserted before sending a data request, which causes data for images captured after tool 520 is fully inserted to be supplied to control system 112 regardless of whether analysis of the plurality of images is performed before or after receiving the data request.

[0153] Reference now Fig.19C, operation 1958 can include multiple operations as shown. Operation 1970 can receive a captured image, where the captured image can include observable features 470, 471, 560, 810, 860, 890. The captured image can be processed by one or more operations to determine the orientation of the observable feature. The process can include removing a subset of pixels associated with background or obstructive objects (e.g., processes 1972, 1974) and / or evaluating a set of multiple images (e.g., processes 1980-1986).

[0154] Operation 1972 can modify the captured image to remove the background and leave a subset of pixels associated with observable features in the image. Removing the background can be done by comparing the baseline image to the captured image having observable features 470, 471, 560, 810, 860, 890. The differences between the two images should be largely due to the observable features 470, 471, 560, 810, 860, 890. Therefore, removing a subset of pixels that are substantially the same between the two images will leave the observable features still observable in the modified image. It should be understood that removing a subset of pixels may not necessarily mean that the pixels are deleted from the image, but are simply changed to cause the pixels in the subset to be red=blue=green values, which is "0 (zero)".

[0155] Another way to remove pixels associated with the background (or set to zero) can be to perform a "white balance correction" by identifying the background color and setting the background pixels to a neutral color, and leaving the contrast of the observable feature(s) 470, 471, 560, 810, 860, 890. The observable feature(s) 470, 471, 560, 810, 860, 890 can then be extracted into another image without the background. Once the observable feature(s) 470, 471, 560, 810, 860, 890 have been isolated in operation 1972, operation 1974 can compare the modified image containing the observable feature(s) 470, 471, 560, 810, 860, 890 (or the centroid of the observable feature) to a plurality of model images, and the angular orientation of the observable feature can be determined based on one or more matches to the model images.

[0156] Processes 1980-1986 may use multiple images captured over a period of time to determine the orientation of observable features. At process 1980, one or more of the captured images may optionally be excluded from further analysis. Exclusion of captured images may be based on a time value associated with the captured image, such as a time value when the image was captured compared to when the tool was fully positioned in the catheter. For example, when the tool 520 is positioned at the distal end of the catheter 522, the captured image may become more obstructed as mucus and other anatomical debris fills the image. In some embodiments, if the last six seconds of captured images (e.g., at approximately 20 captured images per second) prior to tool placement constitute the set of captured images received, the last two seconds of captured images may be excluded from further analysis due to the likelihood that they will be highly obstructed.

[0157] At process 1982, image processing techniques may be used to evaluate the possible orientation of observable features (e.g., 470, 471, 560, 810, 860, 890) in the captured images. For each captured image in the set of captured images (or in the remaining subset in the case where the image is excluded at process 1980), pixel analysis may be performed. Based on the pixel analysis, each captured image may be assigned a possible orientation value associated with the determined orientation of the observable feature, or may be assigned an uncertainty value associated with the determination that the pixel analysis cannot establish the orientation of the observable feature with sufficient probability. For example, if the observable feature is a longitudinal stripe 470, the pixel analysis may determine which pixels are associated with the color, size, or other characteristics of the stripe 470, and may determine the quadrant (Q1-Q4) of the tool in which the pixels associated with the stripe 470 are located. Thus, each captured image may be associated with a possible quadrant in which the stripe 470 is located. If pixel analysis of the captured image is unable to identify pixels associated with stripe 470 or find pixels associated with stripe 470 in multiple quadrants (eg, due to color distortion), an indeterminate value may be assigned.

[0158] At process 1984, the possible determined orientations of the observable features may be weighted with confidence scores based on criteria including, for example, the number of pixels in the captured image associated with the possible determined orientation, the number of pixels in the captured image associated with orientations other than the possible determined orientation, the temporal order of the captured images (e.g., an image captured six seconds before full set-up may be more valuable than an image captured two seconds before full set-up). In one example, a captured image having 8,000 pixels associated with the possible determined orientation may receive a higher confidence score than a captured image having 4,000 pixels associated with the possible determined orientation.

[0159] At process 1986, a composite evaluation of the possible determined orientations of the observable feature in the plurality of captured images may be performed to determine a final orientation of the observable feature. In some embodiments, the composite evaluation may include developing an average of the possible orientation values ​​to determine a final composite orientation of the observable feature. In some embodiments, the composite evaluation may include developing a combination of possible orientation values, such as a weighted average of the possible orientation values ​​(i.e., weighted by the confidence score), to determine a final composite orientation of the observable feature.

[0160] At process 1976 , a rotational offset A1 of the tool within the catheter can be determined based on the determined final orientation (eg, angular position) of the observable features 470 , 471 , 560 , 810 , 860 , 890 . Operation 1978 can provide process 1958 with a rotational offset.

[0161] One or more elements in the embodiments of the present invention can be implemented with software to execute on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present invention are essentially code segments that perform necessary tasks. Program segments or code segments can be stored in a processor-readable storage medium or device, and can be downloaded in the form of a computer data signal contained in a carrier wave through a transmission medium or a communication link. Processor-readable storage devices may include any medium (including optical media, semiconductor media, and magnetic media) capable of storing information. Processor-readable storage device examples include electronic circuits, semiconductor devices, semiconductor storage devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded through computer networks such as the Internet and intranets.

[0162] Note that the processes and displays presented are not inherently related to any particular computer or other device. The structures required for various these systems will appear as elements in the claims. In addition, the embodiments are described without reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0163] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the present invention and not restrictive, and that since various other modifications may be conceived by those skilled in the art, the embodiments of the present invention are not limited to the specific embodiments shown and described.

[0164] Various aspects of the subject matter described in this article are illustrated in the following numbered examples:

[0165] Example 1: A system comprising a tool, a catheter, an image sensor, a processor and a memory, wherein the catheter is sized to receive the tool, the image sensor is carried by the tool, and the memory has computer-readable instructions stored thereon, which, when executed by the processor, cause the system to capture a first image within the lumen of the catheter using the image sensor, the first image comprising a plurality of pixels; identify a first subset of the plurality of pixels comprising observable features and a second subset of the plurality of pixels comprising a background color; adjust the second subset of the plurality of pixels to a neutral color; remove the second subset by filtering the plurality of pixels to produce a modified image; determine the angular orientation of the observable features in the modified image; and determine a rotational offset of the tool relative to the catheter based on the angular orientation of the observable features.

[0166] Example 2: A system according to Example 1, wherein the distal portion of the tool has a square cross-section and the lumen has a complementary square cross-section, and wherein the rotational offset is 0 (zero) degrees, 90 degrees, 180 degrees, or 270 degrees.

[0167] Example 3: The system of Example 1, wherein the tool has a circular cross-section and the lumen has a complementary circular cross-section, and wherein the rotational offset is in the range of 0 degrees to 360 degrees.

[0168] Example 4: A system according to Example 1, wherein when executed by the processor, the computer-readable instructions further cause the system to capture an operation image during a procedure; rotate the operation image by the rotational offset; and display the rotated operation image on a display unit.

[0169] Example 5: A system according to Example 4, wherein when executed by the processor, the computer-readable instructions further cause the system to periodically repeat the capturing of the first image, the identifying, the adjusting, the determining of the angular orientation, and the determining of the rotational offset during the procedure.

[0170] Example 6: The system of Example 1, wherein the computer readable instructions, when executed by the processor, further cause the system to initiate control of movement of the distal portion of the catheter based on the rotational offset.

[0171] Example 7: The system of Example 1, wherein the computer-readable instructions, when executed by the processor, further cause the system to capture the first image, the first image comprising observable features indicative of longitudinal markings extending longitudinally along the inner wall of the catheter.

[0172] Example 8: The system of Example 1, wherein the computer-readable instructions, when executed by the processor, further cause the system to identify observable features including structures protruding from a distal portion of the catheter.

[0173] Example 9: The system of Example 1, wherein the computer-readable instructions, when executed by the processor, further cause the system to identify observable features of a fluid ejection pattern including fluid ejected from a distal portion of the catheter.

[0174] Example 10: The system of Example 1, wherein the computer-readable instructions, when executed by the processor, further cause the system to identify observable features of a light source comprising emitting light from a distal portion of the catheter.

[0175] Example 11: The system of Example 1 further comprises a structured light source that emits structured light from a distal portion of the catheter.

[0176] Example 12: The system of Example 11, wherein the structured light is colored illumination light.

[0177] Example 13: A system according to Example 11, wherein the observable feature is illuminated by the structured light, and wherein determining the angular orientation of the observable feature further includes comparing the first image to a plurality of model images, and determining the angular orientation is based on one or more matches with the model images.

[0178] Example 14: A system according to Example 1, wherein when executed by the processor, the computer-readable instructions further cause the system to illuminate the distal portion of the catheter using at least one of two optical light sources arranged around the distal portion of the catheter; and wherein identifying the observable feature includes identifying an object illuminated by at least one of the optical light sources.

[0179] Example 15: A system according to Example 14, wherein when executed by the processor, the computer-readable instructions further cause the system to illuminate the object using the two optical light sources; capture the first image of the object illuminated by the two optical light sources; illuminate the object using a first optical light source of the two optical light sources while keeping a second optical light source of the two optical light sources off; capture a second image of the object illuminated by the first optical light source of the two optical light sources; illuminate the object using the second optical light source of the two optical light sources while keeping the first optical light source of the two optical light sources off; and capture a third image of the object illuminated by the second optical light source of the two optical light sources.

[0180] Example 16: A system according to Example 15, wherein when executed by the processor, the computer-readable instructions further cause the system to compare the first image with the second image; compare the first image with the third image; and determine the rotational offset of the tool relative to the catheter based on comparing the first image with the second image, comparing the first image with the third image, or a combination thereof.

[0181] Example 17: A system according to Example 14, wherein when executed by the processor, the computer-readable instructions further cause the system to illuminate the object using the two optical light sources, the two optical light sources respectively having different related color temperatures; capture the first image including the object illuminated by the two optical light sources; analyze uneven shading of the object in the first image; and determine the rotational offset of the tool relative to the catheter based on the analysis of the uneven shading of the object in the first image.

[0182] Example 18: A system according to Example 1, wherein the first image is an image of an anatomical structure of a patient, and the second image is generated by one or more CT scans of the anatomical structure of the patient and is the image of the anatomical structure at an angular orientation of 0 (zero) degrees, wherein the determining the angular orientation further includes comparing the first image with the second image and determining the angular orientation of the first image based on the comparison.

[0183] Example 19: A system according to Example 1, wherein when executed by the processor, the computer-readable instructions further cause the system to rotate and scale the first image to produce a third image that matches the second image, thereby determining the rotational offset required to rotate the first image to match the second image.

[0184] Example 20: A method for determining the orientation of a tool at least partially installed within a catheter, the method comprising capturing a first image within the lumen of the catheter via an image sensor of the tool, the first image comprising a plurality of pixels; identifying a first subset of the plurality of pixels comprising observable features and a second subset of the plurality of pixels comprising a background color; adjusting the second subset of the plurality of pixels to a neutral color; removing the second subset by filtering the plurality of pixels to produce a modified image; determining an angular orientation of an observable feature in the modified image; and determining a rotational offset of the tool relative to the catheter based on the angular orientation of the observable feature.

[0185] Example 21: The method according to Example 20 further includes: capturing an operation image during a procedure; automatically rotating the operation image by the rotation offset; and displaying the rotated operation image on a display unit.

[0186] Example 22: According to the method of Example 21, the capturing of the first image, the identifying, the adjusting, the determining of the angular orientation and the determining of the rotational offset are periodically repeated during the procedure.

[0187] Example 23: The method according to Example 20 further includes initiating control of movement of the distal portion of the catheter based on the rotational offset.

[0188] Example 24: The method of Example 20 further comprises capturing the first image, the first image comprising observable features indicative of longitudinal markings extending longitudinally along the inner wall of the catheter.

[0189] Example 25: The method of Example 20, wherein identifying the observable feature comprises identifying a structure protruding from a distal portion of the catheter.

[0190] Example 26: The method of Example 20, wherein identifying the observable feature comprises identifying a fluid ejection pattern of fluid ejected from a distal portion of the catheter.

[0191] Example 27: The method of Example 20, wherein identifying the observable feature comprises identifying a light source emitting light from a distal portion of the catheter.

[0192] Example 28: The method of Example 20, wherein identifying the observable feature comprises identifying an object illuminated by a structured light source emitting structured light from a distal portion of the catheter.

[0193] Example 29: The method according to Example 28 further includes: capturing the first image of the object illuminated by the structured light, wherein the determining the angular orientation of the observable feature further includes comparing the first image with multiple model images, and determining the angular orientation is based on one or more matches with the model images.

[0194] Example 30: The method according to Example 20 further includes illuminating the distal portion of the catheter using at least one of two optical light sources arranged around the distal portion of the catheter; and wherein identifying the observable feature includes identifying an object illuminated by the at least one of the optical light sources.

[0195] Example 31: The method according to Example 30 further includes: illuminating the object using the two optical light sources; capturing the first image of the object illuminated by the two optical light sources; illuminating the object using the first optical light source of the two optical light sources while keeping the second optical light source of the two optical light sources turned off; capturing the second image of the object illuminated by the first optical light source of the two optical light sources; illuminating the object using the second optical light source of the two optical light sources while keeping the first optical light source of the two optical light sources turned off; and capturing a third image of the object illuminated by the second optical light source of the two optical light sources.

[0196] Example 32: The method according to Example 31 further includes: comparing the first image with the second image; comparing the first image with the third image; and determining the rotational offset of the tool relative to the catheter based on comparing the first image with the second image, comparing the first image with the third image, or a combination thereof.

[0197] Example 33: The method according to Example 30 further includes: illuminating the object using the two optical light sources, the two optical light sources respectively having different related color temperatures; capturing the first image including the object illuminated by the two optical light sources; analyzing uneven shadows of the object in the first image; and determining the rotational offset of the tool relative to the catheter based on analyzing the uneven shadows of the object in the first image.

[0198] Example 34: A method according to Example 20, wherein the first image is an image of an anatomical structure of a patient, and the second image is generated by one or more CT scans of the anatomical structure of the patient and is the image of the anatomical structure at an angular orientation of 0 (zero) degrees, wherein determining the angular orientation further includes comparing the first image with the second image, and determining the angular orientation of the first image based on the comparison.

[0199] Example 35: The method according to Example 20 further includes: rotating and scaling the first image to produce a third image that matches the second image, thereby determining the rotation offset required to rotate the first image to match the second image.

[0200] Example 36: A system comprising: a catheter comprising a lumen; a tool configured to be positioned within the lumen; an imaging sensor positioned within the tool; and an observable feature on the catheter within a field of view of the imaging sensor, wherein the imaging sensor is configured to capture multiple images of the field of view, the multiple images comprising a first image, the first image comprising the observable feature, and wherein an orientation of the observable feature in the first image indicates a rotational offset of the tool relative to the catheter.

[0201] Example 37: The system of Example 36 further comprises: a processor configured to collect the plurality of images from the imaging sensor during a procedure, rotationally adjust at least one of the plurality of images to remove the rotational offset, and transmit the adjusted at least one of the plurality of images to a display.

[0202] Example 38: The system of Example 37, wherein the processor is further configured to adjust steering of a control cable in the catheter based on the rotational offset.

[0203] Example 39: A system according to Example 36, wherein the distal portion of the tool has a square cross-section and the lumen has a complementary square cross-section, and wherein the rotational offset is 0 (zero) degrees, 90 degrees, 180 degrees, or 270 degrees.

[0204] Example 40: The system of Example 36, wherein the tool has a circular cross-section and the lumen has a complementary circular cross-section, and wherein the rotational offset is in the range of 0 (zero) degrees to 360 degrees.

[0205] Example 41: A system according to Example 36, wherein the observable feature includes a longitudinal marking extending longitudinally along the inner wall of the catheter.

[0206] Example 42: The system of Example 36, wherein the observable feature comprises a structure protruding from a distal portion of the catheter.

[0207] Example 43: A system according to Example 36, wherein the observable feature includes a fluid jet pattern of fluid ejected from the distal portion of the catheter.

[0208] Example 44: The system of Example 36, wherein the observable feature comprises a light source that emits light from a distal portion of the catheter.

[0209] Example 45: The system of Example 36 further comprises a structured light source configured to emit structured light from a distal portion of the catheter to illuminate the observable feature.

[0210] Example 46: The system according to Example 36 further includes a structured light source, which includes a first light source and a second light source arranged relative to each other around a distal portion of the catheter, wherein the first light source is configured to emit the first structured light at a different time than when the second light source emits the second structured light, wherein the first image is captured when the first light source emits the first structured light, and the second image is captured when the second light source emits the second structured light, and wherein the rotational offset is determined by the difference between the first image and the second image.

[0211] Example 47: A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors associated with a computer-assisted medical system device, are suitable for causing the one or more processors to perform a method comprising the following operations: receiving a first image from within the lumen of the catheter via an image sensor carried by the tool, the first image comprising a plurality of pixels; identifying a first subset of the plurality of pixels comprising observable features and a second subset of the plurality of pixels comprising a background color; adjusting the second subset of the plurality of pixels to a neutral color; generating a modified image by filtering the plurality of pixels to remove the second subset; determining an angular orientation of an observable feature in the modified image; and determining a rotational offset of the tool relative to the catheter based on the angular orientation of the observable feature.

[0212] Example 48: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising: capturing an operating image during a procedure; rotating the operating image by the rotational offset; and displaying the rotated operating image on a display unit.

[0213] Example 49: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising: periodically repeating the capturing of the first image, the identifying, the adjusting, the determining of the angular orientation, and the determining of the rotational offset during the procedure.

[0214] Example 50: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising: initiating control of movement of the distal portion of the catheter based on the rotational offset.

[0215] Example 51: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising: capturing the first image, the first image including observable features indicating longitudinal markings extending longitudinally along the inner wall of the catheter.

[0216] Example 52: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: identifying observable features of a structure protruding from a distal portion of the catheter.

[0217] Example 53: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: identifying observable features of a fluid ejection pattern including fluid ejected from a distal portion of the catheter.

[0218] Example 53: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: identifying an observable feature of a light source that emits light from a distal portion of the catheter.

[0219] Example 54: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising: illuminating the distal portion of the catheter using at least one of two optical light sources arranged around the distal portion of the catheter; and wherein identifying the observable feature includes identifying an object illuminated by the at least one of the optical light sources.

[0220] Example 55: A non-transitory machine-readable medium according to Example 54, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: illuminating the object using the two optical light sources; capturing the first image of the object illuminated by the two optical light sources; illuminating the object using a first optical light source of the two optical light sources while keeping a second optical light source of the two optical light sources off; capturing a second image of the object illuminated by the first optical light source of the two optical light sources; illuminating the object using the second optical light source of the two optical light sources while keeping the first optical light source of the two optical light sources off; and capturing a third image of the object illuminated by the second optical light source of the two optical light sources.

[0221] Example 56: A non-transitory machine-readable medium according to Example 55, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: comparing the first image with the second image; comparing the first image with the third image; and determining the rotational offset of the tool relative to the catheter based on comparing the first image with the second image, comparing the first image with the third image, or a combination thereof.

[0222] Example 57: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising the following operations: illuminating the object using the two optical light sources, the two optical light sources respectively having different associated color temperatures; capturing the first image including the object illuminated by the two optical light sources; analyzing uneven shading of the object in the first image; and determining the rotational offset of the tool relative to the catheter based on the analysis of the uneven shading of the object in the first image.

[0223] Example 58: A non-transitory machine-readable medium according to Example 47, wherein the first image is an image of an anatomical structure of a patient, and the second image is generated by one or more CT scans 4 of the anatomical structure of the patient and is the image of the anatomical structure at an angular orientation of 0 (zero) degrees, wherein the determining the angular orientation further includes comparing the first image with the second image and determining the angular orientation of the first image based on the comparison.

[0224] Example 59: A non-transitory machine-readable medium according to Example 47, wherein when executed by the one or more processors, the plurality of machine-readable instructions are suitable for causing the one or more processors to perform a method further comprising rotating and scaling the first image to produce a third image that matches the second image, thereby determining a rotational offset required to rotate the first image to match the second image.

Claims

1. A system for determining a rotational offset of a tool relative to a catheter, the catheter being dimensioned to receive the tool, the system comprising: said tools; the catheter; processor; a memory having computer readable instructions stored thereon, which, when executed by the processor, cause the system to: receiving, via an image sensor of the tool, a plurality of images, each image of the plurality of images comprising a plurality of pixels; for each image in the plurality of images, identifying a subset of the plurality of pixels associated with an observable feature; for each image in the plurality of images, determining an orientation of the observable feature based on the identified subset of the plurality of pixels; determining the rotational offset of the tool relative to the catheter based on a composite orientation of the observable features, the composite orientation being based on a combination of the orientations of the observable features for each of the plurality of images; and One or more images captured by the image sensor are adjusted based on the rotational offset.

2. The system of claim 1 , wherein the computer readable instructions, when executed by the processor, further cause the system to: One or more images are removed from the plurality of images based on the temporal values ​​of the one or more images. The system of claim 1 , wherein the orientation is a quadrant orientation.

4. The system of claim 1 , wherein the computer readable instructions, when executed by the processor, further cause the system to: For each image in the plurality of images, a confidence score for the determined orientation of the observable feature is determined.

5. The system of claim 4, wherein determining the composite orientation of the observable feature is further based on a combination of the confidence score and the orientation of the observable feature for each of the plurality of images.

6. The system of claim 4, wherein the confidence score is determined based on a quantity of graphical cells associated with the observable feature. 7 . The system of claim 1 , adjusting one or more images captured by the image sensor based on the rotational offset comprises aligning an angular position of the tool with an angular position of the catheter.

8. The system of claim 1, wherein the observable feature protrudes from a distal end of the catheter.

9. The system of claim 8, wherein the observable feature is integrally formed with the catheter.

10. The system of claim 8, wherein the tool comprises an illumination source configured to illuminate the observable feature.

Images