System and method for triple imaging hybrid probe

Abstract

A hybrid vision device is provided. The hybrid vision device includes an articulating elongate member including a proximal end and a distal end, and a position sensor located at the distal end of the articulating elongate member; and a multi-modal sensing probe detachably coupled to the articulating elongate member, and the multi-modal sensing probe includes an ultrasound transducer and a camera located at a distal portion of the multi-modal sensing probe.

Description

[0001] Reference

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 033,624, filed June 2, 2020, which is incorporated by reference herein. BACKGROUND

[0003] Early diagnosis of lung cancer is critical. The five-year survival rate for lung cancer is approximately 18%, significantly lower than the three most common cancers that follow: breast cancer (90%), colorectal cancer (65%), and prostate cancer (99%). In 2018, a total of 14.2 million people were recorded to have died from lung cancer.

[0004] When a patient has been diagnosed with a suspicious lung lesion, they can be referred to a physician for a biopsy of the lesion to determine if it is malignant. If the tissue sample is determined to be malignant, bronchoscopic endobronchial treatment of lung cancer can be performed. In a typical procedure, preoperative imaging such as computed tomography (CT) can be performed to identify a lesion in the patient’s lung. The CT images can be used to provide anatomical location information of the lesion, such as to generate a map to guide navigation of a bronchoscope during bronchoscopy. During bronchoscopy, a bronchoscopy system equipped with sensors such as electromagnetic (EM) three-dimensional (3D) sensors can match itself to the CT images or patient anatomy. The EM information, together with a direct visualization system (e.g., a camera), can allow the physician to maneuver the bronchoscope to reach the lesion site.

[0005] If the lesion resides at least partially within an airway that is captured in the field of view of the direct visualization system, the physician can navigate the endoscope toward the lesion and proceed to take a tissue biopsy of the lesion. However, a challenge arises when the lesion is outside of the airway, where the lesion is outside of the field of view of the direct visualization system of the bronchoscope. In this case, the physician does not see the lesion in the camera view, and the physician can have to rely on information from the EM sensors and preoperative images, which cannot provide accurate and real-time location of the lesion relative to the bronchoscope. SUMMARY

[0006] It is recognized herein that there is a need for a minimally invasive system that allows for performing surgical or diagnostic procedures with improved visualization. The present disclosure provides systems and methods that allow for early lung cancer diagnosis and treatment with improved real-time visualization. In particular, the present disclosure provides a bronchoscopy device with multi-modal hybrid vision. The bronchoscope can integrate electromagnetic (EM) sensors, direct imaging devices, and ultrasound imaging, thereby allowing a physician to visualize tissue changes within the lung, particularly outside of the airways. For example, incorporating an ultrasound probe into a bronchoscope can allow a physician to scan a region of interest and confirm the actual location of a lesion relative to the bronchoscope with improved accuracy. The provided bronchoscope can provide hybrid vision capabilities by incorporating the use of EM sensors, direct imaging sensors, and ultrasound sensors. It should be noted that the provided endoscopic systems can be used for various minimally invasive surgical, therapeutic, or diagnostic procedures involving various types of tissue including cardiac, bladder, and lung tissue, as well as other anatomical regions of a patient’s body such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urethra, or the respiratory system including but not limited to the bronchi, lungs, and various other organs.

[0007] In one aspect of the present disclosure, a hybrid vision device is provided. The hybrid vision device includes an articulating elongate member including a proximal end and a distal end, and a first position sensor located at the distal end of the articulating elongate member; and a multi-modal sensing probe detachably coupled to the articulating elongate member. The multi-modal sensing probe includes an ultrasound transducer and a camera located at a distal portion of the multi-modal sensing probe.

[0008] In some embodiments, the multi-modal sensing probe is inserted through a first lumen of the articulating elongate member. In some embodiments, the multi-modal sensing probe is rotatable and extendable relative to the articulating elongate member. In some embodiments, the multi-modal sensing probe is articulatable relative to the articulating elongate member.

[0009] In some embodiments, the multi-modal sensing probe further includes a second position sensor located at a distal portion of the multi-modal sensing probe to track a position of the distal portion of the multi-modal sensing probe. In some cases, the second position sensor, the camera, and the illumination device are embedded into the distal portion of the multi-modal sensing probe. In some cases, the second position sensor, the camera, and the illumination device are arranged in a compact configuration. In some cases, the articulating elongate member and the multi-modal sensing probe are robotically controlled based at least in part on sensor data captured by the first position sensor and the second position sensor.

[0010] In some embodiments, the ultrasound transducer is an array of linear endobronchial ultrasound (EBUS) transducers. In some embodiments, the camera provides a real-time forward view and the ultrasound transducer provides a real-time side view. In some cases, the articulating elongate member includes a second lumen for receiving an instrument. In some cases, movement of the instrument is captured by the real-time side view.

[0011] In some embodiments, the articulating elongate member further includes an imaging device at a distal end of the articulating elongate member. In some embodiments, the multi-modal sensing probe includes an inflatable tip to provide access to or seal against an interior of a body.

[0012] In another aspect, the present disclosure provides a hybrid vision device, comprising: an articulating elongate member including a distal tip portion and a curved segment; an ultrasound transducer at the distal tip portion to provide a side view; and a channel configured to receive an instrument, wherein the lumen has a port at the curved segment, thereby allowing the instrument to extend out the port along a trajectory that intersects the side view or the ultrasound image.

[0013] In some embodiments, the articulating elongate member includes a position sensor at the distal tip portion. In some cases, the position sensor is embedded into the distal tip portion to track a position of the distal tip portion. In some embodiments, the articulating elongate member includes a camera embedded into the distal tip portion. In some cases, the camera provides a real-time forward view. In some embodiments, the ultrasound transducer is an array of linear endobronchial ultrasound (EBUS) transducers.

[0014] Other aspects and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the following detailed description, which is only illustrative of specific embodiments of the disclosure. As will be realized, the disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

[0015] Incorporation by Reference

[0016] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent there is a contradiction between the disclosure herein and that of any such incorporated publication, patent, or patent application, the present specification shall control.

[0017] BRIEF DESCRIPTION OF DRAWINGS

[0018] The novel features of the application are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present application will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the application are utilized, and the accompanying drawings (also “figure” and “figures” herein), of which:

[0019] Figure 1 An example of an assembly of an endoscope system is shown in accordance with some embodiments of the present disclosure.

[0020] Figure 2 An example of a hybrid probe that can extend distally of a distal tip of a bronchoscope is shown in accordance with some embodiments of the present disclosure.

[0021] Figure 3 An example of a hybrid probe that rotates relative to a distal portion of a catheter is shown in accordance with some embodiments of the present disclosure.

[0022] Figure 4 An example of an endoscope with integrated hybrid vision is shown in accordance with some embodiments of the present disclosure.

[0023] Figure 5 An example of a hybrid probe with an inflatable tip is shown in accordance with some embodiments of the present disclosure.

[0024] Figure 6 An example of a hybrid probe with a balloon tip is shown in accordance with some embodiments of the present disclosure.

[0025] Figure 7 An example of a user interface showing an endobronchial ultrasound (EBUS) view, a direct video / camera view, and a preoperative image-based virtual airway model for real-time accurate lesion location tracking is shown.

[0026] Figure 8 An example of a robotic bronchoscope is shown in accordance with some embodiments of the present disclosure.

[0027] Figure 9 An example of an instrument drive mechanism that provides mechanical and electrical interfaces to a handle portion of a robotic bronchoscope is shown in accordance with some embodiments of the present disclosure.

[0028] Figure 10 An example of a distal portion of a hybrid probe with integrated imaging and illumination devices is shown.

[0029] Figure 11 An example of a compact configuration of electronic components located in a distal portion is shown. DETAILED DESCRIPTION

[0030] While various embodiments of the application have been shown and described herein, it will be apparent to those skilled in the art that many changes, modifications, and substitutions can be made to the embodiments without departing from the application. It is understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application.

[0031] While exemplary embodiments will be directed primarily to bronchoscopes, those skilled in the art will recognize that this is not intended to be limiting and that the devices described herein can be used in other therapeutic or diagnostic procedures, as well as other anatomical regions of the patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system, including but not limited to the bronchial tubes, lungs, and various other organs.

[0032] The embodiments disclosed herein can be combined in one or more of a variety of ways to provide improved diagnosis and treatment to a patient. For example, the disclosed embodiments can be combined with existing methods and devices to provide improved treatment, such as, for example, in conjunction with known methods of pulmonary diagnosis, surgery, and other tissue and organ surgeries. It will be appreciated that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and devices described herein, the figures and supporting text providing descriptions according to embodiments.

[0033] While the treatment planning and definition of the diagnosis or surgery described herein are presented in the context of pulmonary diagnosis or surgery, the methods and devices described herein can be used to treat any tissue of the body and any organ and vessel of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ears, nose, mouth, soft tissue (such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, spinal and neural tissue, cartilage), hard biological tissue (such as teeth, bone, etc.), as well as body lumens and passageways (such as sinuses, ureters, colon, esophagus, pulmonary passageways, blood vessels, and throat).

[0034] Whenever the term "at least," "greater than," or "greater than or equal to" precedes the first numerical value in a series of two or more numerical values, the term "at least," "greater than," or "greater than or equal to" applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0035] Whenever the term“not greater than,”“less than,” or“less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term“not greater than,”“less than,” or“less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0036] As used herein, a processor includes one or more processors, such as a single processor or multiple processors of a distributed processing system, for example. A controller or processor as described herein typically includes a tangible medium to store instructions for implementing process steps, and the processor can include one or more of a central processing unit, programmable array logic, gate array logic, or field programmable gate array, for example. In some cases, the one or more processors can be programmable processors (e.g., central processing units (CPUs), graphics processing units (GPUs), or microcontrollers), digital signal processors (DSPs), field programmable gate arrays (FPGAs), and / or one or more advanced RISC machine (ARM) processors. In some cases, the one or more processors can be operatively coupled to a non-transitory computer readable medium. The non-transitory computer readable medium can store logic, code, and / or program instructions executable by the one or more processor units for performing one or more steps. The non-transitory computer readable medium can include one or more memory units (e.g., removable media or external memory such as an SD card or random access memory (RAM)). For example, one or more methods or operations disclosed herein can be implemented in a hardware component or a combination of hardware and software, such as an ASIC, a special purpose computer, or a general purpose computer, for example.

[0037] As used herein, the terms distal and proximal can generally refer to a position from a device reference and can be opposite to an anatomical reference. For example, a distal position of a bronchoscope or catheter can correspond to a proximal position of an elongate member of a patient, and a proximal position of a bronchoscope or catheter can correspond to a distal position of an elongate member of a patient.

[0038] An endoscopic system as described herein includes an elongated portion or elongated member, such as a catheter. The terms "elongated member" and "catheter" are used interchangeably throughout the specification unless otherwise noted by context. The elongated member can be placed directly in a body lumen or body cavity. In some embodiments, the system can also include a support device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position, or control movement and / or operation of the elongated member. Alternatively or additionally, the support device can be a handheld device or other control device that can or can not include a robotic system. In some embodiments, the system can also include peripheral devices and subsystems, such as an imaging system, that will assist and / or facilitate navigation of the elongated member to a target site within a subject's body.

[0039] An endoscopic system of the present disclosure can combine multiple sensing modalities to provide enhanced visual capabilities. In some embodiments, a multi-modal sensing system can include at least position sensing (e.g., EM sensor system, optical shape sensor, accelerometer, gyroscopic sensor), direct vision (e.g., camera), and ultrasound imaging.

[0040] In some cases, an endoscopic system can implement a position sensing system, such as an electromagnetic (EM) sensor, fiber optic sensor, and / or other sensor, to register and display a medical instrument with preoperatively recorded surgical images, thereby localizing a distal portion of an endoscope relative to a patient's body or a global reference frame. The position sensor can be a component of an EM sensor system that includes one or more electrically conductive coils that can be subjected to an externally generated electromagnetic field. Each coil of the EM sensor system used to implement the position sensor system in turn generates an induced electrical signal having a characteristic that depends on the position and orientation of the coil relative to the externally generated electromagnetic field. In some cases, the EM sensor system used to implement the position sensing system can be configured and positioned to measure at least three degrees of freedom, e.g., three position coordinates X, Y, Z. Alternatively or additionally, the EM sensor system can also be configured and positioned to measure six degrees of freedom (e.g., three position coordinates X, Y, Z and three orientation angles indicative of pitch, yaw, and roll of a base point) or five degrees of freedom (e.g., three position coordinates X, Y, Z and two orientation angles indicative of pitch and yaw of a base point).

[0041] Direct vision can be provided by an imaging device, such as a camera. The camera can include imaging optics (e.g., lens elements), an image sensor (e.g., CMOS or CCD), and illumination (e.g., LED or fiber-based light). The imaging device can be located at the distal tip of the endoscope’s elongate member or catheter. In some cases, the direct vision system can include both an imaging device and an illumination device. In some embodiments, the imaging device can be a video camera. The imaging device can include optics and an image sensor for capturing image data. The image sensor can be configured to generate image data in response to wavelengths of light. Various image sensors, such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD), can be used to capture image data. The imaging device can be a low-cost camera. In some cases, the image sensor can be disposed on a circuit board. The circuit board can be an imaging printed circuit board (PCB). The PCB can include a plurality of electronic components for processing image signals. For example, the circuitry for a CCD sensor can include an A / D converter and an amplifier to amplify and convert analog signals provided by the CCD sensor, as well as circuitry to combine or serialize the data so that it can be transmitted in a minimum number of electrical conductors. Alternatively, the image sensor can be integrated with an amplifier and converter to convert the analog signals to digital signals, so that a circuit board can not be needed. In some cases, the output of the image sensor or circuit board can be image data (digital signals) that can be further processed by camera circuitry or a processor of the camera. In some cases, the image sensor can include an array of optical sensors. As described below, the imaging device can be located at the distal tip of the catheter, a separate hybrid probe assembled to the endoscope, or a combination of the two.

[0042] The provided endoscope system can use ultrasound to help guide the physician to a location outside the airway. For example, the user can use ultrasound to locate a lesion in real-time to guide the endoscope to a location that a computed tomography (CT) scan (preoperative imaging) shows the approximate location of an isolated lung nodule. In some embodiments, the ultrasound can be a linear endobronchial ultrasound (EBUS), also known as a convex probe EBUS, which can image to one side of the endoscope device. For example, a linear endobronchial ultrasound (EBUS) transducer or transducer array can be located at a distal portion of the endoscope, providing a view parallel to the endoscope shaft. In some cases, the ultrasound can be a radial probe EBUS, which can image 360° radially.

[0043] Multi-modal sensing systems can beneficially improve the visual capabilities of endoscopic devices. For example, EM sensors can provide GPS-like navigation information for a user to navigate to a target site and use a direct visualization camera to view anatomical landmarks and features to further confirm the location relative to preoperative images (e.g., CT scan 3D model). Once at the target site, if the lesion is not visible in the airway, ultrasound information can be used to identify the exact location of the lesion in situ and in real-time. Ultrasound imaging can be used to visualize the depth (e.g., a few millimeters or centimeters) in the tissue near the ultrasound transducer to determine the exact location of the lesion for biopsy or to deliver a treatment or therapy (e.g., pharmacological, mechanical, thermal therapy). In some cases, the ultrasound location can be registered into the 3D location information based on the EM sensors so that the lesion location identified by the ultrasound imaging can be used to automatically control the position of the catheter tip or instrument position.

[0044] In some embodiments, an ultrasound probe can be detachably incorporated into a bronchoscopy system to enhance visual guidance to the user. The ultrasound probe can include an ultrasound transducer located at a distal portion of the probe and can be movable along the length of the endoscope.

[0045] In some embodiments, a hybrid probe can be assembled to an existing bronchoscopy system. The hybrid probe can include at least a camera, a location sensor (e.g., EM sensor), and an ultrasound transducer to provide multi-modal sensing capabilities to the endoscope assembly. The bronchoscope can be an articulating device that is controlled to navigate a path under the guidance provided by the hybrid probe. In other embodiments, the hybrid probe can include a location sensor (e.g., EM sensor) and an ultrasound transducer, while the articulating device can include a camera and a location sensor (e.g., EM sensor).

[0046] Figure 1 An example of an assembly of an endoscopic system 100 is shown in accordance with some preferred embodiments of the present disclosure. The endoscopic system 100 can include an articulating bronchoscope 110 and a hybrid probe 120. The hybrid probe 120 can be detachably assembled to the bronchoscope, such as by insertion into a channel of the bronchoscope catheter.

[0047] The bronchoscope 110 can include appropriate means for deflecting the distal tip 111 of the scope with minimal deflection or friction against surrounding tissue to follow the path of the structure being examined. For example, a control cable or pull cable is carried within the endoscope body to connect the articulating segment adjacent the distal end 111 to a set of control mechanisms (e.g., handles) at the proximal end of the endoscope or robotic support system.

[0048] The robotic bronchoscope system 100 can be releasably coupled to an instrument drive mechanism. The instrument drive mechanism can be mounted to the arm of a robotic support system or any actuated support system. The instrument drive mechanism can provide mechanical and electrical interfaces to the robotic bronchoscope system 100. The mechanical interface allows the robotic bronchoscope system 100 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope 110 can be attached to the instrument drive mechanism via quick-release devices such as magnets and spring-loaded levels. In some cases, the robotic bronchoscope 110 can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0049] Figure 8 An example of a robotic bronchoscope system supported by a robotic support system is shown. In some cases, the handle portion can communicate electrically with the instrument drive mechanism (e.g., instrument drive mechanism 820) via an electrical interface (e.g., a printed circuit board), allowing image / video data and / or sensor data to be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface may include pins soldered to an electronic board such as a printed circuit board (PCB). For example, a socket connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This can advantageously allow the endoscope to be quickly inserted into the instrument drive mechanism or robotic support without the use of additional cables. This type of electrical interface can also be used as a mechanical interface, allowing mechanical and electrical coupling to be established when the handle portion is inserted into the instrument drive mechanism. Alternatively or additionally, the instrument drive mechanism may provide only a mechanical interface. The handle portion can communicate electrically with modular wireless communication devices or any other user equipment (e.g., portable / handheld devices or controllers) used for transmitting sensor data and / or receiving control signals.

[0050] like Figure 8 As shown, the robotic bronchoscope 820 may include a handle portion 813 and a flexible elongated member 811. In some embodiments, the flexible elongated member 811 may include a shaft, a maneuverable tip, and a maneuverable segment. The robotic bronchoscope 820 can be used with... Figure 1 The steerable catheter assembly described herein is identical. A robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be released from the instrument drive mechanism and can be discarded. The bronchoscope may have varying degrees of stiffness along its axis to improve functional operation.

[0051] A robotic bronchoscope can be releasably coupled to an instrument drive mechanism 820. The instrument drive mechanism 820 can be mounted to an arm of a robotic support system, or to any of the actuated support systems described elsewhere herein. The instrument drive mechanism can provide a mechanical and electrical interface to the robotic bronchoscope 820. The mechanical interface can allow the robotic bronchoscope 820 to be releasably coupled to the instrument drive mechanism. For example, a handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via a quick mount / release device, such as a magnet and spring-loaded level. In some cases, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0052] In some cases, a separate instrument drive mechanism can be used to control movement of the hybrid probe. For example, a proximal portion of the hybrid probe can be coupled to a second instrument drive mechanism to articulate a tip of the hybrid probe. Alternatively or additionally, rolling movement of the hybrid probe can be controlled by a controller that is operably coupled to a controller of the catheter. This can beneficially provide robotic control of both the hybrid probe and the catheter, allowing coordinated control of the hybrid probe and the catheter with minimal user input.

[0053] Figure 9 An example of an instrument drive mechanism 920 that provides a mechanical interface to a handle portion 913 of a robotic bronchoscope is shown. As shown in the example, the instrument drive mechanism 920 can include a set of motors that are actuated to rotationally drive a set of pull wires of a catheter. The handle portion 913 of the catheter assembly can be mounted on the instrument drive mechanism such that its pulley assembly is driven by the set of motors. The number of pulleys can vary depending on the pull wire configuration. In some cases, one, two, three, four, or more pull wires can be used to articulate the catheter.

[0054] The handle portion can be designed to allow for single use of the robotic bronchoscope at a reduced cost. For example, conventional manual and robotic bronchoscopes can have a cable at the proximal end of the bronchoscope handle. The cable typically includes illumination optical fibers, camera video cables, and other sensor optical fibers or cables, such as electromagnetic (EM) sensors or shape-sensing fibers. Such complex cables can be expensive, adding to the cost of the bronchoscope. The robotic bronchoscopes provided can have an optimized design such that simplified structures and components can be employed while maintaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can employ a cableless design while providing a mechanical / electrical interface to the catheter.

[0055] In some cases, the handle portion can be a housing or include components configured to process image data, provide power, or establish communication with other external devices. In some cases, the communication can be wireless communication. For example, the wireless communication can include Wi-Fi, radio communication, Bluetooth, IR communication, or other types of direct communication. Such wireless communication capabilities can allow the robotic bronchoscope to work in a plug-and-play manner and can be conveniently disposed of after a single use. In some cases, the handle portion can include circuitry elements, such as a power source for powering electronics (e.g., cameras and LED light sources) disposed within the robotic bronchoscope or catheter.

[0056] The handle portion can be designed in conjunction with the catheter such that cables or optical fibers can be removed. For example, the catheter portion can employ a design with a working channel that allows instruments to pass through the robotic bronchoscope, a vision channel that allows a hybrid probe to pass through, and low-cost electronics such as a tip-chip camera, a light source such as a light-emitting diode (LED), and an EM sensor positioned at an optimal location according to the mechanical structure of the catheter. This can simplify the design of the handle portion. For example, by using LEDs for illumination, the terminals at the handle portion can be based on electrical soldering or wire crimping only. For example, the handle portion can include a proximal circuit board where the camera cable, LED cable, and EM sensor cable are terminated, while the proximal circuit board is connected to the interface of the handle portion and establishes electrical connections to the instrument drive mechanism. As described above, the instrument drive mechanism is attached to the robotic arm (robotic support system) and provides a mechanical and electrical interface to the handle portion. This can advantageously improve assembly and implementation efficiency and simplify manufacturing processes and costs. In some cases, the handle portion and the catheter together can be disposed of after a single use.

[0057] Referring back to Figure 1 In some cases, the distal tip of the bronchoscope can include a position sensor such as EM sensor 113 for tracking the position of the distal tip of the bronchoscope relative to a global reference frame or patient anatomy.

[0058] The catheter of the bronchoscope can include a lumen sized to accommodate the hybrid probe and a lumen or working channel 130 to accommodate instruments. Various instruments can be inserted through the lumen, such as biopsy needles, graspers, scissors, baskets, snares, curettes, laser fibers, suturing tools, balloons, morcellators, various implants or stent delivery devices, etc. In some cases, the distal tip position tracked by the EM sensor 113 can be registered with the ultrasound imaging provided by the hybrid probe such that the position of the distal tip of the catheter and / or instrument carried in the working channel 130 can be determined relative to the lesion location identified by ultrasound. In some cases, the movement of the instrument can be captured in the linear EBUS view 122.

[0059] The hybrid probe 120 can include at least a position sensor such as an EM sensor 123, and an ultrasound transducer 121. The transducer 121 can include an array of one or more transducers to perform real-time ultrasound imaging. In some cases, the transducer can include a convex ultrasound transducer located at the tip of the probe and allow for linear scanning parallel to the insertion direction of the bronchoscope in order to assess structures around the central airway (e.g., linear EBUS view 122). The linear EBUS view is parallel to the insertion direction of the bronchoscope. This can be achieved by direct contact of the hybrid probe with the airway wall or via a water-filled distal balloon.

[0060] The hybrid probe can be connected to an ultrasound scanner with a color Doppler system to better distinguish between solid structures and vascular structures. In an example, the ultrasound transducer can include a 7.5 MHz linear curved array ultrasound transducer located at the distal portion of the probe to provide imaging in B-mode and / or color Doppler mode. The ultrasound imaging system can provide the ultrasound view 122 by scanning a field of view at an angle to the longitudinal axis or tilting the camera view forward by an angle. As non-limiting examples, the ultrasound imaging system can provide a scan range defined by a field of view ranging from 30° to 80°, a view direction ranging from 25° to 90° forward tilt, and a depth of field ranging from 2 mm to 80 mm.

[0061] The hybrid probe 120 can include a position sensor such as an EM sensor 123. The EM sensor can be positioned at the distal portion of the hybrid probe for tracking the real-time position of the distal position of the probe relative to a global reference frame. In some cases, the EM sensor can be optional, and the position of the distal tip of the hybrid probe can be obtained based on EM sensor data provided by the EM sensor 113 located at the bronchoscope and the relative position of the hybrid probe to the bronchoscope tip. The EM sensor 113 located at the distal tip of the catheter and the EM sensor 123 located at the hybrid probe can be used to robotically control the position / movement of the hybrid probe and the bronchoscope. For example, control commands can be generated based on position information (e.g., EM sensor data captured by the EM sensor 113 and the EM sensor 123) to coordinate the movement of the catheter 110 and the hybrid probe 120.

[0062] In some cases, the hybrid probe 120 can include an imaging device such as a camera 124. The camera 124 can be located at the tip of the hybrid probe to provide a forward-facing video / camera view 125 parallel to the longitudinal axis of the probe. Alternatively, the camera 124 can be integrated into the distal tip 111 of the bronchoscope.

[0063] In some embodiments, the hybrid probe 120 can include an illumination device 126 located at the distal end of the probe. The illumination device can include one or more light sources positioned at the distal tip of the hybrid probe 120. The illumination device can be located at the distal end of the hybrid probe. Alternatively or additionally, the illumination device can be located at the distal end of the articulating bronchoscope 110. The light sources can be light emitting diodes (LEDs), organic LEDs (OLEDs), quantum dots, or any other suitable light source. In some cases, the light sources can be miniaturized LEDs or dual color strobe LED illumination for compact design. In some cases, the illumination device can be a fiber optic based illumination, which can include a single fiber or a bundle of fibers coupled to an LED or a laser.

[0064] In some cases, each of the one or more LEDs can be connected to a power wire, which can extend to the proximal handle of the hybrid probe. In some embodiments, the LEDs can be soldered to separate power wires, which are then bundled together to form a single strand. In some embodiments, the LEDs can be soldered to a power supply pull wire. In other embodiments, the LEDs can be crimped or directly connected to a single pair of power wires. In some cases, a protective layer such as a thin layer of biocompatible glue can be applied to the front surface of the LEDs to provide protection while allowing light to be emitted out.

[0065] The imaging device 124, the illumination device, the EM sensor 123, and the ultrasound transducer can be integrated into the hybrid probe. For example, the distal portion of the hybrid probe can include appropriate structures that match at least one of the dimensions of the aforementioned electronics. In some cases, the distal tip of the hybrid probe can have dimensions such that one or more electronic components can be embedded into the distal tip. For example, the imaging device 124 can be embedded into a cavity at the distal tip of the hybrid probe. The cavity can be integrally formed with the distal portion of the hybrid probe and can have dimensions that match the length / width of the camera such that the camera can not move relative to the hybrid probe.

[0066] In some cases, the EM sensor 123 can be disposed at the distal portion of the hybrid probe and can be placed next to or behind the illumination light sources (e.g., LEDs) in a stereoscopic arrangement. Figure 10 An example of a distal portion of a hybrid probe with an integrated imaging device and illumination device is shown. It should be noted that the distal tip design of the hybrid probe also applies to hybrid probes with Figure 4The distal tip of the integrated vision catheter described in the background. The camera can be located at the distal portion. The distal tip can have a structure that houses the camera, illumination device, and / or position sensor. For example, the camera can be embedded in a cavity 1010 at the distal tip of the catheter. The cavity 1010 can be integrally formed with the distal portion of the cavity and can have dimensions that match the length / width of the camera so that the camera can not move relative to the catheter. In some cases, the distal portion can include structures 1030 that have dimensions that match the dimensions of the miniaturized LED light source. As shown in the illustrated example, two cavities 1030 can be integrally formed with the distal portion to house two LED light sources. Any number of light sources can be included. The internal structure of the distal portion can be designed to accommodate any number of light sources.

[0067] In some cases, each LED can be connected to a power wire that can extend to the proximal end of the hybrid probe or bronchoscope (e.g., Figure 4 The integrated vision implementation described in the background). In some implementations, the LEDs can be soldered to separate power wires that are then bundled together to form a single strand. In some implementations, the LEDs can be soldered to a power supply pull wire. In other implementations, the LEDs can be crimped or directly connected to a single pair of power wires. In some cases, a protective layer such as a thin layer of biocompatible glue can be applied to the front surface of the LEDs to provide protection while allowing light to be emitted. In some cases, an additional cover 1031 can be placed at the forward end face of the distal tip to provide precise positioning of the LEDs as well as sufficient space for adhesion. The cover 1031 can be composed of a transparent material that matches the refractive index of the glue so that the illumination light is not blocked.

[0068] An electromagnetic coil located at the distal end can be used with an electromagnetic tracking system to detect the position and orientation of the distal end of the hybrid probe when it is disposed within an anatomical system. In some implementations, the coil can be angled to provide sensitivity to electromagnetic fields along different axes, i.e., to be able to measure all 6 degrees of freedom: three positions and three angles. During navigation, such as when the hybrid probe is withdrawn within the catheter, an EM field generator positioned beside, below, or above the patient's torso can locate the EM sensor, thereby tracking the position of the catheter tip in real time. During surgery, such as when the hybrid probe is extended distally out of the catheter, the EM sensor can track the position of the hybrid probe tip in real time.

[0069] Referring back to Figure 1 , the hybrid probe provided can have a compact configuration of electronic components disposed at the distal portion. For example, one or more electronic components can be bundled to provide a compact design. Figure 11An example of a compact configuration of electronic components located in the distal portion is shown. In some cases, the illumination light source 1120 and one or more position sensors 1110 can be combined into a bundle. In some cases, the cable connecting to the ultrasound transducer 1130 can also be bundled with the cables of other electronics to further reduce the size of the hybrid probe.

[0070] Referring back to Figure 1 , power to the camera 124 can be provided by a wired cable. In some cases, the cable can be in the form of a wire bundle, providing power to the camera as well as illumination elements or other circuitry at the distal tip of the hybrid probe. The camera and / or light source can be powered from a power source disposed in the handle portion of the hybrid probe via an electrical wire, copper wire, or via any other suitable means throughout the length of the hybrid probe. In some cases, real-time images or video of the tissue or organ can be transmitted wirelessly to an external user interface or display. The wireless communication can be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, the images or video captured by the camera can be broadcast to multiple devices or systems. In some cases, the image and / or video data from the camera can be transmitted along the length of the hybrid probe to a processor located in the handle portion via an electrical wire, copper wire, or via any other suitable means. The image or video data can be transmitted to an external device / system via a wireless communication component in the handle portion. In some cases, the system can be designed such that the electrical wires are not visible to the operator or are not exposed to the operator.

[0071] As described above, the hybrid probe can be movable relative to the bronchoscope. The hybrid probe can have translational movement along the longitudinal axis of the lumen and rotational movement relative to the bronchoscope. For example, the hybrid probe can be extendable relative to the distal tip of the bronchoscope. For example, the hybrid probe can be slidable along the lumen of the catheter. When hybrid vision is needed, the tip of the hybrid probe can be extended out along the longitudinal axis of the distal tip of the bronchoscope. Alternatively or additionally, the hybrid probe can be articulatable independent of the bronchoscope. For example, the hybrid probe can have a bend segment that can be articulatable independent of the movement of the bronchoscope.

[0072] As described above, the imaging device located at the hybrid probe can provide a forward-looking view aligned with the forward direction of the hybrid probe. Providing the imaging device at the distal tip of the hybrid probe instead of on the catheter can advantageously allow for a clear near-field view of the tissue or organ without being obstructed by the operation of the hybrid probe or tools (e.g., needles). Furthermore, providing the imaging device on the hybrid probe allows for additional degrees of freedom for the camera view to decouple the camera field of view from the working space of the tools. The camera view provided by the hybrid probe can be controlled with improved flexibility (e.g., the pose or orientation of the imaging device can be controlled by controlling the articulation of the hybrid probe relative to the catheter) to better coordinate with the operation of the tools.

[0073] In some cases, an additional imaging device can be provided at the distal tip of the catheter 111. This advantageously allows the imaging devices located at the hybrid probe and catheter to provide different camera views simultaneously. For example, when the hybrid probe is articulated relative to the catheter, the camera view of the imaging device located at the hybrid probe can be different than the camera view of the imaging device located at the catheter.

[0074] In some cases, the catheter / bronchoscope and hybrid probe can be robotically controlled to coordinate the camera views of the imaging devices and the operation of the tools. For example, the catheter can be advanced toward a target site under robotic control of a robotic bronchoscopy system. Once it reaches the target site, the distal tip of the catheter can be locked to provide an anterior camera view, while the hybrid probe can be robotically controlled (e.g., extended, retracted, articulated, rotated / rolled relative to the catheter) to provide a desired ultrasound view of the lesion site and / or camera view of the needle. In some cases, the hybrid probe can automatically adjust the view based on the operation of the tool (e.g., needle).

[0075] Figure 2 An example 200 of a hybrid probe 213 that can be extended distally over the distal tip 211 of a bronchoscope is shown. The bronchoscope can be the same as the bronchoscope described in Figure 1 In some cases, during navigation to a target site that only requires direct vision (e.g., camera view), the hybrid probe can be flush with the distal end surface 217 of the catheter. In some cases, when ultrasound images are needed, the hybrid probe can be extended beyond the catheter tip to expose the ultrasound transducer 219. The hybrid probe and bronchoscope can include a reversible interlocking feature that locks the position of the hybrid probe relative to the bronchoscope to prevent axial and / or rotational movement. In some cases, when the hybrid probe is operating to provide ultrasound imaging, it can be rotatable relative to the bronchoscope to position the lesion site. For example, when the hybrid probe is extended to a desired length and / or rotated to a desired angle, the interlocking feature can lock the position / orientation of the hybrid probe relative to the bronchoscope.

[0076] Figure 3 An example of a hybrid probe 300 that is rotated relative to the distal portion 310 of the catheter is shown. The ultrasound transducer 301 can be rotated to any desired position to provide imaging around the tip of the probe. As shown in the example, the hybrid probe 300 is extended over the catheter tip to a desired extent, and the ultrasound transducer 301 can be rotated to different angular positions for imaging. In some cases, once the hybrid probe is positioned to a desired angular position and / or length position, an interlocking feature can lock the position of the hybrid probe relative to the bronchoscope.

[0077] The hybrid probe can be rotated independently of the catheter, such that any point around the distal portion of the hybrid probe can be visualized by rotating the hybrid probe. The rotational movement of the hybrid probe can be robotically controlled or manually controlled. The rotational operation can allow the user to reposition the linear ultrasound field of view over the entire circumference of the airway. For example, a lesion location can be identified with the aid of the EBUS and by rotating the hybrid probe, once the lesion location is identified, the catheter can be repositioned accordingly (e.g., insertion or rotational movement), such that the biopsy instrument can be introduced and pushed out of the working channel and directly into the lesion location.

[0078] In some embodiments, the ultrasound transducer and camera can be integrated into the bronchoscope as a single device. Figure 4 An example of an endoscope 400 with integrated hybrid vision is shown. As illustrated in the example, the distal tip of the endoscope can include an integrated forward-looking camera 405 and illumination source for direct visualization, and an integrated linear ultrasound probe 401 facing in a lateral direction. One or more position sensors, such as EM sensors 403, can also be integrated into the distal tip.

[0079] The imaging device, illumination device, EM sensor, and ultrasound transducer can be integrated into the catheter. For example, the distal portion of the catheter can include suitable structures that match at least one of the dimensions of the electronics described above. In some cases, the distal tip of the catheter can have dimensions such that one or more electronic components can be integrated onto the catheter. For example, the outer diameter of the distal tip can be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel can be approximately 2 mm, such that one or more electronic components can be embedded into the distal tip. However, it should be noted that the outer diameter can be in any range less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range depending on the tool size or specific application, based on different applications.

[0080] The EM sensor, camera, and ultrasound transducer can be the same as described elsewhere herein. For example, the camera can be a chip-on-tip camera, and the illumination source can be a light-emitting diode (LED), organic LED (OLED), quantum dot, or any other suitable light source. In some cases, the light source can be a miniaturized LED or dual-color flash LED illumination for a compact design. In some cases, the electronics described above can be embedded into the tip of the catheter.

[0081] The distal bronchoscope may include a working channel 413 with a side outlet / port 414. In some cases, the side outlet 414 may be located at the articulated portion 411 of the catheter. The catheter may include a shaft 410, an articulated (bent) section 411, and a maneuverable distal portion 412, wherein the articulated (bent) section 411 connects the maneuverable distal portion to the shaft 410. For example, the bent section 411 may be connected at a first end to a distal tip portion and at a second end to the shaft portion, wherein the bent section is articulated by one or more draw wires. In some cases, the bent section may be manufactured separately as a modular component and assembled onto the shaft. In some cases, the bent section may further incorporate minimalist features to reduce cost and improve reliability. For example, the bent section may incorporate a cut pattern that advantageously allows for a greater degree of tube deflection to achieve a desired tip displacement relative to the shaft. In some cases, the bent section may consist of a stainless steel strip. The bending segment can be formed from other suitable structures or materials to achieve a predetermined bending stiffness while maintaining the desired axial and torsional stiffness with low hinge forces. For example, the bending segment may include a braided structure for torsional stability.

[0082] Side port 414 is located at bend 411. The distal end of the instrument extends along a path that runs from a position within a slender, flexible shaft through a side outlet to an extension outside the catheter at an angle Θ relative to the longitudinal axis of the catheter tip. This allows the instrument to extend from side port 414 and fall within EBUS view 417 (for ultrasound capture), even without articulated instruments.

[0083] The working channel lumen may have a port located at a bend, and the lumen allows the instrument to extend out of the port along a trajectory intersecting the EBUS view. For example, as... Figure 4 As shown, the working channel exit trajectory 416 intersects with the field of view 417 of linear endobronchial ultrasound (EBUS) or an image captured by ultrasound. A biopsy device 415 (e.g., forceps, needle, brush) can extend along a path from a position inside the outer wall through a side outlet / port to a position outside the outer wall at an angle of at least 30 degrees relative to the longitudinal axis. The path of the biopsy device 415 can intersect with / with the EBUS field of view (e.g., ultrasound image) such that at least one tip portion of the instrument can be seen in the ultrasound image. In some cases, the path of the biopsy device can be calibrated to the position of an electromagnetic positioning (EM) sensor 403, which is positioned at the distal portion of an elongated flexible shaft 410 and displayed by a surgical instrument navigation system.

[0084] The steerable catheter can be rotated so that any point around the distal end of the catheter can be visualized by rotating the catheter. The rotational movement of the catheter can be robotically controlled. The rotational maneuver can allow the user to reposition the linear ultrasound field of view over the entire airway circumference. For example, a lesion location can be identified with the aid of an EBUS and by rotating the catheter, once the lesion location is identified, a biopsy instrument can be introduced and pushed out of the working channel, directly into the lesion location.

[0085] In some embodiments, the hybrid vision assembly can include a bronchoscope with integrated direct vision and position sensing, and a hybrid probe with a balloon. The distal end of the bronchoscope can include an integrated forward-looking camera and illumination for direct visualization, and an integrated EM sensor (e.g., at least 3 degrees of freedom) for generating 3D positioning information.

[0086] In some cases, the hybrid probe can include at least an ultrasound transducer and an EM sensor located at a distal portion of the hybrid probe, and be covered by one or more balloons. In some cases, the hybrid probe can include a guide wire with an expandable outer diameter feature at the tip where the EBUS transducer and EM sensor are located. Figure 5 An example of a hybrid probe with an inflatable tip 503 is shown. As Figure 5 shown, the working channel exit trajectory 508 intersects with a linear endobronchial ultrasound (EBUS) or linear EBUS view 509 of an ultrasound image. The hybrid probe can be inserted through the working channel of the catheter / bronchoscope 505 and extend over the catheter to aid in navigation of the air passageway 510 in the lung. In some cases, the tip of the hybrid probe can extend through the tip of the catheter into the desired airway 510, and in turn the catheter can be slid over the guide wire / probe to reach the desired location. Various suitable methods (e.g., inflatable balloon) can be used to achieve the inflatable tip. The balloon 503 can be positioned at or near the distal end of the probe and can cover the linear EBUS transducer 507 and EM sensor 501. The balloon can be connected to a balloon inflation source or pump through the working channel for inflation or deflation of the balloon.

[0087] In some cases, the balloon can also create a seal across one passageway at a sealing point before collapsing the passageways distal to the sealing point. By injecting air, saline, and / or some other gas or liquid, the balloon(s) can expand and / or collapse to create a temporary closure in the passageway. In some examples, the shape of the balloon when expanded is malleable, allowing the balloon to conform to the shape of the passageway to expand around the one or more sensors and tip of the hybrid probe.

[0088] The EM sensors and ultrasound transducers can be the same as described elsewhere herein. For example, the EM sensors located at the tip of the catheter 505 and covered by the balloon can be 6 degrees of freedom (DOF) EM sensors.

[0089] Figure 6 An example of a hybrid probe 603 with a balloon tip is shown. For example, the hybrid probe 603 can provide an ultrasound view of a lesion site and 3D positioning information from the EM sensors. The balloon 601 can cover the EM sensors and ultrasound transducers 605. A hybrid probe 607 with a collapsed balloon is shown. In some cases, the EM sensor information can be registered and correlated with the ultrasound images. The hybrid probe can be detachably coupled to the catheter. The hybrid probe can be movable relative to the catheter, such as can be inserted and retracted along the longitudinal axis of the catheter and rotated in order to capture various ultrasound views and EM data points. Once the lesion site is identified and the 3D position of the lesion is determined, the physician can manipulate the position of the bronchoscope in order to align the trajectory of the biopsy tool with the position of the lesion. The position or trajectory of the biopsy tool can be determined based at least in part on the position data provided by the EM sensors located at the distal tip of the catheter and the known model of the catheter and / or instrument.

[0090] The catheter, bronchoscope, and / or hybrid probe can be robotically controlled. For example, the catheter can be advanced toward the target site under robotic control of a robotic bronchoscopy system. The catheter can be manipulated or advanced toward the target site in a manual manner, autonomously, or semi-autonomously. In another example, the movement of the hybrid probe can be automatically controlled such that the insertion, retraction, rotational movement for taking ultrasound images can be automatically controlled. Alternatively, the bronchoscope and / or hybrid probe can be controlled via a handheld device.

[0091] User Interface

[0092] In some implementations, real-time imaging of a target site (e.g., including a lesion) provided by the multi-modal sensing system can be displayed on a user interface. Figure 7 An example of a user interface showing an EBUS view, a direct video / camera view, and a virtual airway model based on preoperative images (e.g., preoperative CT images) demonstrating real-time accurate lesion location tracking is shown.

[0093] In some cases, a real-time lesion location can be identified from the ultrasound view. The lesion location can be registered with the coordinate system of the bronchoscopy system using the EM sensor data. In some embodiments, the location of the lesion can be segmented in the image data captured by the linear EBUS probe with the aid of a signal processing unit. In some cases, one or more processors of the signal processing unit can be configured to overlay the treatment location (e.g., lesion) on the real-time ultrasound images / video. In some cases, both the segmented lesion image and the optimal path for navigation of the elongated member to the lesion can be overlaid on the real-time ultrasound images. This can allow the operator or user to visualize the accurate location of the lesion and the planned path of the bronchoscope movement.

[0094] In some embodiments, the bronchoscopy system can include a navigation and localization subsystem configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images). The navigation and localization subsystem can be configured to identify an approximate segmented lesion location in the 3D rendered airway model, and based on the location of the lesion, the navigation and localization subsystem can generate an optimal path from the main bronchus to the lesion with a recommended approach angle towards the lesion to perform a surgical procedure (e.g., biopsy). For example, the processing unit can be configured to generate an augmentation layer that includes augmentation information such as the location of the treatment site or lesion. In some cases, the augmentation layer can also include graphical markers that indicate the path to the target site. The augmentation layer can be a substantially transparent image layer that includes one or more graphical elements (e.g., boxes, arrows, etc.). The augmentation layer can be superimposed onto the optical view of the optical images or video stream captured by the fluoroscopy (tomosynthesis) imaging system and / or displayed on the display device. The transparency of the augmentation layer allows the user to view the optical images through the graphical elements overlaid thereon. In some cases, both the segmented lesion image and the optimal path for navigation of the elongated member to the lesion can be overlaid on the virtual airway model or preoperative images. This can allow the operator or user to visualize the approximate location of the lesion and the planned path of the bronchoscope movement. In some cases, the segmented and reconstructed images provided prior to operation of the systems described herein (e.g., CT images as described elsewhere) can be overlaid on the real-time images.

[0095] In a registration step prior to driving the bronchoscope to the target site, the system can align the presented virtual view of the airway with the patient airway. Image registration can include a single registration step or a combination of a single registration step and real-time sensing updates to the registration information. Once registered, all airways can be aligned with the preoperatively presented airway. During driving of the robotic bronchoscope toward the target site, the position of the bronchoscope within the airway can be tracked and displayed. In some cases, a position sensor can be used to track the position of the bronchoscope relative to the airway. Other types of sensors (e.g., cameras) can also be used instead of or in combination with the position sensor using sensor fusion techniques. A position sensor such as an electromagnetic (EM) sensor can be embedded at the distal tip of the catheter, and an EM field generator can be positioned next to the patient torso during the procedure. The EM field generator can locate the EM sensor position in 3D space, or can locate the EM sensor position and orientation in 5D or 6D space. This can provide visual guidance to the operator when driving the bronchoscope toward the target site.

[0096] During the procedure, the lesion position can be updated and tracked in real-time based on the ultrasound view. In some cases, the position of the lesion can be marked on the ultrasound view, and the distal tip of the bronchoscope can be articulated, rotated, or moved to align the trajectory of the instrument with the lesion position.

[0097] In some embodiments, the processor can use the real-time position of one or more of the lesion, bronchoscope, hybrid probe, and instrument to calculate one or more possible geometries / configurations as illustrated in Figure 4 and Figure 5 For example, the system can calculate the appropriate motion of the hybrid probe in order to establish the desired configuration (e.g., the Figure 5In some embodiments, the system can track the position of the bronchoscope, the hybrid probe, and the instrument (e.g., based on the EM tracking system, the ultrasound system, and the instrument tracking system). The system can use this information to determine the optimal path to the lesion (e.g., based on the trajectory in the ultrasound image, the angle of the instrument, the location of the EBUS view) so that, in the case of direct ultrasound imaging, the instrument can access the lesion. In addition, in some embodiments, the system can partially or fully control the motion of the hybrid probe, the bronchoscope, and the instrument in order to establish the optimal geometry / configuration. For example, once the system knows and is able to track the position of all of these elements, it can generate a plan that is executed by the robotic system to move the instrument, the hybrid probe, and the bronchoscope into a desired configuration relative to the lesion site. In some cases, guidance for operating one or more of the instrument, the hybrid probe, and the bronchoscope can be displayed to the user. For example, the user can be provided with guidance such as “stop the bronchoscope here,” “move the hybrid probe to this position,” “insert the instrument now,” etc. to reach the desired configuration. In some cases, the configuration can be achieved in a semi-autonomous manner. For example, the user can be allowed to activate / trigger motion or stop motion while the robotic system automatically controls the motion of the hybrid probe, the instrument, and / or the bronchoscope (e.g., the user can hold an activation trigger and the robotic system moves the hybrid probe, the instrument, and / or the bronchoscope while the trigger is held). In some cases, the robotic system can stabilize the desired geometry / configuration by automatically tracking the patient’s respiratory motion (e.g., respiratory motion compensation).

[0098] Figure 7 An example of a user interface for visualizing real-time ultrasound images 705, a virtual airway model 709 with EM tracked catheter tip location 701, and a direct camera view 711 is shown. In some cases, the virtual airway 709 is overlaid with the optimal path 703, the catheter tip location 701, and the approximate location of the isolated lung nodule (shown by CT scan). In this example, the location of the tip of the catheter is displayed in real-time relative to the virtual airway model 709, providing visual guidance. As Figure 7 As shown in the example of FIG. 7, during robotic bronchoscope driving, the optimal path 703 can be displayed and overlaid on the virtual airway model. The virtual airway model can be constructed based on real-time fluoroscopy images / video (and position data of the imaging system). The user can also be presented with the camera view or images / video 711 captured by the bronchoscope and the ultrasound view 705 displaying the accurate lesion location in real-time.

[0099] While the preferred embodiments of the application have been illustrated and described herein, it will be appreciated that various changes can be made to the embodiments by those skilled in the art without departing from the scope of the application. It should be understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application. The following claims are intended to cover all such alternatives and equivalents.

Claims

1. A hybrid vision device comprising: a hinged elongated member comprising a proximal end and a distal end, wherein the hinged elongated member comprises a first lumen to receive a multi-modal sensing probe and a second lumen to receive an instrument; and the multi-modal sensing probe detachably coupled to the hinged elongated member through the first lumen and extendable out of a port of the first lumen at the distal end of the hinged elongated member, wherein the multi-modal sensing probe comprises an ultrasound transducer and a first position sensor at a distal portion of the multi-modal sensing probe, wherein a camera is located at a distal end of the hinged elongated member, and wherein the camera provides a camera view and the ultrasound transducer provides a side view that is tilted to the camera view, thereby capturing motion of the instrument when the instrument is extended out of a port of the second lumen.

2. The hybrid vision device of claim 1, wherein the multi-modal sensing probe is inserted through the first lumen of the hinged elongated member.

3. The hybrid vision device of claim 1, wherein the multi-modal sensing probe is rotatable relative to the hinged elongated member.

4. The hybrid vision device of claim 1, wherein the multi-modal sensing probe is hingable relative to the hinged elongated member.

5. The hybrid vision device of claim 1, wherein the hinged elongated member further comprises a second position sensor at the distal end of the hinged elongated member to track a position of the distal end of the hinged elongated member.

6. The hybrid vision device of claim 5, wherein the second position sensor, the camera, and an illumination device are embedded into the distal end of the hinged elongated member.

7. The hybrid vision device of claim 1, wherein the hinged elongated member is releasably mounted to a robotic support system by an instrument drive mechanism.

8. The hybrid vision device of claim 5, wherein the hinged elongated member and the multi-modal sensing probe are robotically controlled based at least in part on sensor data captured by the first position sensor and the second position sensor.

9. The hybrid vision device of claim 1, wherein the ultrasound transducer is an array of linear endobronchial ultrasound transducers.

10. The hybrid vision device of claim 1, wherein the side view is a linear scan parallel to an insertion direction of the hybrid vision device.

11. The hybrid vision device of claim 10, wherein a direction of the side view is in a range of 25° to 90° tilted forward relative to a camera view.

12. The hybrid vision device of claim 11, wherein the port of the second lumen is located at a bend segment of the hinged elongated member, allowing the instrument to extend out of the port along a trajectory that intersects the side view.

13. The hybrid vision device of claim 9, wherein the linear endobronchial ultrasound transducers comprise a 7.5 MHz linear curved array ultrasound transducer.

14. The hybrid vision device of claim 1, wherein the multi-modal sensing probe comprises an inflatable tip. ​

Images