System and method for triple imaging hybrid probe

Abstract

The present disclosure provides systems and methods for a triple imaging hybrid probe. A hybrid vision device is provided. The hybrid vision device comprises: an articulated elongate member comprising a proximal end and a distal end, and a position sensor located at the distal end of the articulated elongate member; and a multi-mode sensing probe removably coupled to the articulated elongate member, and the multi-mode sensing probe includes an ultrasound transducer and a camera at a distal portion of the multi-mode sensing probe.

Description

[0001] This application is a divisional application of Chinese patent application No. 202180057601.4, filed on May 28, 2021, entitled "System and Method for Triple Imaging Hybrid Probe" (the corresponding PCT application was filed on May 28, 2021, with application number (PCT / US2021 / 034856)).

[0002] Quote This application claims the benefit of U.S. Provisional Application No. 63 / 033,624, filed June 2, 2020, which is incorporated herein by reference. Background Technology

[0003] Early diagnosis of lung cancer is crucial. The five-year survival rate for lung cancer is approximately 18%, significantly lower than the three most common cancers: breast cancer (90%), colorectal cancer (65%), and prostate cancer (99%). In 2018, 142,000 people were recorded to have died from lung cancer.

[0004] When a patient has been diagnosed with a suspected lung lesion, they can be referred to a physician for a biopsy to determine if it is malignant. If the tissue sample is determined to be malignant, endobronchial treatment for lung cancer can be performed. In a typical procedure, preoperative imaging such as computed tomography (CT) scans can be performed to identify lesions in the patient's lungs. CT images can be used to provide information on the anatomical location of the lesion, such as to generate maps to guide the navigation of the bronchoscope during bronchoscopy. During bronchoscopy, a bronchoscopy system equipped with sensors such as electromagnetic (EM) three-dimensional (3D) sensors can match itself to CT images or the patient's anatomy. EM information, along with direct visualization systems (e.g., cameras), allows the physician to manipulate the bronchoscope to reach the lesion site.

[0005] If the lesion is at least partially located within the airway captured in the field of view of a direct visualization system, the physician can navigate the endoscope toward the lesion and proceed to obtain a tissue biopsy. However, challenges arise when the lesion is outside the airway, meaning it is outside the field of view of the bronchoscope's direct visualization system. In this situation, the physician cannot see the lesion in the camera view, and may have to rely on information from EM sensors and preoperative images that do not provide an accurate and real-time location of the lesion relative to the bronchoscope. Summary of the Invention

[0006] This document recognizes the need for a minimally invasive system that allows for the performance of surgical or diagnostic procedures with improved visualization. This disclosure provides systems and methods that enable early diagnosis and treatment of lung cancer using improved real-time visualization. Specifically, this disclosure provides a bronchoscopic device with multimodal hybrid vision. The bronchoscope can integrate electromagnetic (EM) sensors, direct imaging devices, and ultrasound imaging, allowing physicians to visualize tissue changes within the lungs, particularly those outside the airways. For example, incorporating an ultrasound probe into the bronchoscope allows physicians to scan regions of interest and confirm the actual location of lesions relative to the bronchoscope with improved accuracy. The provided bronchoscope provides hybrid vision capabilities through the combined use of EM sensors, direct imaging sensors, and ultrasound sensors. It should be noted that the provided endoscopic system can be used for a variety of minimally invasive surgical, therapeutic, or diagnostic procedures involving various types of tissue, including the heart, bladder, and lung tissue, as well as other anatomical regions of the patient's body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urethra, or the respiratory system, including but not limited to the bronchi, lungs, and various other organs.

[0007] In one aspect of this disclosure, a hybrid vision device is provided. The hybrid vision device includes: an articulating elongate member comprising a proximal end and a distal end, with a first position sensor located at the distal end of the articulating elongate member; and a multi-mode sensing probe detachably coupled to the articulating elongate member. The multi-mode sensing probe includes an ultrasonic transducer located on the multi-mode sensing probe and a camera on its distal portion.

[0008] In some embodiments, the multi-mode sensing probe is inserted through a first lumen of the articulated elongated member. In some embodiments, the multi-mode sensing probe is rotatable and extendable relative to the articulated elongated member. In some embodiments, the multi-mode sensing probe is articulated relative to the articulated elongated member.

[0009] In some embodiments, the multi-mode sensing probe further includes a second position sensor located at a distal portion of the multi-mode sensing probe to track the position of the distal portion. In some cases, the second position sensor, camera, and illumination device are embedded in the distal portion of the multi-mode sensing probe. In some cases, the second position sensor, camera, and illumination device are arranged in a compact configuration. In some cases, the articulated elongated member and the multi-mode sensing probe are robotically controlled at least in part based on sensor data captured by the first and second position sensors.

[0010] In some embodiments, the ultrasound transducer is an array of linear endobronchial ultrasound (EBUS) transducers. In some embodiments, a camera provides a real-time forward view, and the ultrasound transducer provides a real-time side view. In some cases, an articulated elongated member includes a second lumen for receiving the instrument. In some cases, movement of the instrument is captured via a real-time side view.

[0011] In some embodiments, the articulated elongated member also includes an imaging device located at the distal end of the articulated elongated member. In some embodiments, the multi-mode sensing probe includes an expandable tip to provide access to or seal the interior of the body.

[0012] In another aspect, this disclosure provides a hybrid vision device comprising: an articulated elongated member including a distal tip portion and a curved section; an ultrasonic transducer located at the distal tip portion to provide a side view; and a channel configured to receive an instrument, wherein the lumen has a port located at the curved section, thereby allowing the instrument to extend out of the port along a trajectory intersecting the side view or the ultrasonic image.

[0013] In some embodiments, the articulated elongated member includes a position sensor located at the distal tip portion. In some cases, the position sensor is embedded in the distal tip portion to track the position of the distal tip portion. In some embodiments, the articulated elongated member includes a camera embedded in the distal tip portion. In some cases, the camera provides a real-time forward view. In some embodiments, the ultrasonic transducer is an array of linear endobronchial ultrasound (EBUS) transducers.

[0014] Other aspects and advantages of this disclosure will be readily apparent to those skilled in the art from the following detailed description, in which only illustrative embodiments of this disclosure are shown and described. As will be appreciated, this disclosure is capable of other and different embodiments, and several details thereof can be modified in various obvious ways, all without departing from this disclosure. Therefore, the drawings and descriptions should be considered illustrative in nature and not restrictive.

[0015] Incorporation All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent that each individual publication, patent, or patent application is expressly and individually indicated to be incorporated by reference. If any publication or patent or patent application incorporated by reference contradicts the disclosure contained in this specification, this specification is intended to supersede and / or give precedence to any such contradictory material. Brief description of the attached figures The novel features of the invention are set forth in the appended claims. A better understanding of the features and advantages of the invention will be obtained by referring to the following detailed description of illustrative embodiments in which the principles of the invention are utilized, along with the accompanying drawings (also referred to herein as “figures” and “diagrams”), wherein: Figure 1 Examples of assemblies of endoscope systems according to some embodiments of the present disclosure are shown.

[0016] Figure 2 Examples of hybrid probes that can extend distally from the distal tip of a bronchoscope, according to some embodiments of the present disclosure, are shown.

[0017] Figure 3 An example of a hybrid probe that rotates relative to the distal portion of a catheter, according to some embodiments of this disclosure, is shown.

[0018] Figure 4 Examples of endoscopes with integrated hybrid vision according to some embodiments of the present disclosure are shown.

[0019] Figure 5 Examples of hybrid probes with expandable tips according to some embodiments of this disclosure are shown.

[0020] Figure 6 Examples of hybrid probes with balloon tips according to some embodiments of this disclosure are shown.

[0021] Figure 7 Examples of user interfaces are shown, displaying endobronchial ultrasound (EBUS) views, direct video / camera views, and virtual airway models based on preoperative images for real-time, accurate lesion location tracking.

[0022] Figure 8 Examples of robotic bronchoscopes according to some embodiments of the present invention are shown.

[0023] Figure 9 An example of an instrument drive mechanism that provides mechanical and electrical interfaces to the handle portion of a robotic bronchoscope according to some embodiments of the present invention is shown.

[0024] Figure 10 An example of the distal portion of a hybrid probe with integrated imaging and illumination devices is shown.

[0025] Figure 11 An example of a compact configuration of electronic components located on the far side is shown. Detailed Implementation

[0026] While various embodiments of the invention have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. Many variations, modifications, and substitutions will occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0027] While exemplary embodiments will be primarily directed toward bronchoscopy, those skilled in the art will recognize that this is not intended to be limiting, and the devices described herein may be used for other therapeutic or diagnostic procedures, as well as other anatomical areas of a patient’s body, such as the digestive system, including but not limited to the esophagus, liver, stomach, colon, urinary tract, or the respiratory system, including but not limited to the bronchi, lungs, and various other organs.

[0028] The embodiments disclosed herein can be combined in one or more of a variety of ways to provide improved diagnosis and treatment to patients. For example, the disclosed embodiments can be combined with existing methods and apparatus to provide improved treatment, such as, for example, with known methods for lung diagnosis, surgery, and surgery on other tissues and organs. It should be understood that any one or more of the structures and steps described herein can be combined with any one or more additional structures and steps of the methods and apparatus described herein, as described in the accompanying drawings and supporting text according to the embodiments.

[0029] Although the treatment plans and definitions for diagnosis or surgery described herein are presented in the context of pulmonary diagnosis or surgery, the methods and devices described herein can be used to treat any tissue of the body and any organ and blood vessel of the body, such as the brain, heart, lungs, intestines, eyes, skin, kidneys, liver, pancreas, stomach, uterus, ovaries, testes, bladder, ears, nose, mouth, soft tissues (such as bone marrow, adipose tissue, muscles, glands and mucous membranes, spinal and nerve tissues, cartilage), hard biological tissues (such as teeth, bones, etc.), and body lumens and passages (such as sinuses, ureters, colon, esophagus, pulmonary passages, blood vessels, and pharynx).

[0030] Whenever the terms "at least," "greater than," or "greater than or equal to" precede the first value in a series of two or more values, the terms "at least," "greater than," or "greater than or equal to" apply to each value in the series. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0031] Whenever the terms “not greater than,” “less than,” or “less than or equal to” precede the first value in a series of two or more values, the terms “not greater than,” “less than,” or “less than or equal to” apply to each value in the series. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0032] As used herein, for example, a processor includes one or more processors, such as a single processor or multiple processors in a distributed processing system. A controller or processor as described herein typically includes tangible media for storing instructions for implementing process steps, and a processor may include one or more of, for example, a central processing unit, programmable array logic, gate array logic, or field-programmable gate array. In some cases, one or more processors may be programmable processors (e.g., a central processing unit (CPU), graphics processing unit (GPU), or microcontroller), digital signal processor (DSP), field-programmable gate array (FPGA), and / or one or more advanced RISC machine (ARM) processors. In some cases, one or more processors may be operatively coupled to a non-transitory computer-readable medium. The non-transitory computer-readable medium may store logic, code, and / or program instructions executable by one or more processor units for performing one or more steps. The non-transitory computer-readable medium may include one or more memory units (e.g., removable media or external memory such as an SD card or random access memory (RAM)). For example, one or more methods or operations disclosed herein may be implemented in hardware components or a combination of hardware and software (e.g., for example, an ASIC, a special-purpose computer, or a general-purpose computer).

[0033] As used herein, the terms distal and proximal can generally refer to the location from a device reference and may be the opposite of an anatomical reference. For example, the distal location of a bronchoscope or catheter may correspond to the proximal location of a patient's slender member, and the proximal location of a bronchoscope or catheter may correspond to the distal location of a patient's slender member.

[0034] Endoscopic systems as described herein include elongated portions or components, such as catheters. Unless the context otherwise requires, the terms "elongated component" and "catheter" are used interchangeably throughout the specification. The elongated component may be placed directly within a body lumen or cavity. In some embodiments, the system may also include a support device, such as a robotic manipulator (e.g., a robotic arm), to drive, support, position, or control the movement and / or manipulation of the elongated component. Alternatively or additionally, the support device may be a handheld device or other control devices that may or may not include a robotic system. In some embodiments, the system may also include peripheral devices and subsystems (such as an imaging system) that assist and / or facilitate the navigation of the elongated component to a target site within the body.

[0035] The endoscopic system disclosed herein can combine multiple sensing modes to provide enhanced visual capabilities. In some embodiments, the multi-mode sensing system may include at least position sensing (e.g., EM sensor system, optical shape sensor, accelerometer, gyroscope sensor), direct vision (e.g., camera), and ultrasound imaging.

[0036] In some cases, endoscopic systems can incorporate position sensing systems (such as electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors) to register and display the medical device along with pre-recorded surgical images, thereby locating the distal portion of the endoscope relative to the patient's body or a global reference frame. The position sensor can be a component of an EM sensor system comprising one or more conductive coils that may be subjected to externally generated electromagnetic fields. Each coil of the EM sensor system used to implement the position sensor system then generates an induced electrical signal having characteristics dependent on the coil's position and orientation relative to the externally generated electromagnetic field. In some cases, the EM sensor system used to implement the position sensing system can be configured and positioned to measure at least three degrees of freedom, such as three position coordinates X, Y, and Z. Alternatively or additionally, the EM sensor system can also be configured and positioned to measure six degrees of freedom (e.g., three position coordinates X, Y, and Z and three orientation angles for pitch, yaw, and roll of an indicator base point) or five degrees of freedom (e.g., three position coordinates X, Y, and Z and two orientation angles for pitch and yaw of an indicator base point).

[0037] Direct vision can be provided by an imaging device, such as a camera. A camera may include imaging optics (e.g., lens elements), an image sensor (e.g., CMOS or CCD), and illumination (e.g., LED or fiber-optic light). The imaging device may be located at the distal tip of an elongated member or conduit of an endoscope. In some cases, a direct vision system may include both an imaging device and an illumination device. In some embodiments, the imaging device may be a camera. The imaging device may include optics for capturing image data and an image sensor. The image sensor may be configured to generate image data in response to the wavelength of light. Various image sensors, such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) devices, may be used to capture image data. The imaging device may be a low-cost camera. In some cases, the image sensor may be mounted on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may include multiple electronic components for processing image signals. For example, circuitry for a CCD sensor may include an A / D converter and amplifier to amplify and convert the analog signal provided by the CCD sensor, as well as circuitry for combining or serializing data, thereby enabling data transmission in a minimal number of electrical conductors. Optionally, the image sensor can be integrated with an amplifier and converter to convert analog signals into digital signals, potentially eliminating the need for a circuit board. In some cases, the output of the image sensor or circuit board can be image data (digital signals), which can be further processed by camera circuitry or the camera's processor. In some cases, the image sensor may include an array of optical sensors. As described below, the imaging device can be located at the distal tip of a catheter, a separate hybrid probe assembled to an endoscope, or a combination of both.

[0038] The provided endoscopic system can use ultrasound to help guide the physician to locations outside the airway. For example, the user can use ultrasound to locate lesions in real time to guide the endoscope to a location where a computed tomography (CT) scan (preoperative imaging) shows the approximate location of an isolated lung nodule. In some implementations, the ultrasound may be linear endobronchial ultrasound (EBUS), also known as convex probe EBUS, which can image onto one side of the endoscopic device. For example, a linear endobronchial ultrasound (EBUS) transducer or transducer array may be located on the distal portion of the endoscope, providing a view parallel to the endoscope axis. In some cases, the ultrasound may be radial probe EBUS, which can image radially in 360°.

[0039] Multimodal sensing systems can beneficially enhance the visual capabilities of endoscopic devices. For example, an EM sensor can provide GPS-like navigation information, allowing the user to navigate to the target site and view anatomical landmarks and features using a direct visualization camera to further confirm the location relative to preoperative images (e.g., a 3D CT scan model). Once the target site is reached, if the lesion is not visible in the airway, ultrasound information can be used to identify the precise location of the lesion on-site and in real time. Ultrasound imaging can be used to visualize the depth (e.g., millimeters or centimeters) in tissue near the ultrasound transducer to determine the exact location of the lesion for biopsy or delivery of treatments or therapies (e.g., pharmacological, mechanical, or thermotherapy). In some cases, ultrasound location can be registered to 3D location information based on the EM sensor so that the lesion location identified by ultrasound imaging can be used to automatically control the position of the catheter tip or instrument.

[0040] In some implementations, the ultrasound probe can be detachably integrated into the bronchoscopy system to enhance visual guidance for the user. The ultrasound probe may include an ultrasound transducer located on the distal portion of the probe and may be movable along the length of the endoscope.

[0041] In some embodiments, the hybrid probe can be integrated into an existing bronchoscopy system. The hybrid probe may include at least a camera, a position sensor (e.g., an EM sensor), and an ultrasonic transducer to provide multimodal sensing capabilities to the endoscope assembly. The bronchoscope may be an articulated device controlled to navigate a path guided by the hybrid probe. In other embodiments, the hybrid probe may include a position sensor (e.g., an EM sensor) and an ultrasonic transducer, while the articulated device may include a camera and a position sensor (e.g., an EM sensor).

[0042] Figure 1 Examples of assemblies of an endoscope system 100 according to some preferred embodiments of the present disclosure are shown. The endoscope system 100 may include an articulated bronchoscope 110 and a hybrid probe 120. The hybrid probe 120 may be detachably assembled to the bronchoscope, for example, through a channel for inserting a bronchoscope catheter.

[0043] The bronchoscope 110 may include suitable means for deflecting the distal tip 111 of the scope with minimal deflection or frictional force against surrounding tissue to follow the path of the structure being examined. For example, a control cable or traction cable is carried within the endoscope body to connect an articulated section adjacent to the distal end 111 to a set of control mechanisms (e.g., a handle) at the proximal end of the endoscope or robotic support system.

[0044] The robotic bronchoscope system 100 can be releasably coupled to an instrument drive mechanism. The instrument drive mechanism can be mounted to the arm of a robotic support system or any actuated support system. The instrument drive mechanism can provide mechanical and electrical interfaces to the robotic bronchoscope system 100. The mechanical interface allows the robotic bronchoscope system 100 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope 110 can be attached to the instrument drive mechanism via quick-release devices such as magnets and spring-loaded levels. In some cases, the robotic bronchoscope 110 can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0045] Figure 8 An example of a robotic bronchoscope system supported by a robotic support system is shown. In some cases, the handle portion can communicate electrically with the instrument drive mechanism (e.g., instrument drive mechanism 820) via an electrical interface (e.g., a printed circuit board), allowing image / video data and / or sensor data to be received by the communication module of the instrument drive mechanism and transmitted to other external devices / systems. In some cases, the electrical interface can establish electrical communication without cables or wires. For example, the interface may include pins soldered to an electronic board such as a printed circuit board (PCB). For example, a socket connector (e.g., a female connector) is provided on the instrument drive mechanism as a mating interface. This can advantageously allow the endoscope to be quickly inserted into the instrument drive mechanism or robotic support without the use of additional cables. This type of electrical interface can also be used as a mechanical interface, allowing mechanical and electrical coupling to be established when the handle portion is inserted into the instrument drive mechanism. Alternatively or additionally, the instrument drive mechanism may provide only a mechanical interface. The handle portion can communicate electrically with modular wireless communication devices or any other user equipment (e.g., portable / handheld devices or controllers) used for transmitting sensor data and / or receiving control signals.

[0046] like Figure 8 As shown, the robotic bronchoscope 820 may include a handle portion 813 and a flexible elongated member 811. In some embodiments, the flexible elongated member 811 may include a shaft, a maneuverable tip, and a maneuverable segment. The robotic bronchoscope 820 can be used with... Figure 1 The steerable catheter assembly described herein is identical. A robotic bronchoscope may be a single-use robotic endoscope. In some cases, only the catheter may be disposable. In some cases, at least a portion of the catheter may be disposable. In some cases, the entire robotic bronchoscope may be released from the instrument drive mechanism and can be discarded. The bronchoscope may have varying degrees of stiffness along its axis to improve functional operation.

[0047] The robotic bronchoscope can be releasably coupled to an instrument drive mechanism 820. The instrument drive mechanism 820 can be mounted to the arm of a robotic support system or to any actuated support system described elsewhere herein. The instrument drive mechanism can provide mechanical and electrical interfaces to the robotic bronchoscope 820. The mechanical interface allows the robotic bronchoscope 820 to be releasably coupled to the instrument drive mechanism. For example, the handle portion of the robotic bronchoscope can be attached to the instrument drive mechanism via quick-release devices such as magnets and spring-loaded levels. In some cases, the robotic bronchoscope can be manually coupled to or released from the instrument drive mechanism without the use of tools.

[0048] In some cases, a separate instrument drive mechanism can be used to control the movement of the hybrid probe. For example, the proximal portion of the hybrid probe can be coupled to a second instrument drive mechanism to articulate the tip of the hybrid probe. Alternatively or additionally, the rolling movement of the hybrid probe can be controlled by a controller operatively coupled to the controller of the catheter. This can advantageously provide robotic control of both the hybrid probe and the catheter, thereby allowing coordinated control of the hybrid probe and the catheter with minimal user input.

[0049] Figure 9 An example of an instrument drive mechanism 920 that provides a mechanical interface to the handle portion 913 of a robotic bronchoscope is shown. As shown in the example, the instrument drive mechanism 920 may include a set of motors actuated to rotatably drive a set of draw cables of a catheter. The handle portion 913 of the catheter assembly may be mounted on the instrument drive mechanism such that its pulley assembly is driven by the set of motors. The number of pulleys may vary depending on the draw cable configuration. In some cases, one, two, three, four, or more draw cables may be used to articulate the catheter.

[0050] The handle portion can be designed to allow for single-use of the robotic bronchoscope at a reduced cost. For example, conventional manual and robotic bronchoscopes may have cables at the proximal end of the bronchoscope handle. These cables typically include illumination fibers, camera video cables, and other sensor fibers or cables, such as electromagnetic (EM) sensors or shape-sensing fibers. Such complex cables can be expensive, increasing the cost of the bronchoscope. The robotic bronchoscopes offered can have optimized designs that allow for simplified structures and components while maintaining mechanical and electrical functionality. In some cases, the handle portion of the robotic bronchoscope can be designed without cables, while still providing a mechanical / electrical interface for the catheter.

[0051] In some cases, the handle portion may be a housing or include components configured to process image data, provide power, or establish communication with other external devices. In some cases, the communication may be wireless. For example, wireless communication may include Wi-Fi, radio communication, Bluetooth, IR communication, or other types of direct communication. Such wireless communication capabilities can allow the robotic bronchoscope to operate in a plug-and-play manner and can be easily disposed of after a single use. In some cases, the handle portion may include circuitry elements, such as a power source for powering electronics (e.g., cameras and LED light sources) located within the robotic bronchoscope or duct.

[0052] The handle portion can be designed in conjunction with the catheter, thus eliminating the need for cables or fiber optics. For example, the catheter portion can be designed with a working channel allowing instruments to pass through a robotic bronchoscope, a vision channel allowing mixed probes to pass through, and low-cost electronics (such as advanced chip cameras), an illumination source (such as light-emitting diodes (LEDs)), and an EM sensor positioned optimally according to the catheter's mechanical structure. This simplifies the design of the handle portion. For example, by using LEDs for illumination, the terminals at the handle portion can be based solely on soldering or wire crimping. For example, the handle portion can include a proximal circuit board where camera cables, LED cables, and EM sensor cables are terminated, and the proximal circuit board connects to the handle portion's interface and establishes an electrical connection to the instrument drive mechanism. As described above, the instrument drive mechanism is attached to the robotic arm (robotic support system) and provides mechanical and electrical interfaces to the handle portion. This can advantageously improve assembly and implementation efficiency and simplify manufacturing processes and costs. In some cases, the handle portion and the catheter can be discarded after a single use.

[0053] Return to reference Figure 1 In some cases, the distal tip of the bronchoscope may include a position sensor (such as EM sensor 113) to track the position of the distal tip of the bronchoscope relative to a global reference frame or patient anatomy.

[0054] The bronchoscopy catheter may include a lumen sized to accommodate a hybrid probe and a lumen or working channel 130 to accommodate instruments. Various instruments, such as biopsy needles, grippers, scissors, baskets, snares, curettes, laser fibers, suture tools, balloons, morcellators, various implant or stent delivery devices, can be inserted through the lumen. In some cases, the distal tip position tracked by the EM sensor 113 can be registered with ultrasound imaging (provided by the hybrid probe) to determine the position of the distal tip of the catheter and / or instrument carried in the working channel 130 relative to a lesion location identified by ultrasound. In some cases, instrument movement can be captured in a linear EBUS view 122.

[0055] The hybrid probe 120 may include at least a position sensor such as an EM sensor 123 and an ultrasound transducer 121. The transducer 121 may include an array of one or more transducers to perform real-time ultrasound imaging. In some cases, the transducer may include a convex ultrasound transducer located at the tip of the probe, allowing linear scanning parallel to the bronchoscope insertion direction to evaluate structures surrounding the central airway (e.g., linear EBUS view 122). The linear EBUS view is parallel to the bronchoscope insertion direction. This can be achieved either through direct contact between the hybrid probe and the airway wall or via a distally filled water-filled balloon.

[0056] A hybrid probe can be connected to an ultrasound scanner with a color Doppler system to better distinguish between solid and vascular structures. In an example, the ultrasound transducer may include a 7.5 MHz linear curved array ultrasound transducer located on the distal portion of the probe to provide B-mode and / or color Doppler imaging. The ultrasound imaging system can provide an ultrasound view122 by scanning a field of view at an angle to the longitudinal axis or by tilting the camera view forward at an angle. As an example without limitation, the ultrasound imaging system can provide a scan range defined by a field of view ranging from 30° to 80°, a line of sight ranging from 25° to 90° forward tilt, and a depth of field ranging from 2 mm to 80 mm.

[0057] The hybrid probe 120 may include position sensors, such as an EM sensor 123. The EM sensor may be located on the distal portion of the hybrid probe to track the real-time position of the distal probe relative to a global reference frame. In some cases, the EM sensor may be optional, and the position of the distal tip of the hybrid probe may be obtained based on EM sensor data provided by an EM sensor 113 located at the bronchoscope and the relative position of the hybrid probe to the bronchoscope tip. The EM sensor 113 located at the distal tip of the catheter and the EM sensor 123 located at the hybrid probe can be used for robotic control of the position / movement of the hybrid probe and the bronchoscope. For example, control commands may be generated based on position information (e.g., EM sensor data captured by EM sensors 113 and 123) to coordinate the movement of the catheter 110 and the hybrid probe 120.

[0058] In some cases, the hybrid probe 120 may include an imaging device, such as a camera 124. The camera 124 may be located at the tip of the hybrid probe to provide a forward video / camera view 125 parallel to the longitudinal axis of the probe. Alternatively, the camera 124 may be integrated into the distal tip 111 of the bronchoscope.

[0059] In some embodiments, the hybrid probe 120 may include an illumination device 126 located at the distal end of the probe. The illumination device may include one or more light sources positioned at the distal tip of the hybrid probe 120. The illumination device may be located at the distal end of the hybrid probe. Alternatively or additionally, the illumination device may be located at the distal end of the articulated bronchoscope 110. The light source may be a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be miniaturized LED or dual-color flash LED illumination for a compact design. In some cases, the illumination device may be fiber-optic based illumination, which may include a single fiber or bundle of fibers coupled to an LED or laser.

[0060] In some cases, each of one or more LEDs can be connected to a power cord that extends to the proximal handle of the hybrid probe. In some embodiments, the LEDs can be soldered to separate power cords, which are then bundled together to form a single strand. In some embodiments, the LEDs can be soldered to a power supply pull wire. In other embodiments, the LEDs can be crimped or directly connected to a single pair of power cords. In some cases, a protective layer, such as a biocompatible adhesive film, can be applied to the front surface of the LED to provide protection while allowing light to be emitted.

[0061] Imaging device 124, illumination device, EM sensor 123, and ultrasonic transducer can be integrated into the hybrid probe. For example, the distal portion of the hybrid probe may include a suitable structure matching at least one dimension of the aforementioned electronics. In some cases, the distal tip of the hybrid probe may be sized such that one or more electronic components can be embedded within it. For example, imaging device 124 may be embedded in a cavity at the distal tip of the hybrid probe. The cavity may be integrally formed with the distal portion of the hybrid probe and may have dimensions matching the length / width of a camera, allowing the camera to remain stationary relative to the hybrid probe.

[0062] In some cases, the EM sensor 123 can be located on the distal portion of the hybrid probe and can be placed in a three-dimensional arrangement next to or behind the lighting source (e.g., LED). Figure 10 An example of the distal portion of a hybrid probe with integrated imaging and illumination devices is shown. It should be noted that the distal tip design of the hybrid probe is also suitable for... Figure 4The distal tip of the integrated vision conduit described herein. A camera may be located at the distal portion. The distal tip may have a structure to accommodate the camera, illumination device, and / or position sensor. For example, the camera may be embedded in a cavity 1010 at the distal tip of the conduit. The cavity 1010 may be integrally formed with the distal portion of the cavity and may have dimensions matching the length / width of the camera, such that the camera does not move relative to the conduit. In some cases, the distal portion may include a structure 1030 having dimensions matching the size of a miniaturized LED light source. As shown in the illustrated example, two cavities 1030 may be integrally formed with the distal portion to accommodate two LED light sources. Any number of light sources may be included. The internal structure of the distal portion may be designed to accommodate any number of light sources.

[0063] In some cases, each LED can be connected to a power cord that extends to the proximal end of the hybrid probe or bronchoscope (e.g., Figure 4 (As described in the integrated vision implementation). In some implementations, the LED can be soldered to separate power lines, which are then bundled together to form a single strand. In some implementations, the LED can be soldered to a power supply pull wire. In other implementations, the LED can be crimped or directly connected to a single pair of power lines. In some cases, a protective layer, such as a thin film of biocompatible adhesive, can be applied to the front surface of the LED to provide protection while allowing light to be emitted. In some cases, an additional cap 1031 can be placed at the forward end face of the distal tip to provide precise positioning of the LED and sufficient adhesive space. The cap 1031 can be made of a transparent material with a refractive index matching that of the adhesive, so that the illumination light is not blocked.

[0064] A distal electromagnetic coil can be used in conjunction with an electromagnetic tracking system to detect the distal position and orientation of the hybrid probe as it is positioned within the anatomical system. In some implementations, the coil can be angled to provide sensitivity to electromagnetic fields along different axes, enabling the measurement of all six degrees of freedom: three positions and three angles. During navigation, such as when the hybrid probe is retracted into the catheter, an EM field generator positioned next to, below, or above the patient's torso can position the EM sensor, thereby tracking the position of the catheter tip in real time. During surgery, such as when the hybrid probe is extended distally into the catheter, the EM sensor can track the position of the hybrid probe tip in real time.

[0065] Return to reference Figure 1 The provided hybrid probe can have a compact configuration of electronic components located on the distal portion. For example, one or more electronic components can be bundled together to provide a compact design. Figure 11An example of a compact configuration of the electronics located on the distal portion is shown. In some cases, the illumination source 1120 and one or more position sensors 1110 may be bundled together. In some cases, the cable connected to the ultrasonic transducer 1130 may also be bundled with cables of other electronics to further reduce the size of the hybrid probe.

[0066] Return to reference Figure 1 The camera 124 can be powered by a wired cable. In some cases, the cable may be in the form of a harness to power the camera and illumination elements or other circuitry at the distal tip of the hybrid probe. The camera and / or light source may be powered from a power source located in the handle portion of the hybrid probe via wires, copper wires, or any other suitable means running the length of the hybrid probe. In some cases, real-time images or videos of tissues or organs may be wirelessly transmitted to an external user interface or display. Wireless communication may be WiFi, Bluetooth, RF communication, or other forms of communication. In some cases, images or videos captured by the camera may be broadcast to multiple devices or systems. In some cases, image and / or video data from the camera may be transmitted along the length of the hybrid probe via wires, copper wires, or any other suitable means to a processor located in the handle portion. Image or video data may be transmitted to external devices / systems via a wireless communication component in the handle portion. In some cases, the system may be designed so that the operator cannot see the wires or has no wires exposed to the operator.

[0067] As described above, the mixing probe is movable relative to the bronchoscope. The mixing probe can have translational movement along the longitudinal axis of the lumen and rotational movement relative to the bronchoscope. For example, the mixing probe can extend relative to the distal tip of the bronchoscope. For example, the mixing probe can slide along the lumen of the duct. When mixed vision is required, the tip of the mixing probe can extend along the longitudinal axis of the distal tip of the bronchoscope. Alternatively or additionally, the mixing probe can be hinged independently of the bronchoscope. For example, the mixing probe can have a curved section that can be hinged independently of the movement of the bronchoscope.

[0068] As described above, an imaging device located at the hybrid probe provides a forward-facing view aligned with the forward direction of the hybrid probe. Providing the imaging device at the distal tip of the hybrid probe, rather than on the catheter, advantageously allows for a clear near-field view of the tissue or organ without obstruction by the operation of the hybrid probe or tool (e.g., a needle). Furthermore, providing the imaging device on the hybrid probe allows for additional degrees of freedom in the camera view, decoupling the camera field of view from the tool's workspace. The camera view provided by the hybrid probe can be controlled with improved flexibility (e.g., by controlling the attitude or orientation of the imaging device through the hinge of the hybrid probe relative to the catheter) to better coordinate with the operation of the tool.

[0069] In some cases, an additional imaging device can be provided at the distal tip of catheter 111. This advantageously allows imaging devices located at the hybrid probe and the catheter to simultaneously provide different camera views. For example, when the hybrid probe is articulated relative to the catheter, the camera view of the imaging device located at the hybrid probe may differ from the camera view of the imaging device located at the catheter.

[0070] In some cases, catheters / bronchoscopes and hybrid probes can be robotically controlled to coordinate camera views from imaging equipment and tool operation. For example, a catheter can be advanced toward a target site under robotic control in a robotic bronchoscopy system. Once it reaches the target site, the distal tip of the catheter can be locked to provide a forward-looking camera view, while the hybrid probe can be robotically controlled (e.g., extended, retracted, articulated, rotated / rolled relative to the catheter) to provide the desired ultrasound view of the lesion and / or a camera view of the needle. In some cases, the hybrid probe can automatically adjust its view based on the operation of the tool (e.g., the needle).

[0071] Figure 2 An example 200 is shown, comprising a hybrid probe 213 that can extend distally over the distal tip 211 of a bronchoscope. The bronchoscope can be used with... Figure 1 The bronchoscope described herein is identical. For example, the bronchoscope may include at least one working channel 215 for receiving instruments and an EM sensor located at the distal tip. In some cases, during navigation to a target site where only direct vision (e.g., camera view) is required, the hybrid probe may be flush with the distal surface 217 of the catheter. In some cases, when ultrasound imaging is required, the hybrid probe may extend beyond the catheter tip to expose the ultrasound transducer 219. The hybrid probe and bronchoscope may include reversible interlocking features that lock the position of the hybrid probe relative to the bronchoscope to prevent axial and / or rotational movement. In some cases, when the hybrid probe is being operated to provide ultrasound imaging, it may be rotatable relative to the bronchoscope to locate the lesion site. For example, the interlocking features may lock the position / or orientation of the hybrid probe relative to the bronchoscope when the hybrid probe extends to a desired length and / or rotates to a desired angle.

[0072] Figure 3 An example of a hybrid probe 300 rotating relative to the distal portion 310 of the catheter is shown. The ultrasonic transducer 301 can be rotated to any desired position to provide imaging around the probe tip. As shown in the example, the hybrid probe 300 extends to the desired extent at the catheter tip, and the ultrasonic transducer 301 can be rotated to different angular positions for imaging. In some cases, once the hybrid probe is positioned at the desired angular and / or length position, an interlocking feature locks the position of the hybrid probe relative to the bronchoscope.

[0073] The hybrid probe can rotate independently of the catheter, allowing visualization of any point around the distal portion of the hybrid probe. Rotational movement of the hybrid probe can be robotic or manually controlled. Rotational operation allows the user to reposition the linear ultrasound field of view across the entire circumference of the airway. For example, lesion locations can be identified using EBUS and by rotating the hybrid probe; once identified, the catheter can be repositioned accordingly (e.g., inserted or rotated), allowing biopsy instruments to be introduced and withdrawn from the working channel and directly access the lesion location.

[0074] In some implementations, the ultrasonic transducer and camera can be integrated into the bronchoscope as a single device. Figure 4 An example of an endoscope 400 with integrated hybrid vision is shown. As illustrated in the example, the distal tip of the endoscope may include an integrated forward-facing camera 405 and an illumination source for direct visualization, as well as an integrated linear ultrasound probe 401 oriented laterally. One or more position sensors, such as an EM sensor 403, may also be integrated into the distal tip.

[0075] Imaging devices, illumination devices, EM sensors, and ultrasonic transducers can be integrated into the catheter. For example, the distal portion of the catheter may include a suitable structure matching at least one size of the aforementioned electronics. In some cases, the distal tip of the catheter may be sized to allow integration of one or more electronic components onto the catheter. For example, the outer diameter of the distal tip may be approximately 4 to 4.4 millimeters (mm), and the diameter of the working channel may be approximately 2 mm, allowing one or more electronic components to be embedded in the distal tip. However, it should be noted that, depending on the application, the outer diameter can be in any range, less than 4 mm or greater than 4.4 mm, and the diameter of the working channel can be in any range, depending on the tool size or specific application.

[0076] The EM sensor, camera, and ultrasonic transducer can be the same as those described elsewhere in this document. For example, the camera can be an on-chip camera, and the illumination source can be a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source can be miniaturized LEDs or dual-color flash LEDs for compact designs. In some cases, the aforementioned electronics can be embedded in the tip of the conduit.

[0077] The distal bronchoscope may include a working channel 413 with a side outlet / port 414. In some cases, the side outlet 414 may be located at the articulated portion 411 of the catheter. The catheter may include a shaft 410, an articulated (bent) section 411, and a maneuverable distal portion 412, wherein the articulated (bent) section 411 connects the maneuverable distal portion to the shaft 410. For example, the bent section 411 may be connected at a first end to a distal tip portion and at a second end to the shaft portion, wherein the bent section is articulated by one or more draw wires. In some cases, the bent section may be manufactured separately as a modular component and assembled onto the shaft. In some cases, the bent section may further incorporate minimalist features to reduce cost and improve reliability. For example, the bent section may incorporate a cut pattern that advantageously allows for a greater degree of tube deflection to achieve a desired tip displacement relative to the shaft. In some cases, the bent section may consist of a stainless steel strip. The bending segment can be formed from other suitable structures or materials to achieve a predetermined bending stiffness while maintaining the desired axial and torsional stiffness with low hinge forces. For example, the bending segment may include a braided structure for torsional stability.

[0078] Side port 414 is located at bend 411. The distal end of the instrument extends along a path that runs from a position within a slender, flexible shaft through a side outlet to an extension outside the catheter at an angle Θ relative to the longitudinal axis of the catheter tip. This allows the instrument to extend from side port 414 and fall within EBUS view 417 (for ultrasound capture), even without articulated instruments.

[0079] The working channel lumen may have a port located at a bend, and the lumen allows the instrument to extend out of the port along a trajectory intersecting the EBUS view. For example, as... Figure 4 As shown, the working channel exit trajectory 416 intersects with the field of view 417 of linear endobronchial ultrasound (EBUS) or an image captured by ultrasound. A biopsy device 415 (e.g., forceps, needle, brush) can extend along a path from a position inside the outer wall through a side outlet / port to a position outside the outer wall at an angle of at least 30 degrees relative to the longitudinal axis. The path of the biopsy device 415 can intersect with / with the EBUS field of view (e.g., ultrasound image) such that at least one tip portion of the instrument can be seen in the ultrasound image. In some cases, the path of the biopsy device can be calibrated to the position of an electromagnetic positioning (EM) sensor 403, which is positioned at the distal portion of an elongated flexible shaft 410 and displayed by a surgical instrument navigation system.

[0080] The rotatable and manipulable catheter allows visualization of any point around the distal end of the catheter. The rotational movement of the catheter can be robotically controlled. Rotational manipulation allows the user to reposition the linear ultrasound field of view throughout the entire airway circumference. For example, lesion locations can be identified using EBUS and by rotating the catheter; once identified, biopsy instruments can be introduced and pushed out of the working channel directly to the lesion location.

[0081] In some embodiments, the hybrid vision assembly may include a bronchoscope with integrated direct vision and position sensing, and a hybrid probe with a balloon. The distal end of the bronchoscope may include an integrated forward-looking camera and illumination for direct visualization, and an integrated EM sensor (e.g., at least 3 degrees of freedom) for generating 3D positioning information.

[0082] In some cases, the hybrid probe may include at least an ultrasonic transducer and an EM sensor located in the distal portion of the hybrid probe and covered by one or more balloons. In some cases, the hybrid probe may include a guidewire with an expandable outer diameter feature at the tip where the EBUS transducer and EM sensor are located. Figure 5 An example of a hybrid probe with an expandable tip 503 is shown. Figure 5 As shown, the working channel exit trajectory 508 intersects with a linear EBUS view 509 of a linear endobronchial ultrasound (EBUS) or ultrasound image. A hybrid probe can be inserted through the working channel of the catheter / bronchoscope 505 and extend above the catheter to aid in navigation of the airway 510 in the lungs. In some cases, the tip of the hybrid probe can extend through the tip of the catheter into the desired airway 510, and the catheter can then slide over the guidewire / probe to reach the desired location. Various suitable methods, such as an inflatable balloon, can be used to achieve the inflatable tip. The balloon 503 can be positioned distal to or near the distal end of the probe and can cover the linear EBUS transducer 507 and the EM sensor 501. The balloon can be connected via the working channel to a balloon inflation source or pump for balloon inflation or deflation.

[0083] In some cases, the balloon can also create a seal by traversing a passage at the sealing point before collapsing the passage distal to the sealing point. One or more balloons can inflate and / or collapse to form a temporary closure within the passage by injecting air, saline, and / or some other gas or liquid. In some examples, the balloon is malleable in shape upon inflation, allowing it to conform to the shape of the passage to expand around the tips of one or more sensors and a hybrid probe.

[0084] The EM sensor and ultrasonic transducer can be the same as those described elsewhere in this document. For example, the EM sensor located at the tip of catheter 505 and the EM sensor covered by the balloon can be 6-DOF EM sensors.

[0085] Figure 6 An example of a hybrid probe 603 with a balloon tip is shown. For example, hybrid probe 603 can provide an ultrasound view of the lesion site as well as 3D positioning information from an EM sensor. A balloon 601 may cover the EM sensor and ultrasound transducer 605. A hybrid probe 607 with a collapsible balloon is shown. In some cases, EM sensor information can be registered and correlated with ultrasound images. The hybrid probe can be detachably coupled to the catheter. The hybrid probe can be movable relative to the catheter, such as being able to insert and retract and rotate along the longitudinal axis of the catheter to capture various ultrasound views and EM data points. Once the lesion site is identified and its 3D location is determined, the physician can manipulate the position of the bronchoscope to align the trajectory of the biopsy tool with the location of the lesion. The position or trajectory of the biopsy tool can be determined at least in part based on positional data provided by an EM sensor located at the distal tip of the catheter and the known model of the catheter and / or instrument.

[0086] The catheter, bronchoscope, and / or hybrid probe can be robotically controlled. For example, the catheter can be advanced toward the target site under the robotic control of a robotic bronchoscopy system. The catheter can be manipulated or advanced toward the target site manually, autonomously, or semi-autonomously. In another example, the movement of the hybrid probe can be automatically controlled, allowing for automated control of insertion, retraction, and rotational movement for taking ultrasound images. Alternatively, the bronchoscope and / or hybrid probe can be controlled via a handheld device.

[0087] user interface In some implementations, real-time images of the target site (e.g., including lesions) provided by the multi-mode sensing system can be displayed on the user interface. Figure 7 An example of a user interface is shown, displaying an EBUS view for real-time, accurate lesion location tracking, a direct video / camera view, and a virtual airway model based on preoperative images (e.g., preoperative CT images).

[0088] In some cases, the real-time location of the lesion can be identified from the ultrasound image. The lesion location can be registered to the coordinate system of the bronchoscopy system using EM sensor data. In some implementations, the location of the lesion can be segmented from the image data captured by the linear EBUS probe using a signal processing unit. In some cases, one or more processors of the signal processing unit can be configured to overlay the treatment location (e.g., the lesion) onto the real-time ultrasound image / video. In some cases, the segmented lesion image and the optimal path for navigation of elongated components to the lesion can both be overlaid on the real-time ultrasound image. This allows the operator or user to visualize the precise location of the lesion and the planned path of bronchoscopy movement.

[0089] In some implementations, the bronchoscopy system may include a navigation and positioning subsystem configured to construct a virtual airway model based on preoperative images (e.g., preoperative CT images). The navigation and positioning subsystem may be configured to identify approximate segmented lesion locations within the 3D-rendered airway model, and based on the lesion's location, generate an optimal path from the main bronchus to the lesion with a recommended approach angle toward the lesion to perform a surgical procedure (e.g., biopsy). For example, the processing unit may be configured to generate an enhancement layer that includes enhancement information such as the location of the treatment site or the lesion. In some cases, the enhancement layer may also include graphic markers indicating the path to the target site. The enhancement layer may be a substantially transparent image layer comprising one or more graphic elements (e.g., boxes, arrows, etc.). The enhancement layer may be overlaid on an optical view of an optical image or video stream captured by a fluoroscopic imaging system and / or displayed on a display device. The transparency of the enhancement layer allows a user to view the optical image through the graphic elements overlaid thereon. In some cases, segmented lesion images and the optimal path for navigation of slender components to the lesion can be overlaid on a virtual airway model or preoperative images. This allows the operator or user to visualize the approximate location of the lesion and the planned path of bronchoscopy movement. In some cases, segmented and reconstructed images (e.g., CT images as described elsewhere) provided prior to operation of the system described herein can be overlaid on real-time images.

[0090] In the registration step before driving the bronchoscope toward the target site, the system aligns a virtual view of the presented airway with the patient's airway. Image registration may include a single registration step or a combination of a single registration step and real-time sensing updates of the registration information. Once registered, all airways are aligned with the pre-presented airways. During the driving of the robotic bronchoscope toward the target site, the position of the bronchoscope within the airway can be tracked and displayed. In some cases, position sensors can be used to track the position of the bronchoscope relative to the airway. Other types of sensors, such as cameras, can also be used instead of or in combination with position sensors using sensor fusion technology. Position sensors, such as electromagnetic (EM) sensors, can be embedded at the distal tip of the catheter, and an EM field generator can be positioned next to the patient's torso during the procedure. The EM field generator can position the EM sensors in 3D space, or position and orient the EM sensors in 5D or 6D space. This provides visual guidance to the operator when driving the bronchoscope toward the target site.

[0091] During the procedure, the location of the lesion can be updated and tracked in real time based on ultrasound images. In some cases, the location of the lesion can be marked on the ultrasound image, and the distal tip of the bronchoscope can be hinged, rotated, or moved to align the instrument's trajectory with the lesion location.

[0092] In some implementations, the processor can use the real-time location of one or more of the lesion, bronchoscope, hybrid probe, and instrument to calculate, for example... Figure 4 and Figure 5 The illustration shows one or more possible geometries / configurations. For example, the system can calculate the appropriate motion of the hybrid probe in order to establish the desired configuration (e.g., Figure 5The system tracks, positions, and positions of the instruments (as seen in the EBUS view) to allow the instruments to approach the lesion in the case of direct ultrasound imaging. Furthermore, in some implementations, the system can partially or completely control the movement of the hybrid probe, bronchoscope, and instruments to establish optimal geometry / configuration. For example, once the system knows and can track the positions of all these elements, it can generate a plan executed by the robotic system to move the instruments, hybrid probe, and bronchoscope relative to the lesion site into the desired configuration. In some cases, guidance for operating one or more instruments, hybrid probes, and bronchoscopes can be displayed to the user. For example, guidance such as "Stop the bronchoscope here," "Move the hybrid probe to this position," or "Insert the instrument immediately" can be provided to achieve the desired configuration. In some cases, configuration can be achieved in a semi-autonomous manner. For example, the user can be allowed to activate / trigger movement or stop movement while the robotic system automatically controls the movement of the hybrid probe, instruments, and / or bronchoscope (e.g., the user can hold an activation trigger, and the robotic system moves the hybrid probe, instruments, and / or bronchoscope while the trigger is held). In some cases, robotic systems can stabilize the desired geometry / configuration by automatically tracking the patient's respiratory movements (e.g., respiratory movement compensation).

[0093] Figure 7 An example user interface is shown for visualizing real-time ultrasound images 705, a virtual airway model 709 with EM tracking of the catheter tip position 701, and a direct camera view 711. In some cases, the virtual airway 709 overlays an optimal path 703, the catheter tip position 701, and the approximate location of the solitary pulmonary nodule (displayed via CT scan). In this example, the position of the catheter tip is displayed in real-time relative to the virtual airway model 709, thus providing visual guidance. Figure 7 As illustrated in the example, during robotic bronchoscopy, an optimal path 703 can be displayed and overlaid on a virtual airway model. The virtual airway model can be constructed based on real-time fluorescence fluoroscopic images / videos (and positional data from the imaging system). The user can also be presented with camera views or images / videos 711 captured by the bronchoscopy, as well as ultrasound views 705 displaying the accurate location of lesions in real time.

[0094] This invention provides, but is not limited to, the following embodiments: 1. A hybrid vision device, comprising: A hinged elongated member including a proximal end and a distal end, wherein a first position sensor is located at the distal end of the hinged elongated member; and A multi-mode sensing probe detachably coupled to the articulated elongated member, wherein the multi-mode sensing probe includes an ultrasonic transducer and a camera located on the distal portion of the multi-mode sensing probe.

[0095] 2. The hybrid vision device according to embodiment 1, wherein the multi-mode sensing probe is inserted through a first cavity of the articulated elongated member.

[0096] 3. The hybrid vision device according to embodiment 1, wherein the multi-mode sensing probe is rotatable and extendable relative to the hinged elongated member.

[0097] 4. The hybrid vision device according to embodiment 1, wherein the multi-mode sensing probe is hinged to the hinged elongated member.

[0098] 5. The hybrid vision device according to embodiment 1, wherein the multi-mode sensing probe further includes a second position sensor located at a distal portion of the multi-mode sensing probe to track the position of the distal portion of the multi-mode sensing probe.

[0099] 6. The hybrid vision device according to embodiment 5, wherein the second position sensor, the camera, and the illumination device are embedded in the distal portion of the multi-mode sensing probe.

[0100] 7. The hybrid vision device according to embodiment 6, wherein the second position sensor, the camera, and the lighting device are arranged in a compact configuration.

[0101] 8. The hybrid vision device according to embodiment 5, wherein the articulated elongated member and the multi-mode sensing probe are robotically controlled at least in part based on sensor data captured by the first position sensor and the second position sensor.

[0102] 9. The hybrid vision device according to embodiment 1, wherein the ultrasonic transducer is an array of linear endobronchial ultrasound (EBUS) transducers.

[0103] 10. The hybrid vision device according to embodiment 1, wherein the camera provides a real-time forward view and the ultrasonic transducer provides a real-time side view.

[0104] 11. The hybrid vision device according to embodiment 10, wherein the articulated elongated member includes a second lumen for receiving an instrument.

[0105] 12. The hybrid vision device according to embodiment 11, wherein the movement of the device is captured by the real-time side view.

[0106] 13. The hybrid vision device according to embodiment 1, wherein the articulated elongated member further includes an imaging device located at the distal end of the articulated elongated member.

[0107] 14. The hybrid vision device according to embodiment 1, wherein the multi-mode sensing probe includes an expandable tip.

[0108] 15. A hybrid vision device, comprising: A hinged, slender member comprising a distal tip portion and a curved section; An ultrasonic transducer, located at the distal tip portion, is used to provide ultrasonic imaging; and The channel is configured as a receiving device, wherein the lumen has a port located at the bend, thereby allowing the device to extend out of the port along a trajectory intersecting the ultrasound image.

[0109] 16. The hybrid vision device according to embodiment 15, wherein the articulated elongated member includes a position sensor located at the distal tip portion.

[0110] 17. The hybrid vision device according to embodiment 16, wherein the position sensor is embedded in the distal tip portion to track the position of the distal tip portion.

[0111] 18. The hybrid vision device according to embodiment 15, wherein the articulated elongated member includes a camera embedded in the distal tip portion.

[0112] 19. The hybrid vision device according to embodiment 18, wherein the camera provides a real-time forward view.

[0113] 20. The hybrid vision device according to embodiment 15, wherein the ultrasonic transducer is an array of linear endobronchial ultrasound (EBUS) transducers.

[0114] While preferred embodiments of the invention have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. Many variations, modifications, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of the invention. The following claims are intended to define the scope of the invention and thereby cover the methods and structures within the scope of these claims and their equivalents.

Claims

1. A hybrid vision device, comprising: A hinged elongated member including a proximal end and a distal end, wherein a first position sensor is located at the distal end of the hinged elongated member; and A multi-mode sensing probe detachably coupled to the articulated elongated member, wherein the multi-mode sensing probe includes an ultrasonic transducer and a camera located on the distal portion of the multi-mode sensing probe.

2. The hybrid vision device of claim 1, wherein the multi-mode sensing probe is inserted through a first cavity of the articulated elongated member.

3. The hybrid vision device of claim 1, wherein the multi-mode sensing probe is rotatable and extendable relative to the hinged elongated member.

4. The hybrid vision device of claim 1, wherein the multi-mode sensing probe is hinged to the hinged elongated member.

5. The hybrid vision device of claim 1, wherein the multi-mode sensing probe further comprises a second position sensor located at a distal portion of the multi-mode sensing probe to track the position of the distal portion of the multi-mode sensing probe.

6. The hybrid vision device of claim 5, wherein the second position sensor, the camera, and the illumination device are embedded in the distal portion of the multi-mode sensing probe.

7. The hybrid vision device of claim 6, wherein the second position sensor, the camera, and the lighting device are arranged in a compact configuration.

8. The hybrid vision device of claim 5, wherein the articulated elongated member and the multi-mode sensing probe are robotically controlled at least in part based on sensor data captured by the first position sensor and the second position sensor.

9. The hybrid vision device of claim 1, wherein the ultrasonic transducer is an array of linear endobronchial ultrasound (EBUS) transducers.

10. A hybrid vision device, comprising: A hinged, slender member comprising a distal tip portion and a curved section; An ultrasonic transducer, located at the distal tip portion, is used to provide ultrasonic images; and The channel is configured as a receiving device, wherein the lumen has a port located at the bend, thereby allowing the device to extend out of the port along a trajectory intersecting the ultrasound image.

Images