Bronchoscope Graphical User Interface with Improved Navigation

Abstract

A navigation system, medical system, method of use, and medium for use in navigating a bendable medical device are provided. The system and method include a display device and a control device. The control device is configured to display, on the display device, an image of a biological lumen taken from a distal end of the bendable medical device and a representation of an airway structure. The representation of the airway structure includes a navigation path through at least a portion of the airway and a guidance reference surface disposed perpendicular to the navigation path and located at an insertion depth of the distal end of the bendable medical device.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 307,878, filed February 8, 2022. The disclosure of said provisional application is incorporated herein by reference in its entirety for all purposes. The benefit of priority is claimed under 35 U.S.C. § 119(e).

[0002] The present disclosure relates generally to systems and methods for medical applications. More particularly, the present disclosure is directed to a system for using an articulated medical device and displaying information from the medical device, the medical device being capable of being steered within a patient. [Background technology]

[0003] Bendable medical devices, such as endoscopic surgical instruments and catheters, are well known and continue to gain acceptance in the medical field. Bendable medical devices generally include a flexible body, commonly referred to as a sleeve or sheath. One or more tool channels extend along (typically internally) the flexible body, allowing access to a target site located at the distal end of the flexible body.

[0004] The medical device is intended to provide flexible access within a patient with one or more curves leading to a target of interest while maintaining torsional and longitudinal rigidity, allowing a clinical user to control a tool located at the distal end of the medical device by manipulating the proximal end of the device.

[0005] A medical device may be implemented via a system. The system includes both hardware and software that together allow a user to guide and observe the movement of the medical device through a passageway within a patient. By way of example, U.S. Patent Publication No. 2019 / 0105468 describes a system for implementing an articulated medical device having a cavity. The device is capable of being steered within the patient and allows medical tools, such as an endoscope, camera, catheter, etc., to be guided through the cavity for a medical procedure.

[0006] To aid in the navigation of the medical device towards the target site, the physician may use pre-operative and / or intra-operative imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US) or other similar techniques to provide a 'roadmap' for the navigation of the surgical tool in and around the patient's internal structures and organs. Navigation through the lungs is complicated by the fact that the lung anatomy fortuitously provides multiple branching paths for selecting the best route to the target, but also by the fact that the airways branch continuously, making it difficult to determine which airway the surgical tool is in relative to the target. An image of the entire lung anatomy may be provided to aid navigation. However, in order for the user to steer the catheter towards the target, the user needs to know the position of the medical device relative to the entire lung anatomy, its progress along the planned navigation path, and the possible airways available for orientation.

[0007] The difficulty is that current state-of-the-art "roadmaps" fall into one of two categories: current view models that provide only the current airway as the medical device 'sees' (see, e.g., U.S. Pat. No. 9,727,963), and 2D representation models that provide no airway options or structural detail (see, e.g., U.S. Pat. Appl. Pub. No. 2020 / 0054399).Similarly inadequate teachings are provided in U.S. Pat. No. 10,617,324, U.S. Pat. No. 8,337,397, U.S. Pat. Appl. Pub. No. 2020 / 0054399, U.S. Pat. Appl. Pub. No. 2019 / 0254649, and WO 2018 / 00586.

[0008] Thus, there is a need for improved displays and systems to aid in the navigation and use of bendable medical devices in the lungs, which would aid in visualization of the location of the bendable medical device tip within the airway structure, visualization of where the tip would be along the navigation path, and virtual endoscopic views for reviewing available options and orienting a live endoscopic view. Summary of the Invention

[0009] According to at least one embodiment of the present invention, a navigation system is provided that includes a display device and a control device. The control device is configured to display, on the display device, an image of a biological lumen taken from a distal end of a bendable medical device and a representation of an airway structure. The representation of the airway structure (which may be, for example, a segmented model of the airway obtained from a CT image) also includes, on the model image: a navigation path through at least a portion of the airway structure; and a guidance reference plane oriented perpendicular to the navigation path and located at an insertion depth of the distal end of the bendable medical device within the biological lumen. The representation of the airway structure may include a centerline, a target site, and a ring or other feature indicating the position of the endoscope tip.

[0010] The guiding reference plane is optionally updated as the bendable medical device moves through the airway structure. The controller may be configured to display a guiding virtual endoscopic view, the guiding virtual endoscopic view having an image center at a centerline of the guiding reference plane and a view orientation in the distal direction along the orientation of the guiding reference plane. Alternatively or additionally, the controller is further configured to display a second virtual endoscopic view having an image center at a distal end of the bendable medical device.

[0011] The control device may also be configured to display at least one of the following: navigation modality; distance from the bendable medical device to the target site; insertion depth of the bendable medical device; information regarding the location of markers displayed on the airway structure; and a warning that the bendable medical device is approaching a force or bend angle limit.

[0012] The navigation system described herein may also display one or more markers indicating one or more branching points of the airway structure, which may be selected and the guiding reference plane may be moved to the selected branching point of the airway structure to become the approximate guiding reference plane.

[0013] Also provided are methods and computer readable storage media for performing the steps of the methods. For example, a method is provided that includes the steps of: acquiring an image of a biological lumen from a distal end of a bendable medical device; acquiring a representation of an airway structure; acquiring a target site; generating a centerline of at least a portion of the airway structure; generating a navigation path through at least a portion of the airway structure; and displaying the representation of the airway structure on a display device. The representation of the airway structure includes the navigation path and a guiding reference plane that is positioned perpendicular to the navigation path and at an insertion depth of the distal end of the bendable medical device.

[0014] These and other objects, features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments of the present disclosure, which should be read in conjunction with the accompanying drawings and the appended claims. [Brief description of the drawings]

[0015] Further objects, features and advantages of the present invention will become apparent from the following detailed description when taken in conjunction with the accompanying drawings illustrating embodiments of the invention.

[0016] [Figure 1] FIG. 1 illustrates an example embodiment of a robotic-assisted endoscopic system 1000 in a medical environment, such as a surgical suite. [Diagram 2] FIG. 2 illustrates an exemplary embodiment of a system that allows a user to guide and observe the movement of a medical device within a patient. [Diagram 3] FIG. 3 illustrates an exemplary embodiment of a steerable medical system 1000 represented in a functional block diagram. [Figure 4] FIG. 4 illustrates the lungs along with a pathway for endoscope insertion. [Diagram 5] FIG. 5 illustrates an endoscopic view of the airway structures. [Figure 6] 6(A) and 6(B) provide example displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 7] Figures 7(A) and 7(C) provide other example displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs, while Figures 7(B) and 7(D) provide the actual catheter positions within the lungs for the images of Figures 7(A) and 7(C), respectively. [Figure 8-1] 8(A) and 8(B) provide other example displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 8-2] FIG. 8(C) provides another example display showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 9] FIG. 9 provides an example display showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 10] FIG. 10 provides an example display showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs.

[0017] Throughout the figures, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the figures, it is done so in connection with the illustrated embodiments. It is intended that changes and modifications can be made to the described embodiments without departing from the true scope and spirit of the present disclosure, as defined by the appended paragraphs. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following paragraphs, certain illustrative embodiments are described. Other embodiments may include alternatives, equivalents, and modifications. In addition, the illustrative embodiments may include some novel features, and certain features may not be essential to some embodiments of the devices, systems, and methods described herein.

[0019] Those skilled in the art will appreciate that the terms used herein, in general, and in the appended claims, particularly (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," "includes" should be interpreted as "including, but not limited to," etc.). Moreover, those skilled in the art will appreciate that, where specific numbers of introduced claim recitations are intended, such intent will be expressly set forth in the claims, and, in the absence of such recitation, no such intent exists. For example, as an aid to understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that the introduction of a claim statement with the indefinite article "a" or "an" means that a particular claim containing such an introduced claim statement is limited to claims containing only one such statement, even if the same claim contains both the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should typically be construed to mean "at least one" or "one or more"). The same is true for the use of definite articles used to introduce claim statements.

[0020] In this specification, when a feature or element is referred to as being "on" another feature or element, it may be directly on the other feature or element, or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It should also be understood that when a feature or element is referred to as being "connected," "attached," "coupled," or the like, to another feature or element, it may be directly connected, attached, or coupled to the other feature, or there may be intervening features or elements present. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or illustrated with respect to one embodiment, features and elements so described or illustrated in one embodiment may be applicable to other embodiments. Additionally, as will be understood by one of ordinary skill in the art, a reference to a structure or feature being located "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0021] In this specification, terms such as first, second, third, etc. may be used to describe various elements, components, regions, parts, and / or portions. It is to be understood that these elements, components, regions, parts, and / or portions are not limited by these designated terms. These designated terms are used only to distinguish one element, component, region, part, or portion from another region, part, or portion. Thus, a first element, component, region, part, or portion described below can be referred to as a second element, component, region, part, or portion merely for purposes of distinction, but without limitation, and without departing from the structural or functional meaning.

[0022] As used herein, the singular is intended to include the plural unless the context clearly dictates otherwise. Furthermore, it should be understood that the terms "comprise," "comprising," and "consisting," when used in this specification and claims, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof that are not expressly recited. Furthermore, in this disclosure, the transitional phrase "consisting of" excludes any element, step, or component not specified in the claim. It should also be noted that some claims or some features of the claims may be drafted to exclude any element, and such claims may use exclusive language such as "solely," "only," or "negative" limitations in connection with the recitation of the claim elements.

[0023] Unless specifically stated otherwise, and as will be apparent from the disclosure that follows, throughout this disclosure, references using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like, are understood to refer to operations and processes of a processor, such as a computer system, or similar electronic computing device, or data processing device that manipulates data represented as physical (electronic) quantities in the registers and memory of the computer system, and converts it into other data, also represented as physical quantities in the memory or registers of the computer system, or other devices for storing, transmitting, or displaying such information. The computational or electronic operations described herein or recited in the appended claims can generally be performed in any order, unless the context dictates otherwise. Also, while various operational flow diagrams are presented in a sequential order, it will be understood that the various operations can be performed in orders other than those shown or claimed, or operations can be performed simultaneously. Examples of such alternative orderings include overlapping, interleaving, interrupting, reordering, incrementing, preparing, supplementing, simultaneous, reverse, or other variations of ordering, unless the context dictates otherwise. Moreover, terms such as "in response to," "in response to," "in connection with," "based on," or other similar past tense adjectives generally do not intend to exclude such variations unless the context indicates otherwise.

[0024] As is known in the medical device art, the terms "proximal" and "distal" are used in reference to the operating ends of an instrument that extend from a user to a surgical or diagnostic site. In this regard, the term "proximal" refers to the portion of the instrument that is closer to the user (e.g., the handle) and the term "distal" refers to the portion of the instrument that is further from the user and closer to the surgical or diagnostic site (the tip). It will further be appreciated that for convenience and brevity, spatial terms such as "vertical," "parallel," "upper," and "lower" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0025] As used herein, the term "bendable medical device" generally refers to a flexible, thin, tubular instrument made of medical grade material designed to be inserted through a narrow opening into a body cavity (e.g., blood vessels or bronchi) to perform a wide range of medical functions. A catheter may be a bendable medical device. The more specific term "optical catheter" refers to a bendable medical device that includes an elongated bundle of one or more flexible, light-conducting fibers with optical imaging capabilities disposed within a protective sheath made of medical grade polymeric material. A specific example of an optical catheter is a fiber optic catheter that includes a flexible sheath, a coil, and an optical probe or imaging core housed within the coil. In some applications, the catheter may include a "guide catheter" that functions similarly to a sheath. A bendable medical device may be configured to be used with one or more tools. For example, a camera may be inserted into the bendable medical device, or a camera or other optical probe may be an integral part of the device. A biopsy tool may also be used with a bendable medical device.

[0026] As used herein, the term "endoscope" refers to a rigid or flexible medical instrument that uses light guided by an optical probe to view the interior of a body cavity or organ. A medical procedure in which an endoscope is inserted through a natural orifice is called endoscopy. Specialized endoscopes are generally named for how or where they are used, such as bronchoscope (mouth and lungs), sigmoidoscope (rectum), cystoscope (bladder), nephroscope (kidneys), bronchoscope (bronchi), pharyngoscope (pharynx), otoscope (ear), arthroscope (joints), laparoscope (abdomen), and gastrointestinal endoscope. Embodiments of the present disclosure may be applicable to one or more of the aforementioned endoscopes.

[0027] The present disclosure generally relates to medical devices and illustrates embodiments of an optical probe applicable to imaging devices (e.g., endoscopes). The embodiments of the optical probe and portions thereof are described with respect to their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates), the term "orientation" refers to the rotational configuration of an object or part of an object (three rotational degrees of freedom - e.g., roll, pitch, yaw), the term "pose" refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six degrees of freedom in total), and the term "shape" refers to a set of poses, positions and / or orientations measured along the elongated body of an object.

[0028] An exemplary configuration of a medical system 1000, such as a robotic endoscopic system, is described with reference to FIG. 1. This figure illustrates an exemplary representation of a medical environment, such as an operating room, in which the robotic endoscopic system 1000 may be used. The robotic endoscopic system 1000 may include a steerable instrument 100 (steerable medical device) that a user 10 (e.g., a physician) can operate to perform an endoscopic procedure on a patient 80. The robotic endoscopic system 1000 may include a computer system 400 operatively attached to the steerable instrument 100 via a robotic platform 90. The computer system 400 (e.g., a system console) includes a processor or central processing unit (CPU) 410 and a display screen 420, such as a liquid crystal display (LCD), OLED, or QLED display. A storage device 411 (ROM and RAM memory), a system interface 412 (FPGA card), and a user interface 413 (e.g., a mouse and a keyboard) are operatively connected to the processor or CPU 410 and the display screen 420.

[0029] The steerable instrument 100 includes a handle 200 and a bendable medical device (e.g., a steerable sheath) 110, which are detachably connected to each other by a connector assembly 50. The handle 200 includes an actuator system 300, which receives electronic commands from a computer system 400 to mechanically actuate the bendable medical device 110. The handle 200 is configured to be detachably disposed on a robotic platform 90. The robotic platform 90 has a robotic arm 92 and a stage 91 for robotically guiding the bendable medical device 110 toward a target site 82 in a subject or patient 80. When the handle 200 is not necessarily disposed on the robotic platform 90, the handle 200 can be manually manipulated by a user 10 to control the bendable medical device 110. For treatment or examination of a patient 80, the steerable instrument 100 may include one or more access ports 250 disposed on or about the handle 200. The access port 250 can be used to insert an end effector, pass fluids to and from the patient. The electromagnetic (EM) field generator 60 interacts with one or more EM sensors 190 disposed on the steerable sheath 110, which track the position, shape, and / or orientation of the steerable sheath 110 during insertion through the body cavity 81 toward a target site 82 within the patient 80. The medical device 110 may include a tool channel for a biopsy or other interventional tool. A clinical user can insert and withdraw the medical device 110, for example, to perform a biopsy within the patient's airway.

[0030] During an endoscopic procedure, a system processor or CPU 410 of the computer system 400 is configured to execute operations based on computer executable code pre-stored in the system's memory 411. The display screen 420 may include a graphical user interface (GUI) configured to display one or more of patient information in an information window 421, a live endoscopic image 422, intraoperative images 423 (e.g., fluoroscopy), and preoperative images 424 (e.g., slice images) of the patient 80.

[0031] 2 illustrates an exemplary embodiment of a medical system 1000. The medical system 1000 (also referred to herein as a continuum robotic system) comprises a drive 310, a bendable medical device 110 (or sheath), a positioning cart 500, an operation console 600, and navigation software 700. The system 100 also interacts with clinical users and external systems (e.g., computed tomography (CT) and / or magnetic resonance imaging (MRI) scanners, fluoroscopes, patients, biopsy tools).

[0032] The navigation software 700 and the driver 310 are communicatively coupled via a bus that transmits data between them. Additionally, the navigation software 700 may be coupled to and communicate with a CT scanner, an MRI scanner, an X-ray fluoroscope, and an image server (not shown in FIG. 2) external to the medical system 1000. The image server may be, for example, a DISCOM server coupled to a medical imaging device such as a CT scanner, an MRI scanner, an X-ray fluoroscope, etc. The navigation software 700 processes data provided by the driver 310 and data provided by images stored in the image server, images from the CT scanner / MRI scanner, and images from the X-ray fluoroscope in order to display images on a display device.

[0033] Images from the CT / MRI scanner are provided to the navigation software 700 preoperatively. Using the navigation software 700, a clinical user can create an anatomical computer model from the images. In some embodiments, the anatomical structures are biological lumens, such as lung airways. From the chest images from the CT / MRI scanner, a clinical user can segment the lung airways for clinical use. From this data, a lung-airway map can then be created, which can be used to create a planned path. The data, with or without the path, can also be used to guide and inform a procedure, such as a biopsy, with a bendable medical device 110 inserted into the biological lumen. In other embodiments, images from, for example, intraoperative fluoroscopy, can be used by the navigation software 700 to create (or add to) the anatomical computer model of the biological lumen.

[0034] The actuator 310 includes an actuator and control circuitry. The control circuitry is communicatively coupled to the operation console 600. The actuator 310 is also connected to the bendable medical device 110 such that the actuator of the actuator 310 moves the medical device 110. Thus, the clinical user can control the actuator 310 to move the medical device 110. 310 3, the medical device 110 can be controlled via the drive 310. The drive 310 is also physically connected to a positioning cart 500. The positioning cart 500 may include one or more positioning arms and translation stages, and the positioning cart 500 positions the drive 310 and the medical device 110 in their intended positions relative to the patient.

[0035] The operator console 600 most preferably includes one or more displays and input devices such as a mouse, joystick, touch screen, voice activation, or the like.

[0036] The medical device may include a camera at the distal tip of the device (i.e. a 'tip-on-tip' design). Alternatively, the device includes imaging means for generating an image of the area at the distal tip. For example, the image may be generated via a conventional CCD endoscope, a borescope, a fiberscope, or by spectrally encoded endoscopy (see, for example, U.S. Patent Nos. 7,551,293, 9,295,391, 10,288,868, and 10,401,610). The medical device forms an image at the distal end of the tip. This image can be used for navigation of the flexible medical device. Such an image of the interior of a body lumen (e.g. lung) or other hollow organ (e.g. renal pelvis) can be combined with CT, MRI, fluoroscopic or other images of the area of ​​interest to assist in guiding the medical device towards the target site.

[0037] The medical device 110 may include a tool channel for a biopsy or other interventional tool, such that the medical device 110 can guide a biopsy tool to a lesion in a patient, from which a clinical user can take a biopsy sample.

[0038] FIG. 3 illustrates the overall structure of the steerable medical system 1000 of FIG. 1 in a functional block diagram, without including a user and / or a patient. The medical system 1000 includes a handle 200 and a bendable medical device 110, which are detachably connected to each other by a connector assembly 50. The handle 200 includes an actuator system 300, which is part of a drive 310, which receives electronic commands from a computer system 400 to mechanically actuate the bendable medical device 110. The handle 200 is configured to be detachably placed on a robotic platform 90, which may be part of a positioning cart 500. The robotic platform 90 has a robotic arm 92 and a stage 91 for robotically guiding the bendable medical device 110 toward a target site 82 in a subject or patient 80. When the handle 200 is not placed on the robotic platform 90, the handle 200 can be manually operated by a user 10 to control the bendable medical device 110. For treatment or examination of the patient 80, the steerable medical system 1000 may include one or more access ports 250 located on or about the handle 200. The access ports 250 may be used to insert an end effector and to pass fluids to and from the patient. The electromagnetic (EM) field generator 60 interacts with one or more EM sensors 190 on the bendable medical device 110, which track the position, shape and / or orientation of the bendable medical device 110 during insertion through the body cavity 81 towards a target site 82 within the patient 80.

[0039] The steerable medical system 1000 includes a computer system 400 (e.g., a system console), a robotic actuator system 300, and a steerable medical system 100 connected to the actuator system 300 via a handle 200. The steerable medical system 100 includes a bendable medical instrument 110 (also called a steerable sheath) consisting of a proximal section 103, a middle section 102, and a distal section 101, which are arranged in that order along a longitudinal axis (Ax). The proximal section 103 is a non-steerable section and serves to connect the steerable section to the handle 200 and the actuation system. The middle section 102 and the distal section 101 constitute the steerable sections of the bendable medical instrument and are configured to be inserted into a body cavity 81 of a patient 80. The steerable distal section 101 (and intermediate section 102) is divided into a number of curved segments 1, 2, 3...N, which are configured to bend, curve, twist, and / or rotate when advancing the bendable medical device through a tortuous path within the lumen of a body cavity. Each curved segment includes at least one annular component. By convention, the steerable medical system 100 operates in 3D space defined by a three-dimensional (3D) coordinate system of x, y, z Cartesian coordinates. The bendable medical device 110 defines at least one tool channel 105 extending from the proximal end to the distal end along a longitudinal axis Ax. The bendable medical device 110 may include one or more position and / or orientation sensors 190 disposed in the wall of the catheter sheath, and may include a removable imaging device 180, such as a fiber camera or a miniature electronic CMOS sensor, disposed in the tool channel 105. The imaging device 180 is positioned such that its imaging surface lies in the xy plane and the longitudinal axis Ax of the bendable medical device 110 extends along the z-axis of the coordinate system.

[0040] Examples of bendable medical devices 110 and methods of using the medical devices via medical systems 100 are described in U.S. Patent Application Publication No. 2019 / 0105468, which is incorporated herein by reference in its entirety. Other examples of bendable medical devices and methods of using the medical devices via medical systems are disclosed in U.S. Patent Application Publication Nos. 2018 / 0243900, 2018 / 0311006, 2019 / 0105468, 2019 / 0015978, and 2019 / 0105468, as well as International Publication Nos. WO2018 / 204202, WO2020 / 086749, and WO / 2020 / 092096, all of which are incorporated herein by reference in their entirety.

[0041] During endoscope insertion into a biological lumen 81 (such as the airway of a patient 80), the tip (distal end) of the bendable medical device 110 is advanced (navigated) along the centerline of the lumen. In this case, an imaging device 180 (e.g., a miniature camera) can be placed in the tool channel 105 to provide a live view image of the lumen 81 taken directly from the instrument's field of view (FOV). However, in some embodiments, the bendable medical device 110 may not have room to place a camera in the tool channel. In this case, navigation can be provided by intraoperative guided imaging based on the position and / or orientation provided by one or more sensors 190 placed along the sheath. In either case, to reach the desired target site 82, the bendable medical device 110 must bend, twist, and / or rotate in various directions such that the distal section of the bendable medical device continuously changes shape and direction until it reaches an optimal position aligned with the target site 82 (such as a tumor).

[0042] The bending, twisting and / or rotation (steering) of the bendable medical device 110 is controlled by a system consisting of a handle 200, an actuator system 300 and / or a computer system 400. The actuator system 300 includes a microcontroller 320 and Drive unit310, which are operatively connected to a computer system 400 via a network connection 425. The computer system 400 includes appropriate software, firmware, and peripheral hardware operated by a processor or CPU 410. The computer system 400, the actuator system 300, and the handle 200 are operatively connected to each other by a network connection 425 (e.g., a cable bundle or a wireless link). Additionally, the computer system 400, the actuator system 300, and the handle 200 are operatively connected to each other by a robotic platform 90, which may include one or more robotic arms 92 and translation stages 91, and may be integrated into the drive 310. In some embodiments, the actuator system 300 may include or be connected to a handheld controller (such as a gamepad controller) or a portable computing device (such as a smartphone or tablet). Among other functions, the computer system 400 and actuator system 300 can provide a graphical user interface (GUI) and patient information displayed on the display screen 420 to a surgeon or other operator to operate the steerable medical system 100 according to its application.

[0043] FIG. 4 illustrates a bendable medical device having three sections (101, 102, 103). end is the most distal portion of the distal section 101. The bendable medical device is placed in a body cavity, which may be a lung 120. As illustrated by the lung 120, at each bifurcation the airway may narrow, causing the bendable medical device to not fit into the airway.

[0044] 5 shows an example endoscopic view of a patient's airway anatomy 800. This is a rough image and does not provide information on how the bendable medical device will fit into the airway or how it will navigate through it. This view can be used by a clinical user to navigate through the lungs. This view may be combined with CT or MRI imaging of the lungs and fluoroscopic imaging to aid in navigation of the medical procedure.

[0045] Thus, to achieve the objectives of the previous section, a medical system such as a robotic endoscopic system described herein provides a guided virtual bronchoscopy reference plane or guided reference plane that is displayed in a view with either a real-time bronchoscopy view or a traditional virtual bronchoscopy view. The guided virtual bronchoscopy reference plane is a cross-section based on the current tip position of the bendable medical device 110 and the orientation of the distal end of the bendable medical device. The guided reference plane is a cross-sectional view centered on the airway where all path options can be seen, instead of displaying a non-cross-sectional view where structures may be obscured and difficult to visualize and / or navigate.

[0046] FIG. 6(A) provides a display screen 420 including a representation of an airway structure 620 and a live endoscopic view 422 from an imaging device 180. The representation of the airway structure 620 may be an anatomically accurate representation of the patient's lungs. In some embodiments, it is a complete and anatomically accurate lung. In other embodiments, it includes at least a portion of the lung between the bendable medical device and the target site or location or nearby bifurcation. The representation of the airway structure 620 may be a segmented model of the airway, for example, constructed from a CT image. The segmentation may be manual, semi-automatic, or automatic, and may include machine learning algorithms. See, for example, Garcia-Ucede et al., “Automatic airway segmentation from Computed Tomography using robust and efficient 3-D convolutional neural networks,” Sci Rep 11, 16001 (2021).

[0047] air road Center line of 622 is calculated by the robotic catheter system 1000. A navigational path 624 is created that follows a centerline 622 from the trachea to the target 82 or a portion thereof. As shown, a centerline 622 for each of the airways is calculated from the airway structure 620. The centerlines 622 are then displayed for at least a portion of the airway structure. A navigational path 624 is plotted from the trachea or other point in the airway to the target 82 based on information from the user regarding the location of the target. The navigational path 624 may be defined as a portion of a group of airway centerlines that is a centerline that defines a path from a starting location to the target site 82 or a path from a starting location to a location in the airway proximate to the target site (if the target site is not in the airway). The navigational path 624 may be defined as a range of centerlines or may have been smoothed or otherwise manipulated.

[0048] Guidance reference plane 630 is generated and displayed. This view is perpendicular to the navigation path 624 and road Center of line 622 A cross-sectional view is created centered at . A guiding reference plane 630 is a cross section that indicates where the cross section will be taken. In this embodiment, a ring 631 is displayed on the guiding reference plane 630, which indicates the location of the distal tip of the bendable medical device 110. This ring 631 represents the location of the live endoscopic view.

[0049] The guiding reference plane 630 is displayed on the airway structure 620 as a semi-transparent rectangle. Other embodiments may provide other visualization methods. For example, instead of a square, a circle or an ellipse may be used. The guiding reference plane 630 may also be displayed with thickness to aid visualization. One side of the plane may be differentiated (e.g., by color or shape) to indicate the top of the bendable medical device defined in relation to the handle 200.

[0050] A target site 82 is a location or place within the anatomical structure where a user (e.g., a clinical user) intends to interact with, such as to take a biopsy, perform surgery, administer a therapeutic drug, etc. The target site is displayed on the representation of the airway structure. The location of the target site may be obtained, for example, by having the user specify a location on a display, or the location of the target site may be annotated on a pre-procedure image and the location obtained from this information. Alternatively, the target site may be determined by using an algorithm trained to find tumors or other regions of interest. In some embodiments, there are multiple target sites during the procedure and the clinical user will decide, for example, which target site should be accessed first.

[0051] Information from such calculations and / or other information (such as information about the distance from the bendable medical device to the target site, the insertion depth of the bendable medical device, the location of markers displayed on the airway structure, etc.) may be displayed in a patient information window 421. Other information that may be displayed includes the navigation modality used (such as a label indicating that the display is showing an endoscopic view), CT images, computer generated paths, etc.

[0052] The display 420 may include a guided virtual endoscopy view 426. The guided virtual endoscopy view 426 is a virtual endoscopy view from the viewpoint of the center of a guided reference plane 630. The direction of the view is the normal vector of the guided reference plane 630, which points toward the distal side of the steerable medical device 110. The guided virtual endoscopy view 426 includes a guided reference plane 630. surface When used in conjunction with the designation of the airway structure 620, the catheter tip 630 has the advantage of providing the user with a clear view of the next airway. 101 Unlike the normal endoscopic view, which shows a live image 422 from the catheter tip, the guided virtual endoscopic view 426 shows 101 The airway anatomy is always captured from the center of the airway at the insertion depth, even when the catheter is oriented in a direction that is not optimal for capturing the next airway. For example, the catheter tip 101If the catheter tip is pointing towards the wall of the airway, the user will not be able to see anything in the next airway other than the wall. However, the guided virtual endoscopy view 426 can provide a view of the next airway from the current insertion depth position, allowing the user to see the catheter tip. 101 It can assist in steering the robot in the optimal direction.

[0053] The information window 421 may additionally or alternatively provide a warning that the bendable medical device is approaching a force or bend angle limit (e.g., greater force or angle may cause damage or system malfunction). The warning may be, for example, a warning in a text box, a color change in the display of the area where damage or system malfunction may occur, etc. The force limit or bend angle limit may be set by the bendable medical device manufacturer or may be set by the user depending on the patient or procedure. In some embodiments, the controller may initiate a corrective action such as retracting the bendable medical device or loosening one or more tendons / wires of the bendable medical device.

[0054] FIG. 6(B) illustrates an embodiment with a display screen that is simpler in design than the display screen of FIG. 6(A). This display screen 420 shows a representation of the airway structure 620, but does not show the portion of the airway that is distant from the target site. In addition, a live endoscopic view 422 is displayed. The smaller portion of the airway is displayed, and the fewer views allow for a larger view of both the structure and the views compared to other embodiments. The representation of the airway structure 620 includes both a navigation path 624 and a guided virtual reference plane. In this embodiment, the centerline 622 and the target site 82 are not displayed, since they may overlap significantly with the navigation path 624 and are not necessary for navigation. Each of these elements can be included or excluded as desired. It is further contemplated that only a portion of the centerline 622 and / or the navigation path 624 may be displayed on the representation of the airway structure 620. Similarly, a live endoscopic view 422 is displayed. Alternatively, a guided virtual endoscopic view 426 may be displayed instead of the live view.

[0055] The guiding reference surface 630 is the catheter tip. 101 In FIG. 7A, the bendable medical device 110 has entered the airway, and the display screen 420 shows the catheter tip. 101 7(C) shows the guiding reference plane 630 at a position corresponding to the insertion depth position of the bendable medical device 110. For reference, FIG. 7(B) shows the position in the lung 120 where the bendable medical device 110 is placed. In FIG. 7(C), the bendable medical device 110 has advanced deeper into the lung compared to FIG. 7(A). This can be confirmed by the position of the bendable medical device 110 in the lung 120 in FIG. 7(D). Again, the corresponding guiding reference plane 630 shown in FIG. 7(C) has been lowered to the position shown. Also, note that the guiding reference plane 630 is always perpendicular to the navigation path. Thus, the position of the guiding reference plane 630 is a plane perpendicular to the navigation path, on which the current catheter tip is located. 101 can be determined as including

[0056] Guidance criteria surface In 630, more than just a momentary portion of the airway is displayed, providing the user with a broader view "roadmap" than the live image endoscopic view 422.

[0057] 620 Airway Structures of the Lungs expression , all possible airway direction options can be identified. Furthermore, the relative position of the guidance reference plane 630 allows easy confirmation of progression along the navigation path 624 to the target 82.

[0058] In another embodiment, the navigable medical system provides a navigation point at a bifurcation 626 on a navigation path 624 (see FIG. 8(A)). Thus, when a user identifies a navigation point 626 (e.g., by clicking on the display with a mouse), the navigation reference plane moves to that location in the airway and the guided virtual endoscopic view 426 is adjusted to correspond to the location of the selected navigation point 626. This is illustrated in FIG. 8(A). When a user clicks on the bifurcation point 626 marked with a green arrow in FIG. 8(A), the displayed navigation reference plane 630 moves to that location as shown by the navigation reference plane 630 in FIG. 8(B). The guided virtual endoscopic view of FIG. 8(B) 426 is similarly adjusted to correspond to the view at the navigation point selected by the user. However, in FIG. 8(A), the tip of the bendable medical device 110 is 101 and the actual position of the tip of the bendable medical device 110. 101 However, in the display of FIG. 8(B), the bendable medical device has not moved, so Catheter tip 101 and the corresponding live image endoscopic view 422 remains unchanged. This allows the user to preview what they will see during navigation as they move along the planned path of the airway, while also allowing them to see where the proposed navigation route is located within the airway. This also provides a means for the user to review what they have traversed during navigation.

[0059] The guided virtual endoscopy view 426 of FIG. 8(C) is in the same user-selected position as FIG. 8(B). To However, this display 420 includes both a guided virtual endoscopy view 426 located at a selected bifurcation / navigation point 626 and a second virtual endoscopy view 428 showing the location of the bronchoscope. is displayed. .

[0060] The guided virtual endoscopy view 426 in conjunction with the guidance reference plane 630 is orientation stabilized, so the user can orient themselves using the structures displayed in the oriented cross-sectional bronchoscopic view.

[0061] Displaying the navigation points at the bifurcations 626 allows the user to preview (or review) the anatomical structures along the navigation path. In some embodiments, during the preview or review step, an indicator (such as a marker or color change) is present when the guiding virtual endoscopy view 426 and / or the guiding reference plane 630 are displayed, such that the indicator indicates the location of the tip of the bendable medical device 110. 101 is in a position that does not correspond to the position of

[0062] In some embodiments, the virtual bronchoscopic cross-sectional view and the patient's airway structure 620 may be used for navigation. route 624 can be superimposed on the guided virtual endoscopy view 426 as shown by the arrow in FIG.

[0063] An XYZ indicator 432 is oriented relative to the tip of the catheter, as shown in Figure 10. This XYZ indicator 432 updates in real time to reflect changes in the orientation and / or rotation of the catheter tip. Guidance reference plane 630 has a different XYZ indicator 434 superimposed thereon. This XYZ indicator 434 reflects the changes in the XYZ indicator 432, but Guidance reference plane 630 coordinate system. This allows the user to Guidance reference plane The user can check the direction of the catheter tip by looking at 630. Guidance reference plane At 630, the motion inputs required to move in the desired direction can be made.

[0064] The system 100 may be regulated, controlled and / or directed by one or more processors in communication with a controller and any other components and / or subsystems of the overall system 1. The processor may operate based on computer readable program instructions stored in a non-transitory computer readable memory. The processor may be or include one or more of a CPU, MPU, GPU, ASIC, FPGA, DSP and general purpose computer. The processor may be a dedicated controller or a general purpose computing device configured to be a controller. Examples of non-transitory computer readable memory include RAM, ROM, CD, DVD, Blu-Ray, hard drives, network attached storage (NAS), non-transitory computer readable storage connected to an intranet, non-transitory computer readable storage connected to the Internet, etc.

[0065] The embodiments of the present disclosure may be implemented by a computer system 400 or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may more fully be referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the above-described embodiments, and / or by a computer system 400 or device that includes one or more circuits (e.g., application specific integrated circuits (ASICs)) for performing one or more functions of the above-described embodiments, or by a method performed by a computer system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform one or more functions of the above-described embodiments and / or by controlling one or more circuits for performing one or more functions of the above-described embodiments. The computer system may include one or more processors (e.g., a central processing unit (CPU) 410, a microprocessing unit (MPU)), and may include a separate computer or a network of separate processors for reading and executing the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), a Blu-ray Disc (BD)®, etc.), a flash memory device, a memory card, etc. The I / O interface may be used to provide a communication interface to input / output devices, including keyboards, displays, mice, touch screens, touchless interfaces (e.g., gesture recognition devices), printing devices, light pens, optical storage devices, scanners, microphones, cameras, drives, communication cables and networks (wired or wireless).

[0066] When referring to the description, specific details are set forth to provide a thorough understanding of the disclosed examples. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily lengthen the present disclosure. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. The breadth of the present invention is not limited by this specification, but rather only by the plain meaning of the claim terms employed.

[0067] In describing the exemplary embodiments shown in the drawings, specific terminology is used for clarity, however, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected, and it is understood that each specific element includes all technical equivalents that function in a similar manner.

[0068] Although the present disclosure has been described with reference to exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. For example, the present disclosure has been described above with reference to exemplary embodiments. However, there are many variations not specifically described to which the present disclosure may be applicable. For example, although various embodiments have been described with reference to an endoscope for use in a medical procedure, the present disclosure is also applicable with reference to a borescope mechanical procedure for use in various mechanical structures. Thus, the scope of the following claims should be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A display device; On the display device, an image of the biological lumen taken from a distal end of the bendable medical device within the biological lumen; Representation of airway structure, a control device configured to display A navigation system comprising: The representation of the airway structure comprises: a navigation path through at least a portion of the airway structure; a guidance reference surface oriented perpendicular to the navigation path and positioned at an insertion depth of the distal end of the bendable medical device within the biological lumen; Further comprising: Navigation system.

2. The representation of the airway structure comprises: a centerline, a target site, or both the centerline and the target site are displayed on the representation of the airway structure; The navigation system of claim 1 , comprising:

3. The control device is further configured to display a guided virtual endoscopy view, the guided virtual endoscopy view having an image center at the centerline of the guided reference plane and having a distal view orientation along an orientation of the guided reference plane. The navigation system of claim 1 .

4. The control device is further configured to display a second virtual endoscopic view having an image center at the distal end of the bendable medical device.

4. The navigation system according to claim 3.

5. The control device may include: Navigation modality; a distance from the bendable medical device to a target site; and an insertion depth of the bendable medical device; information regarding the location of markers displayed on the airway structure; a warning that the bendable medical device is approaching a force or bend angle limit; and further configured to display navigation information including at least one of: The navigation system of claim 1 .

6. the controller is configured to initiate a corrective action when the bendable medical device approaches a force or bend angle limit. The navigation system of claim 1 .

7. the controller is configured to update the representation of the guiding reference surface as the bendable medical device moves through the airway structure. The navigation system of claim 1 .

8. The guiding reference plane is displayed so as to have a three-dimensional perspective. The navigation system of claim 1 .

9. the representation of the airway structure is a model of the patient's airway based on one or more computed tomography (CT) scanner data and / or magnetic resonance imaging (MRI) scanner data; The navigation system of claim 1 .

10. the representation of the airway structure further comprises one or more markers indicating one or more branching points of the airway structure. The navigation system of claim 1 .

11. The guiding reference surface is instead located at the bifurcation point of the airway structure and is a rough guiding reference surface. The navigation system of claim 1 .

12. the branching points of the airway structure are selected based on user input, the user input including selecting a marker located on the representation of the airway structure indicative of one or more branching points of the airway structure. The navigation system according to claim 11.

13. A bendable medical device; an actuator system for actuating the bendable medical device; The navigation system according to claim 1 ; A medical system comprising:

14. The control device is further configured to display a guided virtual endoscopy view, the guided virtual endoscopy view having an image center at the centerline of the guided reference plane and having a distal view orientation along an orientation of the guided reference plane. The medical system of claim 13.

15. 1. A method for controlling a display, comprising: acquiring an image of a biological lumen from a distal end of the bendable medical device; obtaining a representation of an airway structure; obtaining a target site; generating a centerline of at least a portion of the airway structure; generating a navigation path to the target site through at least a portion of the airway structure; displaying said representation of the airway structure on a display device; Including, The representation of the airway structure may include: a navigation path through at least a portion of the airway structure; a guidance reference surface oriented perpendicular to the navigation path and positioned at an insertion depth of the distal end of the bendable medical device within the biological lumen; Further comprising: method.

16. displaying the image of the biological lumen on the display device; The method of claim 15 further comprising:

17. The representation of the airway structure is obtained from a preoperative CT image. The method of claim 15.

18. The target site is obtained from a user. The method of claim 15.

Images