Improved navigation in the bronchoscopy graphical user interface

Abstract

A navigation system, medical system, method of use, and medium for use in navigating a bendable medical device are provided. The system and method include a display device and a control device. The control device is configured to display, on the display device, an image of a biological lumen taken from a distal end of the bendable medical device and a representation of an airway structure. The representation of the airway structure includes a navigation path through at least a portion of the airway and a guidance reference surface disposed perpendicular to the navigation path and located at an insertion depth of the distal end of the bendable medical device.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 307,878, filed on February 8, 2022. The disclosure of the above provisional application is hereby incorporated by reference in its entirety for all purposes. The benefit of priority is claimed under 35 U.S.C. § 119(e).

[0002] The present disclosure generally relates to systems and methods for medical use. More particularly, the present disclosure is directed to systems that use multi - joint medical devices and to the display of information from such medical devices. The medical device is capable of being maneuvered within a patient.

Background Art

[0003] In the medical field, bendable medical devices such as endoscope surgical devices and catheters are well - known and continue to be accepted. Bendable medical devices generally include a flexible body, commonly referred to as a sleeve or sheath. One or more tool channels extend along (typically, inside) the flexible body, enabling access to a target site located at the distal end of the flexible body.

[0004] Since the medical device is designed to provide flexible access within a patient with one or more curves leading to a target, while maintaining torsional rigidity and longitudinal stiffness, a clinical user can control a tool located at the distal end of the medical device by manipulating by manipulating the proximal end of the device.

[0005] Medical devices may be implemented via systems. These systems include both hardware and software, which, when used together, allow users to guide and observe the movement of medical devices through passages within a patient. For example, U.S. Patent Publication 2019 / 0105468 describes a system for implementing a multi-joint medical device with a cavity. This device is maneuverable within a patient and allows for the guidance of medical tools such as endoscopes, cameras, and catheters through the cavity for medical procedures.

[0006] To assist in navigating medical devices toward a target site, physicians may use preoperative and / or intraoperative imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), and other similar techniques to provide a 'roadmap' for navigating surgical instruments within or around the patient's internal structures and organs. Navigation through the lungs presents complex challenges, as the lung's structure incidentally provides multiple branching pathways to select the best route to the target, while the continuous branching of the airways can make it difficult to determine which airway the surgical tool is in relative to the target. Images of the entire lung structure may also be provided to aid navigation. However, for a user to maneuver a catheter toward a target, they need to understand the position of the medical device within the entire lung structure, its progress along the planned navigation path, and the possible airways available for orientation.

[0007] The difficulty lies in the fact that the current state of technology's "roadmap" falls into one of two categories: current-view models that only provide the airway as it is currently 'seen' by medical devices (see, e.g., U.S. Patent No. 9,727,963), and 2D representation models that do not provide airway options or structural details (see, e.g., U.S. Patent Publication No. 2020 / 0054399). Similarly inadequate teachings are provided in U.S. Patent No. 10,617,324, U.S. Patent No. 8,337,397, U.S. Patent Publication No. 2020 / 0054399, U.S. Patent Publication No. 2019 / 0254649, and International Publication No. 2018 / 00586.

[0008] Therefore, improved displays and systems are needed to assist in the navigation and use of flexible medical devices in the lungs. Improved displays would help visualize the position of the tip of the flexible medical device within the airway structure, visualize its location along the navigation path, and provide virtual endoscopic views for checking available options and oriented the live endoscopic view. [Overview of the project]

[0009] According to at least one embodiment of the present invention, a navigation system is provided comprising a display device and a control device. The control device is configured to display on the display device an image of a biological lumen taken from the distal end of a flexible medical device, and a representation of the airway structure. The representation of the airway structure (which may be, for example, a segmented model of the airway obtained from a CT image) includes on the model image: a navigation path through at least a portion of the airway structure; and a guide reference plane oriented perpendicular to the navigation path and positioned at the insertion depth of the distal end of the flexible medical device in the biological lumen. The representation of the airway structure may also include a centerline, a target site, and a ring or other features indicating the position of the endoscope tip.

[0010] The guiding reference plane is optionally updated as the flexible medical device moves through the airway structure. The control device may be configured to display a guiding virtual endoscopic view, which has an image center on the centerline of the guiding reference plane and a distal view orientation along the orientation of the guiding reference plane. Alternatively or additionally, the control device may be further configured to display a second virtual endoscopic view with an image center at the distal end of the flexible medical device.

[0011] The control device may also be configured to display at least one of the following: navigation modality; distance from the flexible medical device to the target site; insertion depth of the flexible medical device; information regarding the location of markers displayed on the airway structure; and a warning that the flexible medical device is approaching a limit of force or bending angle.

[0012] The navigation systems described herein may also display one or more markers indicating one or more branching points of the airway structure. These are selectable, and the guidance reference plane can be moved to a selected branching point of the airway structure to become a schematic guidance reference plane.

[0013] Also provided are a method and a computer-readable storage medium for performing the steps of the method. For example, a method is provided which includes the following steps: acquiring an image of a biological lumen from the distal end of a flexible medical device; acquiring a representation of an airway structure; acquiring a target site; generating a centerline of at least a portion of the airway structure; generating a navigation path through at least a portion of the airway structure; and displaying the representation of the airway structure on a display device. The representation of the airway structure includes a navigation path and a guide reference plane positioned perpendicular to the navigation path and at the insertion depth of the distal end of the flexible medical device.

[0014] These and other purposes, features and advantages of this disclosure will become apparent when the following detailed description of exemplary embodiments of this disclosure is read in conjunction with the accompanying drawings and the provided claims. [Brief explanation of the drawing]

[0015] Further objects, features, and advantages of the present invention will become apparent from the following detailed description, in conjunction with the accompanying drawings illustrating exemplary embodiments of the invention.

[0016] [Figure 1] Figure 1 illustrates an exemplary embodiment of the robot-assisted endoscopy system 1000 in a medical environment such as an operating room. [Figure 2] Figure 2 illustrates an exemplary embodiment of a system that allows a user to guide and observe the movement of medical devices within a patient. [Figure 3] Figure 3 illustrates an exemplary embodiment of the controllable medical system 1000 as represented by a functional block diagram. [Figure 4] Figure 4 illustrates the lung along with the route for inserting the endoscope. [Figure 5] Figure 5 illustrates an endoscopic view of the airway structure. [Figure 6] Figures 6(A) and 6(B) provide illustrative displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 7] Figures 7(A) and 7(C) provide a virtual endoscopic view, a live endoscopic view, and other illustrative displays showing a representation of the lung. Figures 7(B) and 7(D) provide the actual catheter position within the lung with respect to the images in Figures 7(A) and 7(C), respectively. [Figure 8-1] Figures 8(A) and 8(B) provide other illustrative displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 8-2] Figure 8(C) provides a virtual endoscopic view, a live endoscopic view, and other illustrative displays showing a representation of the lungs. [Figure 9] Figure 9 provides illustrative displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs. [Figure 10] Figure 10 provides illustrative displays showing a virtual endoscopic view, a live endoscopic view, and a representation of the lungs.

[0017] Throughout the figures, unless otherwise noted, the same reference numerals and letters are used to indicate similar features, elements, components, or parts of the exemplary embodiments. Further, the present disclosure will be described in detail with reference to the figures, but this will be done in relation to the exemplary embodiments. It is intended that changes and modifications may be made to the embodiments described without departing from the true scope and spirit of the present disclosure as defined by the accompanying paragraphs. [Modes for carrying out the invention]

[0018] The following paragraphs describe specific descriptive embodiments. Other embodiments may include alternatives, equivalents, and modifications. In addition, descriptive embodiments may include some novel features, and certain features may not be essential to some embodiments of the apparatus, systems, and methods described herein.

[0019] It is a matter of course for those skilled in the art that, generally, the terms used in this specification, particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be construed as "including but not limited to", the term "having" should be construed as "having at least", the term "includes" should be construed as "includes but not limited to", etc.). Further, it is a matter of course for those skilled in the art that if a specific number introduced in the claim description is intended, such intention is explicitly stated in the claims, and if there is no such statement, such intention does not exist. For example, for the sake of understanding, the following appended claims may include the use of introductory phrases such as "at least one" or "one or more" to introduce the claim description. However, the use of such phrases should not be construed to mean that a particular claim including such an introduced claim description is limited to a claim that includes only one such description even if the same claim includes both an introductory phrase such as "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should typically be construed to mean "at least one" or "one or more"). The same can be said for the use of definite articles used to introduce claim descriptions.

[0020] As used herein, when a feature or element is referred to as being "above" another feature or element, it may be directly above the other feature or element, or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly above" another feature or element, no intervening features or elements are present. Also, of course, when a feature or element is referred to as being "connected to", "attached to", "coupled to", etc. another feature or element, it may be directly connected, attached, or coupled to the other feature, or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected to", "directly attached to", or "directly coupled to" another feature or element, no intervening features or elements are present. Although described or illustrated with respect to one embodiment, the features and elements so described or illustrated in one embodiment can be applied to other embodiments. Also, as would be understood by one of ordinary skill in the art, a reference to a structure or feature being "adjacent" to another feature may include portions that overlap or are beneath the adjacent feature.

[0021] In this specification, terms such as first, second, third, etc. may be used to describe various elements, components, regions, parts, and / or portions. Of course, these elements, components, regions, parts, and / or portions are not limited by these designating terms. These designating terms are used only to distinguish one element, component, region, part, or portion from another. Thus, a first element, component, region, part, or portion described hereinafter can be referred to as a second element, component, region, part, or portion for the sole purpose of distinction, but without limitation and without departing from its structural or functional meaning.

[0022] Where used herein, the singular form is intended to include the plural form unless explicitly indicated otherwise in the context. Furthermore, naturally, the terms “contains,” “comprising,” and “consisting of,” where used herein and in the claims, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof that are not expressly described. Furthermore, in this disclosure, the transitional phrase “consisting of” excludes any elements, steps, or components not specified in the claims. Furthermore, it should be noted that some claims or some features of a claim may be drafted to exclude any element, and such claims may use exclusive terms such as “alone,” “only,” or “negative” limitations in relation to the description of the claim elements.

[0023] Unless otherwise specified, as will be apparent from the following disclosure, any use of terms such as “process,” “calculate,” “determine,” and “display” throughout this disclosure is understood to refer to the operations and processes of a processor, such as a computer system, or similar electronic computing equipment, or data processing equipment that manipulates data represented as physical (electronic) quantities in the registers and memory of a computer system and converts it into other data represented as physical quantities in the memory or registers of a computer system or other equipment for storing, transmitting, or displaying such information. The computational or electronic operations described herein or in the appended claims may generally be performed in any order unless otherwise indicated in the context. While various operation flowcharts are presented in sequence, it is naturally possible to perform the operations in an order other than that shown or claimed, or to perform the operations simultaneously. Examples of such alternative orderings, unless otherwise indicated in the context, include overlapping, interleaving, interrupting, reordering, incremental, preparing, supplementing, simultaneous, reverse, or other variations of ordering. Furthermore, terms such as "in response to," "in reply to," "in relation to," and "based on," or other similar past adjectives, are not usually intended to exclude such variations unless otherwise indicated by the context.

[0024] As is well known in the field of medical devices, the terms “proximal” and “distal” are used in relation to the operation of the end of an instrument that extends from the user to the surgical or diagnostic site. In this regard, “proximal” refers to the part of the instrument closer to the user (e.g., the handle), and “distal” refers to the part of the instrument that is further away from the user and closer to the surgical or diagnostic site (the tip). Furthermore, naturally, for the sake of brevity and convenience, spatial terms such as “perpendicular,” “parallel,” “up,” and “down” may be used in reference to drawings in this specification. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0025] As used herein, the term “flexible medical device” generally refers to a flexible, slender tubular instrument made from medical-grade material, designed to be inserted into a body cavity (e.g., a blood vessel or bronchi) through a narrow opening to perform a wide range of medical functions. A catheter may be a flexible medical device. A more specific term, “optical catheter,” refers to a flexible medical device that includes an elongated bundle of one or more flexible optical fibers with optical imaging capabilities, housed within a protective sheath made from medical-grade polymer material. A specific example of an optical catheter is a fiber optic catheter comprising a flexible sheath, a coil, and an optical probe or imaging core housed within the coil. In some applications, a catheter may include a “guide catheter” that functions similarly to a sheath. Flexible medical devices may be configured to be used in conjunction with one or more tools. For example, a camera may be inserted into a flexible medical device, or a camera or other optical probe may be an essential part of the device. Biopsy instruments may also be used in conjunction with flexible medical devices.

[0026] As used herein, the term “endoscope” refers to a rigid or flexible medical instrument used to observe the inside of a body cavity or organ using light guided by an optical probe. The medical procedure in which an endoscope is inserted through a natural orifice is called an endoscopy. Specialized endoscopes are generally named after the method or location of use, such as bronchoscopes (mouth and lungs), sigmoidoscopy (rectum), cystoscopes (bladder), nephroscopes (kidneys), bronchoscopes (bronchi), pharyngoscopes (pharynx), otoscopes (ears), arthroscopes (joints), laparoscopes (abdomen), and gastrointestinal endoscopes. Embodiments of this disclosure may be applicable to one or more of the aforementioned endoscopes.

[0027] This disclosure generally relates to medical devices and illustrates embodiments of optical probes applicable to imaging devices (e.g., endoscopes). Embodiments of optical probes and parts thereof are described with respect to their states in three-dimensional space. As used herein, the term “position” refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates); the term “orientation” refers to the rotational orientation of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, yaw); the term “posture” refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six degrees of freedom in total); and the term “shape” refers to a set of postures, positions and / or orientations measured along the elongated body of an object.

[0028] An example configuration of a medical system 1000, such as a robot-assisted endoscopy system, will be described with reference to Figure 1. This figure illustrates an example of a medical environment, such as an operating room, in which the robot-assisted endoscopy system 1000 may be put into practical use. The robot-assisted endoscopy system 1000 may include a controllable instrument 100 (controllable medical device) that can be operated by a user 10 (e.g., a physician) when performing an endoscopic examination procedure on a patient 80. The robot-assisted endoscopy system 1000 may include a computer system 400 that is operationally attached to the controllable instrument 100 via a robot platform 90. The computer system 400 (e.g., a system console) includes a processor or central processing unit (CPU) 410 and a display screen 420 such as a liquid crystal display (LCD), OLED, or QLED display. A storage device 411 (ROM and RAM memory), a system interface 412 (FPGA card), and a user interface 413 (e.g., a mouse or keyboard) are operationally connected to the processor or CPU 410 and the display screen 420.

[0029] The controllable instrument 100 includes a handle 200 and a bendable medical device (e.g., a controllable sheath) 110, which are detachably connected to each other by a connector assembly 50. The handle 200 includes an actuator system 300, which receives electronic commands from a computer system 400 to mechanically actuate the bendable medical device 110. The handle 200 is configured to be detachably mounted on a robot platform 90. The robot platform 90 has a robotic arm 92 and a stage 91 for robotically guiding the bendable medical device 110 toward a target site 82 in a subject or patient 80. If the handle 200 is not necessarily mounted on the robot platform 90, the handle 200 can be manually operated by a user 10 to control the bendable medical device 110. For treatment or examination of the patient 80, the controllable instrument 100 may include one or more access ports 250 located on or around the handle 200. The access port 250 can be used to insert an end effector, to deliver fluid to the patient, or to deliver fluid from the patient. The electromagnetic (EM) field generator 60 interacts with one or more EM sensors 190 positioned on the maneuverable sheath 110, which track the position, shape, and / or orientation of the maneuverable sheath 110 while it is being inserted through the body cavity 81 toward a target site 82 in the patient 80. The medical device 110 may include a tool channel for biopsy or other interventional tools. A clinical user can insert and withdraw the medical device 110, for example, to perform a biopsy in the patient's airway.

[0030] During an endoscopic procedure, the system processor or CPU 410 of the computer system 400 is configured to perform operations based on computer executable code pre-stored in the system's memory 411. The display screen 420 may include a graphical user interface (GUI) configured to display one or more of the following: patient information in an information window 421, endoscopic live images 422, intraoperative images 423 (e.g., fluoroscopic images), and preoperative images 424 of the patient 80 (e.g., slice images).

[0031] Figure 2 illustrates an exemplary embodiment of the medical system 1000. The medical system 1000 (also referred to herein as a continuum robotic system) comprises a drive unit 310, a bendable medical device 110 (or sheath), a positioning cart 500, an operating console 600, and navigation software 700. The system 100 also interacts with clinical users and external systems (e.g., computed tomography (CT) scanners and / or magnetic resonance imaging (MRI) scanners, fluoroscopy equipment, patients, biopsy tools).

[0032] The navigation software 700 and the drive unit 310 are communicatively coupled via a bus that transmits data between them. Furthermore, the navigation software 700 may be coupled to and communicate with a CT scanner, MRI scanner, X-ray fluoroscopy device, and image server (not shown in Figure 2) located outside the medical system 1000. The image server may be, for example, a DISCOM server coupled to medical imaging equipment such as a CT scanner, MRI scanner, and X-ray fluoroscopy device. The navigation software 700 processes data provided by the drive unit 310 and data provided by images stored in the image server, images from the CT scanner / MRI scanner, and images from the X-ray fluoroscopy device in order to display images on the display device.

[0033] Images from a CT scanner / MRI scanner are provided to the navigation software 700 preoperatively. Using the navigation software 700, a clinical user can create an anatomical computer model from the images. In some embodiments, the anatomical structures are biological lumens such as the lung airways. From chest images from a CT scanner / MRI scanner, the clinical user can segment the lung airways for clinical use. Thus, a lung-airway map can be created from this data, and this lung-airway map can be used to create a planned route. With or without this route, the data can also be used to guide and provide information for procedures such as biopsies using a flexible medical device 110 inserted into a biological lumen. In other embodiments, when the navigation software 700 creates (or adds to) an anatomical computer model of a biological lumen, images from, for example, intraoperative fluoroscopy may be used.

[0034] The drive unit 310 has an actuator and a control circuit configuration. The control circuit configuration is communicatively coupled to the operation console 600. The drive unit 310 is also connected to the flexible medical device 110 so that the actuator of the drive unit 310 operates the medical device 110. Therefore, the clinical user can operate the drive unit 310 The medical device 110 can be controlled via the drive unit 310. The drive unit 310 is also physically connected to the positioning cart 500. The positioning cart 500 may include one or more positioning arms and a translation stage, and the positioning cart 500 positions the drive unit 310 and the medical device 110 in the intended position relative to the patient.

[0035] The operation console 600 most preferably includes one or more displays and input devices such as a mouse, joystick, touchscreen, or voice activation.

[0036] The medical device may be equipped with a camera at its distal tip (i.e., a 'tip-on-tip' design). Alternatively, the medical device may be equipped with imaging means for generating an image of the region at the distal tip. For example, the image can be generated via a conventional CCD endoscope, borescope, fiberscope, or spectral coding endoscopy (see, for example, U.S. Patents 7,551,293, 9,295,391, 10,288,868, and 10,401,610). The medical device forms an image at the distal end of its tip. This image can be used for navigation of the flexible medical device. Such images of the inside of biological lumens (e.g., lungs) or other hollow organs (e.g., renal pelvis) can be combined with CT, MRI, fluoroscopy, or other images of the region of interest to assist in guiding the medical device toward a target site.

[0037] The medical device 110 may include a tool channel for biopsy or other interventional tools. Thus, the medical device 110 can guide a biopsy tool to the patient's lesion. A clinical user can then use the biopsy tool to collect a biopsy sample from the lesion.

[0038] Figure 3 illustrates the overall structure of the controllable medical system 1000 of Figure 1 in a functional block diagram that does not include the user and / or patient. The medical system 1000 includes a handle 200 and a bendable medical device 110, which are detachably connected to each other by a connector assembly 50. The handle 200 includes an actuator system 300, which is part of a drive unit 310, and which receives electronic commands from a computer system 400 to mechanically actuate the bendable medical device 110. The handle 200 is configured to be detachably mounted on a robot platform 90, which may be part of a positioning cart 500. The robot platform 90 has a robot arm 92 and a stage 91 for robotically guiding the bendable medical device 110 toward a target site 82 in the object or patient 80. When the handle 200 is not mounted on the robot platform 90, the handle 200 can be manually operated by the user 10 when controlling the bendable medical device 110. For the treatment or examination of patient 80, the maneuverable medical system 1000 may include one or more access ports 250 located on or around the handle 200. The access ports 250 can be used to insert an end effector, to deliver fluid to the patient, or to deliver fluid from the patient. An electromagnetic (EM) field generator 60 interacts with one or more EM sensors 190 provided on the bendable medical device 110, which track the position, shape, and / or orientation of the bendable medical device 110 while it is inserted through a body cavity 81 toward a target site 82 in patient 80.

[0039] The controllable medical system 1000 includes a computer system 400 (e.g., a system console), a robotic actuator system 300, and a controllable medical system 100 connected to the actuator system 300 via a handle 200. The controllable medical system 100 includes a bendable medical device 110 (also called a controllable sheath) consisting of a proximal section 103, an intermediate section 102, and a distal section 101, the sections arranged in this order along the longitudinal axis (Ax). The proximal section 103 is a non-controllable section and functions to connect the controllable section to the handle 200 and the actuator system. The intermediate section 102 and the distal section 101 constitute the controllable section of the bendable medical device and are configured to be inserted into a body cavity 81 of a patient 80. The maneuverable distal section 101 (and intermediate section 102) is divided into a plurality of curved segments 1, 2, 3…N, which are configured to bend, curve, twist, and / or rotate as the maneuverable medical device is advanced through a meandering path within the lumen of the body cavity. Each curved segment includes at least one annular component. By convention, the maneuverable medical system 100 operates in 3D space defined by a three-dimensional (3D) coordinate system of x,y,z Cartesian coordinates. The maneuverable medical device 110 defines at least one tool channel 105 extending from the proximal end to the distal end along the longitudinal axis Ax. The maneuverable medical device 110 may include one or more position sensors and / or orientation sensors 190 located on the wall of the catheter sheath, and may also include a removable imaging device 180, such as a fiber camera or a miniature electronic CMOS sensor, located in the tool channel 105. The imaging device 180 is positioned such that its imaging plane is in the xy plane, and the longitudinal axis Ax of the flexible medical device 110 extends along the z axis of the coordinate system.

[0040] Examples of the flexible medical device 110 and methods of using the medical device via the medical system 100 are described in U.S. Patent Application Publication 2019 / 0105468, which is incorporated herein by reference in its entirety. Other examples of the flexible medical device and methods of using the medical device via the medical system are disclosed in U.S. Patent Application Publications 2018 / 0243900, 2018 / 0311006, 2019 / 0105468, 2019 / 0015978 and 2019 / 0105468, as well as in International Publications WO2018 / 204202, WO2020 / 086749 and WO / 2020 / 092096, all of which are incorporated herein by reference in their entirety.

[0041] When inserting an endoscope into a biological lumen 81 (such as the patient's airway 80), the tip (distal end) of the flexible medical instrument 110 advances (is navigated) along the centerline of the lumen. In this case, an imaging device 180 (e.g., a small camera) can be placed within the tool channel 105 to provide a live view image of the lumen 81 taken directly from the instrument's field of view (FOV). However, in some embodiments, the flexible medical instrument 110 may not have room to place a camera within the tool channel. In this case, navigation can be provided by intraoperative guided imaging based on position and / or orientation provided by one or more sensors 190 positioned along the sheath. In any case, in order to reach a desired target site 82, the flexible medical instrument 110 must bend, twist, and / or rotate in various directions so that the distal section of the flexible medical instrument continuously changes shape and orientation until it reaches an optimal position aligned with the target site 82 (such as a tumor).

[0042] The bending, twisting, and / or rotation (maneuvering) of the bendable medical device 110 is controlled by a system consisting of a handle 200, an actuator system 300, and / or a computer system 400. The actuator system 300 is controlled by a microcontroller 320 and Drive unitThe actuator system 300 and the handle 200 are operationally connected to the computer system 400 via a network connection 425. The computer system 400 includes appropriate software, firmware, and peripheral hardware operated by a processor or CPU 410. The computer system 400, the actuator system 300, and the handle 200 are operationally connected to each other by a network connection 425 (e.g., a cable bundle or a wireless link). Furthermore, the computer system 400, the actuator system 300, and the handle 200 are operationally connected to each other by a robot platform 90, which may include one or more robot arms 92 and translational stages 91, and may also be incorporated into the drive unit 310. In some embodiments, the actuator system 300 may include, or be connected to, a handheld controller (such as a gamepad controller) or a portable computing device (such as a smartphone or tablet). Among other functions, the computer system 400 and the actuator system 300 can provide surgeons and other operators with a graphical user interface (GUI) and patient information displayed on the display screen 420 in order to operate the operable medical system 100 according to its intended use.

[0043] Figure 4 illustrates a bendable medical device having three sections (101, 102, 103). end This is the most distal part of the distal section 101. The flexible medical device is placed in a body cavity, which may be the lung 120. As shown by the lung 120, the airway may narrow at each bifurcation, so the flexible medical device may no longer fit into the airway.

[0044] Figure 5 shows an illustrative endoscopic view of a patient's airway structure. This is a rough image and does not provide information on how the flexible medical device conforms to the airway or how it moves through the airway. This view can be used by clinical users when navigating within the lung. This view may be combined with CT or MRI imaging or fluoroscopic imaging of the lung to assist in the navigation of the medical procedure.

[0045] Therefore, in order to achieve the objectives of the preceding sections, the medical systems, such as the robot-assisted endoscopy system described herein, provide a guided virtual bronchoscopic reference plane or guided reference plane displayed in a view that includes either a real-time bronchoscopic view or a conventional virtual bronchoscopic view. The guided virtual bronchoscopic reference plane is a cross-section based on the current tip position of the bendable medical device 110 and the orientation of the distal end of the bendable medical device. The guided reference plane is a cross-sectional view positioned at the center of the airway, and in the guided reference plane, all path options can be seen instead of a non-cross-sectional view which would obscure the structure and make visualization and / or navigation difficult.

[0046] Figure 6(A) provides a display screen 420 including a representation of the airway structure 620 and a live endoscopic view 422 from the imaging device 180. The representation of the airway structure 620 may be an anatomically accurate representation of the patient's lung. In some embodiments, it is a complete and anatomically accurate lung. In other embodiments, it includes at least a portion of the lung between a bendable medical device and a target site or location or nearby bifurcation. The representation of the airway structure 620 may be, for example, a segmented model of the airway composed of CT images. Segmentation may be manual, semi-automatic, or automatic and may include machine learning algorithms. See, for example, Garcia-Ucede et al., “Automatic airway segmentation from Computed Tomography using robust and efficient 3-D convolutional neural networks” (Sci Rep 11, 16001 (2021)).

[0047] air road Centerline 622 This is calculated by the robotic catheter system 1000. A navigation path 624 is created that follows the centerline 622 from the trachea to the target 82 or a portion thereof. As shown, each centerline 622 of the airway is calculated from the airway structure 620. Next, the centerlines 622 are displayed for at least a portion of the airway structure. Based on user information regarding the target's location, the navigation path 624 is plotted from the trachea or other points in the airway to the target 82. The navigation path 624 can be defined as a subset of the group of airway centerlines that define the path from the starting point to the target site 82, or the path from the starting point to a location in the airway close to the target site (if the target site is not in the airway). The navigation path 624 may be defined as various centerlines, or it may have been smoothed or otherwise manipulated.

[0048] Guidance reference plane 630 is generated and displayed. This view is perpendicular to the navigation path 624, and road Center line 622 A cross-sectional view is created centered on the guiding reference plane 630. The guiding reference plane 630 is a cross-section that indicates the position from which the cross-section is taken. In this embodiment, a ring 631 indicating the position of the distal tip of the flexible medical device 110 is displayed on the guiding reference plane 630. This ring 631 represents the position of the live endoscope view.

[0049] The guide reference plane 630 is displayed on the airway structure 620 as a translucent rectangle. Other embodiments may provide other visualization methods. For example, a circle or ellipse can be used instead of a square. The guide reference plane 630 may also be displayed with thickness to aid visualization. One side of the plane may be distinguished (e.g., by color or shape) to indicate the upper part of a bendable medical device as defined in relation to the handle 200.

[0050] The target site 82 is a location or position within an anatomical structure where the user (e.g., a clinical user) intends to interact with the system, such as by taking a biopsy sample, performing surgery, or administering a drug. This target site is displayed on a representation of the airway structure. The location of the target site can be obtained, for example, by having the user specify the location on the display, or the location of the target site may be annotated on a pre-procedure image, from which the location is obtained. Alternatively, the target site may be determined using an algorithm trained to find tumors or other areas of interest. In some embodiments, there may be multiple target sites during the procedure, and the clinical user will decide, for example, which target site to access first.

[0051] Such calculations and / or information from other sources (such as the distance from the flexible medical device to the target site, the insertion depth of the flexible medical device, and the location of markers displayed on the airway structure) may be displayed in the patient information window 421. Other information that may be displayed includes the navigation modality being used (such as a label indicating that the display is showing an endoscopic view), CT images, and computer-generated pathways.

[0052] The display 420 may include a virtual endoscopic view 426. The virtual endoscopic view 426 is a virtual endoscopic view from the viewpoint of the center of the guiding reference plane 630. The orientation of the view is the normal vector of the guiding reference plane 630, directed toward the distal side of the maneuverable medical device 110. The virtual endoscopic view 426 includes the guiding reference surface When used in conjunction with the symbols 630 and 620 for airway structure, it has the advantage of providing the user with a clearer view of the next airway. Catheter tip 101 Unlike a normal endoscopic view that displays live images 422 from the catheter tip, the guided virtual endoscopic view 426 shows the catheter tip. 101 Even when the catheter is not facing the optimal direction for identifying the next airway, it always captures the anatomical structure of the airway from the center of the airway at the insertion depth. For example, the catheter tip. 101If the catheter is facing the airway wall, the user cannot see anything in the next airway other than the wall. However, the guided virtual endoscope view 426 can provide the user with a view of the next airway from the current insertion depth position, allowing them to see the catheter tip. 101 It can assist in maneuvering it in the optimal direction.

[0053] This information window 421 may, as an addition or alternative, provide a warning that the bendable medical device is approaching a force or bending angle limit (e.g., that if the force or angle exceeds this limit, damage or system malfunction may occur). This warning may, for example, be displayed in a text box or the color of the area where damage or system malfunction may occur may be changed. The force limit or bending angle limit may be set by the manufacturer of the bendable medical device or by the user depending on the patient or procedure. In some embodiments, the control device may initiate corrective actions such as retracting the bendable medical device or loosening one or more tendons / wires of the bendable medical device.

[0054] Figure 6(B) shows an embodiment with a display screen that is simpler in design than the display screen in Figure 6(A). This display screen 420 shows a representation of the airway structure 620, but portions of the airway that are far from the target site are not displayed. A live endoscopy view 422 is also displayed. By reducing the amount of airway displayed and the number of views, both the structure and the view can be enlarged compared to other embodiments. The representation of the airway structure 620 includes both the navigation path 624 and the guiding virtual reference plane. In this embodiment, the centerline 622 and the target site 82 may overlap significantly with the navigation path 624 and are not displayed because they are not necessary for navigation. Each of these elements can be included or excluded according to preference. Furthermore, it is also conceivable to display only a portion of the centerline 622 and / or the navigation path 624 on the representation of the airway structure 620. Similarly, a live endoscopy view 422 is displayed. Alternatively, a guiding virtual endoscopy view 426 may be displayed instead of the live view.

[0055] The guide reference plane 630 is the catheter tip. 101 It can be updated to reflect only the insertion depth position. In Figure 7(A), the flexible medical device 110 has entered the airway, and the display screen 420 shows the catheter tip. 101 The guide reference plane 630 at the position corresponding to the insertion depth is shown. For reference, Figure 7(B) shows the position of the flexible medical device 110 within the lung 120. In Figure 7(C), the flexible medical device 110 has advanced further into the lung compared to Figure 7(A). This can be confirmed by the position of the flexible medical device 110 within the lung 120 in Figure 7(D). Again, the corresponding guide reference plane 630 shown in Figure 7(C) has been lowered to the position shown. Also note that the guide reference plane 630 is always perpendicular to the navigation path. Therefore, the position of the guide reference plane 630 is a plane perpendicular to the navigation path, and the current catheter tip is on that plane. 101 It can be determined that it includes [something].

[0056] Guidance criteria surface With the 630, not only instantaneous parts of the airway but also more are displayed, providing users with a "roadmap" of a wider view than the live image endoscopic view 422.

[0057] Lung airway structure 620 expression This allows us to confirm all possible airway direction options. Furthermore, the relative position of the guide reference plane 630 makes it easy to confirm progress along the navigation path 624 to the target 82.

[0058] In another embodiment, the controllable medical system provides a navigation point at a branch 626 on the navigation path 624 (see Figure 8(A)). Thus, when the user identifies a navigation point 626 (for example, by clicking on the display with a mouse), the guiding reference plane moves to that location in the airway, and the guiding virtual endoscopy view 426 is adjusted to correspond to the location of the selected navigation point 626. This is illustrated in Figure 8(A). When the user clicks on the branch point 626 marked with a green arrow in Figure 8(A), the displayed guiding reference plane 630 moves to that location, as shown by the guiding reference plane 630 in Figure 8(B). (See Figure 8(B) for the guiding virtual endoscopy view.) 426 Similarly, it is adjusted to correspond to the view at the navigation point selected by the user. However, Figure 8(A) shows the tip of the flexible medical device 110. 101 The actual location and the tip of the flexible medical device 110 101 A corresponding guideline plane 630 is shown, but in the display in Figure 8(B), the flexible medical device did not move. Catheter tip 101 It should be noted that the corresponding live image endoscopic view 422 remains unchanged. This allows the user to preview what they will see during navigation as they proceed along the planned airway path, and at the same time, see where the proposed navigation route is located within the airway. This also provides a means for the user to review what they have passed through during navigation.

[0059] The guided virtual endoscope view 426 in Figure 8(C) is at the same user-selected position as in Figure 8(B). to Yes, however, this display 420 shows both the guiding virtual endoscope view 426 located at the selected bifurcation / navigation point 626 and the second virtual endoscope view 428 showing the position of the bronchoscope. This will be displayed. .

[0060] The virtual endoscopic view 426, which is linked to the guidance reference plane 630, has a stable orientation. Therefore, the user can orient the view using the structure displayed in the oriented transverse bronchoscope view.

[0061] By displaying navigation points at the branching point 626, the user can preview (or review) anatomical structures along the navigation path. In some embodiments, indicators (such as markers or color changes) are present when the guiding virtual endoscopy view 426 and / or guiding reference plane 630 are displayed during the preview or review step, and the indicators are located at the tip of the bendable medical device 110. 101 It is located in a position that does not correspond to the position of [the other location].

[0062] In some embodiments, a virtual bronchoscope cross-sectional view and the patient's airway structure 620 are provided for navigation. route 624 can be superimposed. This is shown in Figure 9, indicated by the arrow in the guided virtual endoscopy view 426.

[0063] As shown in Figure 10, the XYZ indicator 432 is oriented to match the tip of the catheter. This XYZ indicator 432 is updated in real time to reflect changes in the direction and / or rotation of the catheter tip. Guidance reference plane 630 has a different XYZ indicator 434 superimposed on it. This XYZ indicator 434 reflects the changes in the XYZ indicator 432, Guidance reference plane It will be displayed in the 630 coordinate system. This allows the user to Guidance reference plane The user can check the orientation of the catheter tip while looking at 630. Furthermore, this allows the user to: Guidance reference plane At 630, it is possible to input the necessary actions to move in the desired direction.

[0064] System 100 can be coordinated, controlled, and / or directed by one or more processors that communicate with the control unit of the entire System 1 and any other components and / or subsystems. The processors can operate based on instructions of computer-readable programs stored in non-temporary computer-readable memory. The processors may be one or more of the following: CPU, MPU, GPU, ASIC, FPGA, DSP, and general-purpose computer, or may include one or more of these. The processors may be dedicated control units or general-purpose computing devices configured to be control units. Examples of non-temporary computer-readable memory include RAM, ROM, CD, DVD, Blu-ray®, hard drives, network-attached storage (NAS), non-temporary computer-readable storage devices connected to an intranet, and non-temporary computer-readable storage devices connected to the Internet.

[0065] Embodiments of the present disclosure can also be realized by a computer system 400 or apparatus that reads and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may more fully be called a “non-temporary computer-readable storage medium”) performing one or more functions of the embodiments described above, and / or by a computer system 400 or apparatus that includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the embodiments described above, or by a computer system or apparatus that is implemented, for example, by reading and executing computer executable instructions from a storage medium to perform one or more functions of the embodiments described above, and / or by controlling one or more circuits for performing one or more functions of the embodiments described above. The computer system may include one or more processors (e.g., a central processing unit (CPU) 410, a microprocessing unit (MPU)), and may include a separate computer or a network of separate processors for reading and executing computer executable instructions. Computer executable instructions can be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, one or more of the following: hard disk, random access memory (RAM), read-only memory (ROM), storage for distributed computing systems, optical discs (such as compact discs (CDs), digital versatile discs (DVDs), Blu-ray Discs (BDs) (registered trademarks), flash memory devices, memory cards, etc. The I / O interface can be used to provide a communication interface to input / output devices, and examples of input / output devices include keyboards, displays, mice, touchscreens, touchless interfaces (e.g., gesture recognition devices), printing equipment, light pens, optical storage devices, scanners, microphones, cameras, drives, communication cables, and networks (wired or wireless).

[0066] When referring to the descriptions, specific details are provided to ensure that the disclosed examples are fully understood. In other examples, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily lengthening this disclosure. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the invention pertains. The scope of the invention is not limited by this specification, but rather by the plain meaning of the terms used in the claims adopted.

[0067] In describing the exemplary embodiments shown in the drawings, specific technical terms are used for clarity. However, the disclosure in this patent specification is not intended to be limited to such selected specific technical terms, and naturally, each specific element includes all similarly functioning technical equivalents.

[0068] While this disclosure has been described with reference to exemplary embodiments, it is naturally not limited to the exemplary embodiments disclosed. For example, this disclosure has been described in relation to exemplary embodiments. However, there are many variations not specifically described to which this disclosure may be applicable. For example, while various embodiments are described with respect to endoscopes for use in medical procedures, this disclosure is also applicable to mechanical procedures of borescopes for use in various mechanical structures. Therefore, the following claims should be given the broadest possible interpretation to encompass all such variations and equivalent structures and functions.

Claims

1. Display device and On the aforementioned display device, An image of the biological lumen taken from the distal end of a flexible medical device located within the biological lumen, Representation of airway structure, A control device configured to display, A navigation system equipped with, The aforementioned expression for the airway structure is, A navigation path passing through at least a portion of the aforementioned airway structure, A guiding reference plane is oriented perpendicular to the navigation path and positioned at the insertion depth of the distal end of the flexible medical device within the biological lumen, This also includes, Navigation system.

2. On the representation of the airway structure, a center line, a target area, or both the center line and the target area are displayed. The navigation system according to claim 1.

3. The control device is further configured to display a virtual endoscopic view, the virtual endoscopic view having an image center on the center line of the guiding reference plane and a distal view orientation along the orientation of the guiding reference plane. The navigation system according to claim 1.

4. The control device is further configured to display a second guided virtual endoscopic view having the image center at the distal end of the bendable medical device. The navigation system according to claim 3.

5. The control device provides the display device with Navigation modality and The distance from the aforementioned flexible medical device to the target site, The insertion depth of the aforementioned flexible medical device, Information regarding the position of the marker displayed on the aforementioned airway structure, A warning that the aforementioned flexible medical device is approaching its limit in terms of force or bending angle, Further configured to display navigation information including at least one of the following, The navigation system according to claim 1.

6. The control device is configured to initiate corrective action when the bendable medical device approaches a limit value in terms of force or bending angle. The navigation system according to claim 1.

7. The control device is configured to update the display on the guide reference surface when the flexible medical device moves through the airway structure. The navigation system according to claim 1.

8. The aforementioned guide reference plane is displayed to have a three-dimensional sense of perspective. The navigation system according to claim 1.

9. The aforementioned representation of the airway structure is a model of the patient's airway based on one or more computed tomography (CT) scanner data and / or magnetic resonance imaging (MRI) scanner data. The navigation system according to claim 1.

10. The representation of the airway structure further includes one or more markers indicating one or more branching points of the airway structure. The navigation system according to claim 1.

11. The aforementioned guide reference plane is movable to the branching point of the airway structure in response to user input selecting the branching point. The navigation system according to claim 1.

12. The user input includes selecting markers that indicate one or more branching points of the airway structure, which are placed on the representation of the airway structure. The navigation system according to claim 11.

13. Flexible medical devices, An actuator system for operating the aforementioned flexible medical device, The navigation system according to claim 1, A medical system equipped with [a specific feature / feature].

14. The control device is further configured to display a virtual endoscopic view, the virtual endoscopic view having an image center on the center line of the guiding reference plane and a distal view orientation along the orientation of the guiding reference plane. The medical system according to claim 13.

15. A method for controlling a display, A step of acquiring an image of a biological lumen from the distal end of a flexible medical device, Steps to obtain a representation of the airway structure, Steps to acquire the target site, The steps include generating the centerline of at least a portion of the airway structure, The steps include generating a navigation path to the target site that passes through at least a portion of the airway structure, The steps include displaying the aforementioned representation of the airway structure on a display device, Includes, The aforementioned expression for the airway structure is, The navigation path passing through at least a portion of the airway structure, A guiding reference plane is oriented perpendicular to the navigation path and positioned at the insertion depth of the distal end of the flexible medical device within the biological lumen, This also includes, method.

16. The step of displaying the image of the biological lumen on the display device, The method according to claim 15, further comprising:

17. The representation of the airway structure is obtained from a preoperative CT image. The method according to claim 15.

18. The aforementioned target area is obtained from the user. The method according to claim 15.

Images