Graphical user interface for defining anatomical boundaries

Abstract

The invention relates to a graphical user interface for defining anatomical boundaries. A medical system includes a display system and a user input device. The medical system also includes a control system communicatively coupled to the display system and the user input device. The control system is configured to display, via the display system, image data corresponding to the three-dimensional anatomical region, and to receive, via the user input device, a first user input to generate a first curve in the three-dimensional anatomical region. The control system is also configured to receive a second user input via the user input device to generate a second curve in the three-dimensional anatomical region, and to determine an anatomical boundary defined by the first curve and the second curve. The anatomical boundary is indicative of a surface of an anatomical structure in the three-dimensional anatomical region.

Description

[0001] This application is a divisional application of Chinese patent application 2019800614269 (PCT / US2019 / 053820) entitled "Graphical User Interface for Defining Anatomical Boundaries", filed on September 30, 2019.

[0002] Cross-references to related applications This application claims the benefit of U.S. Provisional Application 62 / 741,157, filed October 4, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to systems and methods for planning and executing image-guided procedures, and more specifically to systems and methods for defining anatomical boundaries using a graphical user interface. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, clinicians can insert minimally invasive medical instruments (including surgical instruments, diagnostic instruments, therapeutic instruments, or biopsy instruments) to reach target tissue locations. One such technique utilizes flexible, elongated devices (such as catheters) that can be steered / manipulated, inserted into an anatomical channel, and navigated toward areas of interest within the patient's anatomy. Control of such elongated devices by medical personnel during image-guided procedures involves the management of several degrees of freedom, including at least the management of the insertion and retraction of the elongated device and the radius of rotation or bending of the device. Additionally, different operating modes can be supported.

[0005] Therefore, a graphical user interface that supports intuitive planning of medical procedures, including minimally invasive medical techniques, would be advantageous. Summary of the Invention

[0006] The embodiments of the present invention can be best summarized by the claims appended to the specification.

[0007] In one embodiment, the medical system includes a display system and a user input device. The medical system also includes a control system communicatively coupled to the display system and the user input device. The control system is configured to display image data corresponding to a three-dimensional anatomical region via the display system and to receive first user input via the user input device to generate a first curve in the three-dimensional anatomical region. The control system is further configured to receive second user input via the user input device to generate a second curve in the three-dimensional anatomical region and to determine an anatomical boundary defined by the first and second curves. This anatomical boundary indicates the surface of an anatomical structure within the three-dimensional anatomical region.

[0008] In another embodiment, the method for planning a medical procedure includes displaying image data corresponding to a three-dimensional anatomical region via a display system and receiving multiple user inputs via a user input device to generate multiple curves in the three-dimensional anatomical region. The method further includes determining anatomical boundaries from the multiple curves. These anatomical boundaries define vulnerable portions of the three-dimensional anatomical region. The method also includes displaying the anatomical boundaries superimposed on the image data via the display system.

[0009] In another embodiment, the non-transitory machine-readable medium includes a plurality of machine-readable instructions, which, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method. The method includes displaying CT image data corresponding to the lungs via a display system, and receiving a plurality of user inputs via a user input device to generate a plurality of curves in different slices of the CT image data. The method also includes interpolating between the plurality of curves to determine anatomical boundaries indicating the location of the pleura in the CT image data; and displaying the anatomical boundaries superimposed on the CIT image data via the display system.

[0010] It should be understood that the above overview and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of this disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] This patent or application document contains at least one color drawing. A copy of this patent or application with the color drawing will be provided by the International Bureau upon request and at the necessary cost.

[0012] Figure 1 This is a simplified diagram of a medical system based on some embodiments.

[0013] Figure 2A and Figure 2B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space.

[0014] Figure 3A This is a simplified diagram of a method for defining anatomical boundaries according to some embodiments.

[0015] Figure 3B This is an illustration of a method for defining anatomical boundaries according to other embodiments.

[0016] Figure 3C This is an illustration of a method for presenting anatomical boundaries according to some embodiments.

[0017] Figure 3D This is an illustration of a method for providing guidance information to guide the determination of anatomical boundaries according to some embodiments.

[0018] Figures 3E to 3G This is an illustration of a method for providing guidance information according to some embodiments.

[0019] Figures 4A to 4F This is a simplified diagram of a graphical user interface during the execution of a method for defining anatomical boundaries, according to some embodiments.

[0020] Figure 5A and Figure 5B This is a simplified diagram illustrating the range guidance associated with anatomical boundaries according to some embodiments.

[0021] Figure 5C A graphical user interface is shown that presents range guidance and two-dimensional image data.

[0022] The embodiments of this disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that the same drawing numbers are used to identify the same elements shown in one or more of the drawings, wherein the illustrations in the drawings are for the purpose of illustrating embodiments of this disclosure and not for the purpose of limiting the embodiments of this disclosure. Detailed Implementation

[0023] In the following description, specific details of some embodiments consistent with this disclosure are set forth. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Other elements, though not specifically described herein, may be implemented by those skilled in the art within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described or if one or more features would render the embodiment inoperable.

[0024] In some cases, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring various aspects of the embodiments.

[0025] This disclosure describes various instruments and parts thereof in three-dimensional space. As used herein, the term "orientation" refers to the position of an object or part thereof in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part thereof (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the orientation of an object or part thereof in at least one translational degree of freedom and the orientation of an object or part thereof in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, orientations, or orientations measured along the object.

[0026] like Figure 1 As shown, medical system 100 typically includes a manipulator assembly 102 for operating medical device 104 during various procedures performed on patient P. Medical device 104 can extend through an opening in patient P's body to an internal surgical site within patient P's body. Medical system 100 can be remotely operated, non-remotely operated, or a hybrid of both. Manipulator assembly 102 can be a remotely operated, non-remotely operated, or hybrid remotely and non-remotely operated component, having selectable degrees of freedom of movement that can be electrically and / or remotely operated, and selectable degrees of freedom of movement that can be non-electrically and / or non-remotely operated. Manipulator assembly 102 is mounted to or near operating table T. Main assembly 106 allows operator O (e.g., as...) Figure 1 The surgeon, clinician, or physician shown views the access site and controls the manipulator assembly 102.

[0027] The main component 106 may be located at an operator console, which is typically located in the same room as the operating table T, such as on the side of the operating table on which the patient P is located. However, it should be understood that the operator O may be located in a different room from the patient P or in a completely different building. The main component 106 typically includes one or more control devices for controlling the manipulator component 102. The control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manually operated controllers, voice recognition devices, body motion or presence sensors and / or the like.

[0028] Manipulator assembly 102 supports medical device 104 and may include one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, commonly referred to as a setup structure) and / or one or more servo-controlled links (e.g., one or more links that can be controlled in response to commands from a control system) and the kinematic structure of the manipulator. Manipulator assembly 102 may optionally include multiple actuators or motors that drive inputs on medical device 104 in response to commands from a control system (e.g., control system 112). Actuators may optionally include a drive system that, when coupled to medical device 104, can advance medical device 104 to a naturally or surgically generated anatomical orifice. Other drive systems may move the distal end of medical device 104 with multiple degrees of freedom, which may include three linear movements (e.g., linear movements along the X, Y, Z Cartesian axes) and three rotational movements (e.g., rotations about the X, Y, Z Cartesian axes). Additionally, the actuator can be used to actuate the articulated end effector of the medical device 104 for gripping tissue in the jaws of a biopsy device and / or the like.

[0029] The medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the manipulator assembly 102 and / or the medical device 104. Such subsystems may include a position / location sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the orientation, velocity, rate, posture, and / or shape of a distal end and / or along one or more segments that may constitute the flexible body of the medical device 104; a visualization system for capturing images from the distal end of the medical device 104; and actuator orientation sensors (such as resolvers, encoders, potentiometers, and the like) describing the rotation and orientation of motors controlling the device 104.

[0030] The medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and the medical device 104. The display system 110 and the main component 106 can be oriented such that the operator O can remotely control the medical device 104 and the main component 106 using telepresent perception.

[0031] In some embodiments, medical device 104 may include a visualization system that includes an image capture component that records instantaneous or real-time images of the surgical site and provides the images to operator O via one or more displays in display system 110. The instantaneous images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator component may be used with medical device 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or otherwise executes with one or more computer processors, which may include the processor of control system 112.

[0032] Display system 110 can also display images of the surgical site and medical instruments captured by the visualization system. In some examples, medical system 100 can configure the controls for medical instrument 104 and main component 106 such that the relative positions of the medical instruments resemble the relative positions of the operator O's eyes and hands. In this way, operator O can manipulate medical instrument 104 and hand controls as if viewing the workspace in a substantially real-world setting. Real-world presence means that the image presentation is a true perspective image, simulating the viewpoint of a physician physically manipulating medical instrument 104.

[0033] In some examples, the display system 110 can use image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein scanning, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data can be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information) and / or as images from models created from the preoperative or intraoperative image dataset.

[0034] In some embodiments, often for the purpose of image-guided medical procedures, the display system 110 may display virtual navigation images, wherein the actual position of the medical device 104 is registered (i.e., dynamically referenced) with preoperative or real-time images / models. This can be done from the perspective of the medical device 104 to present a virtual image of the internal surgical site to the operator O.

[0035] The medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, main component 106, sensor system 108, and display system 110. The control system 112 also includes programmable instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement some or all of the methods described according to various aspects of the disclosure herein, including instructions for providing information to the display system 110. While the control system 112... Figure 1 The simplified diagram is shown as a single box, but the system may include two or more data processing circuits, wherein a portion of the processing may be performed on or adjacent to the manipulator assembly 102, another portion of the processing may be performed at the main assembly 106, and / or similarly. The control system 112 may execute instructions, including instructions corresponding to the processes disclosed herein and described in more detail below. In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to this feedback, the control system 112 may transmit a signal to the main assembly 106. In some examples, the control system 112 may transmit signals instructing one or more actuators of the manipulator assembly 102 to move the medical device 104.

[0036] The control system 112 may optionally further include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device 104 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of the acquired anatomical pathway. Software, which may be used in conjunction with operator input, is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of parts or entire anatomical organs or regions. The image dataset is associated with the composite representation. The virtual visualization system obtains sensor data from the sensor system 108, which is used to calculate the approximate anatomical position of the medical device 104 relative to the patient P. The system may implement the sensor system 108 to register and display the medical device together with surgical images recorded preoperatively or intraoperatively. For example, such a system is disclosed by reference in its entirety to PCT Publication WO 2016 / 191298 (published on 1 December 2016) (disclosed “Systems and Methods of Registration for Image Guided Surgery”), which is incorporated herein by reference.

[0037] The medical system 100 may further include optional operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some embodiments, the medical system 100 may include more than one manipulator assembly and / or more than one main assembly. Among other factors, the exact number of manipulator assemblies will depend on the medical procedure and space constraints within the operating room. The main assemblies 106 may be juxtaposed, or they may be positioned in separate locations. Multiple main assemblies allow more than one operator to control one or more manipulator assemblies in various combinations.

[0038] Figure 2A and Figure 2B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space. Figure 2A and Figure 2B As shown, the surgical environment 300, including patient P, is located in... Figure 1 On the operating table T. Patient P can remain still within the surgical environment in the sense that the patient's overall movement is restricted by sedation, restraint, and / or other means. Circulatory anatomical movement, including the patient P's respiratory and cardiac movements, can continue. Within the surgical environment 300, medical device 304 is used to perform medical procedures, which may include, for example, surgery, biopsy, ablation, illumination, irrigation, aspiration, or system registration procedures. Medical device 304 may be, for example, an instrument 104. Instrument 304 includes a flexible, elongated device 310 (e.g., a catheter) coupled to an instrument body 312. The elongated device 310 includes one or more channels (not shown) sized and shaped to receive medical instruments (not shown).

[0039] The elongated device 310 may also include one or more sensors (e.g., components of sensor system 108). In some embodiments, an optical fiber shape sensor 314 is fixed to a proximal end 316 on the instrument body 312. In some embodiments, the proximal end 316 of the optical fiber shape sensor 314 may move with the instrument body 312, but the position of the proximal end 316 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal end 316 to another point, such as the distal end 318 of the elongated device 310. The shape sensor 314 may be aligned with the flexible elongated device 310 (e.g., provided within an internal passage (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The shape sensor 314 may be used to determine the shape of the flexible elongated device 310. In one alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one or more dimensions within the structure. Various systems and methods for three-dimensionally monitoring the shape and relative orientation of optical fibers are described in the following documents: U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (disclosing "Fiber opticposition and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber optic shape and relative orientation sensing"); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fibre Bend Sensor"), which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. Various systems for registering and displaying surgical instruments and surgical images using fiber optic sensors are provided in PCT Publication WO 2016 / 191298 (published December 1, 2016) (which discloses "Systems and Methods of Registration for Image Guided Surgery"), which is incorporated herein by reference in its entirety.

[0040] In various embodiments, orientation sensors, such as electromagnetic (EM) sensors, may be incorporated into medical device 304. In various embodiments, a series of orientation sensors may be positioned along elongated device 310 and then used for shape sensing. In some embodiments, the orientation sensors may be configured and positioned to measure six degrees of freedom (e.g., three orientation coordinates X, Y, Z and three orientation angles of pitch, yaw, and roll indicating a base point) or five degrees of freedom (e.g., three orientation coordinates X, Y, Z and two orientation angles of pitch and yaw indicating a base point). Further description of the orientation sensor system is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999) (disclosing "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0041] The elongated device 310 may also accommodate cables, linkages, or other steering controls (not shown) extending between the device body 312 and the distal end 318 to controllably bend the distal end 318. In some examples, at least four cables are used to provide independent control of the pitch "up" steering of the distal end 318 and the yaw "left" steering of the distal end 318. Steering elongated devices are described in detail in U.S. Patent Application No. 13 / 274,208 (filed October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. The device body 312 may include a drive input that is removably coupled to and receives power from a drive element (such as an actuator) of the manipulator assembly.

[0042] The instrument body 312 may be coupled to an instrument holder 306. The instrument holder 306 is mounted to an insertion stage 308 fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable but have a known position within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument holder 306 may be a component of a manipulator assembly (e.g., manipulator assembly 102) coupled to the medical device 304 to control insertion motion (i.e., movement along axis A) and optionally control movement of the distal end 318 of the elongated device 310 in multiple directions including yaw, pitch, and roll. The instrument holder 306 or the insertion stage 308 may include actuators, such as servo motors (not shown), for controlling movement of the instrument holder 306 along the insertion stage 308.

[0043] Sensor device 320, which may be a component of sensor system 108, provides information about the orientation of the instrument body 312 as it moves along insertion axis A on insertion stage 308. Sensor device 320 may include a resolver, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of actuators controlling the movement of instrument carrier 306 and thus the movement of instrument body 312. In some embodiments, insertion stage 308 is linear. In some embodiments, insertion stage 308 may be curved or have a combination of curved and linear segments.

[0044] Figure 2A The instrument body 312 and instrument holder 306 are shown in a retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is at orientation L0 on axis A. In this orientation along the insertion stage 308, the position of the proximal point 316 can be set to zero and / or another reference value to provide a reference for describing the orientation of the instrument holder 306 and therefore the proximal point 316 on the insertion stage 308. With this retracted orientation of the instrument body 312 and instrument holder 306, the distal end 318 of the elongated device 310 can be positioned precisely inside the access port of the patient P. Additionally, in this orientation, the sensor device 320 can be set to zero and / or another reference value (e.g., I=0). Figure 2BIn this process, the instrument body 312 and instrument holder 306 are advanced along a linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advancement orientation, the proximal point 310 is at orientation L1 on axis A. In some examples, encoder and / or orientation data from one or more actuators controlling the movement of the instrument holder 306 along the insertion stage 308 and / or one or more orientation sensors associated with the instrument holder 306 and / or the insertion stage 308 are used to determine the orientation Lx of the proximal point 316 relative to orientation L0. In some examples, orientation Lx may further be used as an indicator of the distance or insertion depth of the distal end 318 of the elongated device 310 into the channel of the anatomical body of the patient P.

[0045] In illustrative applications, medical systems such as Medical System 100 may include robotic catheter systems for use in lung biopsy procedures. The catheters of the robotic catheter system provide conduits for tools, such as endoscopes, endobronchial ultrasound (EBUS) probes, and / or biopsy tools, to be delivered to a location within the airway where one or more anatomical targets (such as lesions, nodules, tumors, and / or similar objects) for lung biopsy are located. As the catheter is driven through the anatomy, an endoscope is typically mounted, allowing a clinician, such as a surgeon O, to monitor a live camera feed at the distal end of the catheter. The live camera feed and / or other real-time navigation information may be displayed to the clinician via a graphical user interface. An example of a graphical user interface for monitoring a biopsy procedure is covered in U.S. Provisional Patent Application No. 62 / 486,879, filed April 18, 2017, entitled “Graphical User Interface for Monitoring an Image-Guided Procedure,” which is incorporated herein by reference in its entirety.

[0046] Before performing a biopsy procedure using a robotic catheterization system, preoperative planning steps can be performed to plan the biopsy procedure. Preoperative planning steps may include segmenting image data (such as a patient's CT scan) to create a 3D model of the anatomy, selecting anatomical targets within the 3D model, identifying airways within the model, growing airways to form a connected tree-like airway network, and planning the trajectory between the target and the connected tree. One or more of these steps can be performed on the same robotic catheterization system used to perform the biopsy. Alternatively or additionally, planning can be performed on a different system (such as a workstation dedicated to preoperative planning). The plan for the biopsy procedure is saved (e.g., as one or more digital files) and transferred to the robotic catheterization system used to perform the biopsy procedure. The saved plan may include the 3D model, airway identification, target location, trajectory to the target location, route through the 3D model, and / or similar elements.

[0047] The following provides illustrative embodiments of a graphical user interface (GUI) for planning medical procedures, including but not limited to the lung biopsy procedure described above. The GUI may include multiple modes, including a data selection mode, a mixed segmentation and planning mode, a preview mode, a save mode, a management mode, and a viewing mode. Some aspects of the GUI resemble features described in the following documents: U.S. Provisional Patent Application No. 62 / 357,217, filed June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information During and Image-Guided Procedure,” and U.S. Provisional Patent Application No. 62 / 357,258, filed June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information in a Plurality of Modes During and Image-Guided Procedure,” both of which are incorporated herein by reference in their entirety.

[0048] In the planning and execution of medical procedures, anatomical boundaries, or virtual “hazard fences,” can be used to identify surfaces where medical devices do not intersect during the procedure. Anatomical boundaries can shield vulnerable portions of anatomy near a target location or other part of interest from unintentional penetration by the medical device. Parts of interest that include vulnerable anatomical structures or surfaces can include, for example, the pleura, pulmonary fissures, large bullae, and blood vessels. For example, puncturing the pleura during a medical procedure can result in a dangerous pneumothorax in the patient. Consistent with this embodiment, defining an anatomical boundary corresponding to the pleura can allow the operator to restrict the path of the medical device to avoid vulnerable portions of the anatomical structure. For example, a candidate path may be invalid if it damages the vulnerable portion and / or the like when crossing within a threshold distance of a vulnerable portion of the anatomical structure.

[0049] Figure 3A This is a simplified diagram of a method 400A for defining anatomical boundaries according to some embodiments. Figures 4A to 4F This is a simplified diagram corresponding to the graphical user interface 500 during the execution of method 400A according to some embodiments. Figures 1 to 2B In some consistent embodiments, the graphical user interface 500 may be displayable on a display system (such as display system 110 and / or a display system for a standalone planning workstation).

[0050] The graphical user interface 500 displays information associated with planning medical procedures in one or more views, which are viewable to a user (such as operator O). Although Figures 4A to 4F The illustration shows an illustrative arrangement of views; however, it should be understood that the graphical user interface 500 can display any suitable number of views in any suitable arrangement and / or on any suitable number of screens. In some examples, the number of views displayed simultaneously can be changed by turning views on and off, minimizing and maximizing views, moving between the foreground and background of the graphical user interface 500, switching between screens, and / or otherwise completely or partially blurring views. Similarly, the arrangement of views, including size, shape, orientation, order (in the case of overlapping views), and / or such arrangements, can be changed and / or can be user-configurable.

[0051] The methods disclosed herein are shown as a set of operations or procedures. Not all illustrated procedures can be performed in all embodiments of the illustrated methods. Additionally, one or more procedures not explicitly shown may be included before, after, between, or as part of illustrated procedures. In some embodiments, one or more procedures within a process may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), causes one or more processors to execute one or more procedures within the process. In one or more embodiments, the process may be executed by control system 112.

[0052] At process 410, image data 510 corresponding to the three-dimensional anatomical region of patient P is displayed via a graphical user interface 500. For example... Figures 4A to 4F As shown, image data 510 may include, for example, calculated computed tomography (CT) image data. Image data 510 may include multiple images of a three-dimensional anatomical region, wherein... Figure 4A A single plane or "slice" of the image data is shown. Alternatively or additionally, image data 510 may include a three-dimensional anatomical model, such as the three-dimensional anatomical model shown in thumbnail 512 of the graphical user interface 500. In some embodiments, image data 510 may include segmented data 514 indicating anatomical features (such as airways, blood vessels, or similar locations in the lungs) identified from CT image data. In some embodiments, image data 510 may include anatomical targets 516 of a medical device, such as a biopsy site. In various alternative embodiments, the image data may be generated using other imaging techniques such as magnetic resonance imaging (MRI), fluorescence examination, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like.

[0053] At process 420, first user input for generating or defining curve 520 in the three-dimensional anatomical region is received via a user input device. Curve 520 is generated in a plane of image data 510. In some embodiments, the first user input may be provided by an operator via a mouse, touchscreen, stylus, or the like. Figure 4B As shown, curve 520 can be displayed via a graphical user interface 500. In this embodiment, curve 520 may correspond to a surface identified by the operator that is part of the pleura of the lung.

[0054] At process 430, a second user input for generating or defining a second curve 530 in the three-dimensional anatomical region is received via a user input device. Curve 530 is generated in a plane of image data 510 that is different from the image plane in which curve 520 is defined. Figure 4C As shown, curve 530 can be displayed via graphical user interface 500.

[0055] At process 440, optionally, additional user input may be received, each additional user input generating or defining additional curves in the three-dimensional anatomical region (e.g., additional curve 532). Figure 4D Typically, curves 530, 532, and any additional curves are defined in a manner similar to that of curve 520. Any additional curves may be located in planes that are different from curves 520 and 530 and in any order relative to the image data 510 (e.g., in different slices of CT image data).

[0056] At point 450 in the process and as follows Figure 4E As shown, an anatomical boundary 540 is defined by curves 520, 530, and any additional curves. In some embodiments, the anatomical boundary is determined by interpolation or otherwise identifying intermediate curves that combine with the boundary 540. According to some embodiments, the anatomical boundary 540 may indicate the surface of a three-dimensional anatomical region or a fragile or interesting surface that is not crossed by a medical device during a medical procedure.

[0057] Optionally, at process 460, the anatomical boundary 540 is displayed via a graphical user interface 500. According to some embodiments, a visual representation of the anatomical boundary 540 can be overlaid on the image data. Figure 4E and Figure 4F As shown, a cross-sectional representation of the anatomical boundary 540 can be displayed as a curve overlaid on a CT slice, and a three-dimensional representation of the anatomical boundary 540 can be displayed as a translucent or grid-line mesh or similar on a three-dimensional anatomical model in thumbnail view 512.

[0058] In some cases, the interpolated portion of the anatomical boundary 540 may not precisely follow the actual anatomical boundary that the operator is seeking to define. For example, in Figure 4E In the illustrative example shown, the interpolated portion of the anatomical boundary 540 of the pleura, intended to track the lung, between curves 520 and 530, is clearly misaligned with the pleura. To correct this misalignment, method 400A can return to processes 420-460 to receive parameters that define the anatomical boundary 540 for updating to more closely align with the desired anatomical boundary (e.g., ...). Figure 4FAdditional user input for additional curves (as shown in the diagram). In this way, processes 420-450 can be executed iteratively until a satisfactory alignment is achieved. Similarly, the extent of anatomical boundary 540 can be extended by returning to processes 420-450 to receive additional user input for additional curves that are confined outside the current extent of anatomical boundary 540.

[0059] Figure 3B This is an illustration of a method 400B for defining anatomical boundaries according to some embodiments. Some procedures in method 400B are the same as those identified in FIG400A and are indicated by the same reference numerals.

[0060] Before or after the display of image data at process 410, at optional process 412, the user can present the curve using a selectable option between freehand and polyline drawing options. In some embodiments, curve 520 may be drawn as a freehand, as a polyline, as a series of plotted points, or the like. In the case of polyline input (e.g., a series of straight line segments) or a series of plotted points, curve 520 may be determined, for example, by spline fitting. Optionally, spline fitting may be performed when all points are received. Optionally, spline fitting may be performed on all received points and updated when a new point is received. Optionally, spline fitting may be performed using all received points and the current mouse position, allowing the user to see the shape of the fitted curve in real time before receiving a point. According to some embodiments, first user input may be received in response to the operator receiving a selection of the anatomical boundary tool 518. The selection of the anatomical boundary tool 518 instructs the operator to define anatomical boundaries via the graphical user interface 500.

[0061] At process 450, the anatomical boundary 540 can be determined based on a three-dimensional surface mesh stored or displayed, which includes multiple vertices. Figure 3CA method 470 for representing anatomical boundaries according to some embodiments is illustrated. At process 472, the anatomical boundary 540 may be generated as a three-dimensional surface mesh including a plurality of vertices. In an alternative technique, at process 473, the vertices of the three-dimensional surface mesh can be determined by resampling curves 520, 530, and any additional curves to an equal number of sampling points. At process 474, spline fitting is performed between matching sampling points from the respective curves, producing a plurality of splines. At process 475, each of the plurality of splines is resampled to produce the vertices of the three-dimensional surface mesh. In another alternative technique, at process 476, the vertices of the three-dimensional surface mesh can be determined by fitting the three-dimensional spline surface to curves 520, 530, and any additional curves. At process 477, the three-dimensional spline surface is resampled to produce the vertices of the three-dimensional surface mesh.

[0062] Refer again Figure 3B In optional process 452, the anatomical boundary 540 can be further determined based on features of the image data 510. For example, the anatomical boundary 540 can be snapped to regions of the image data 510 with high-intensity gradients, since high-intensity gradients indicate the presence of surfaces of interest (e.g., the pleura of the lung, the wall of blood vessels, etc.). Similarly, computer vision techniques, including machine learning algorithms, can be applied to the image data 510 to identify candidate anatomical boundaries. Consistent with this embodiment, the anatomical boundary 540 can be snapped to candidate anatomical boundaries determined by such computer vision or machine learning techniques.

[0063] At optional step 462, the anatomical boundary 540 can be deformed based on patient movement. During navigation, the patient's anatomy and therefore the model can move or be deformed by forces, for example, from the medical device, lung exhalation and inspiration, and the beating heart. The deformation can be measured, for example, by a shape sensor in the medical device, or predicted by simulation, and the deformation can be applied to the model. The anatomical boundary 540 can be similarly adjusted or deformed to correspond to the deformation of the model.

[0064] Figure 3D This is an illustration of a method 400C for defining anatomical boundaries according to some embodiments. Some procedures in method 400C are the same as those identified in Figure 400A and are indicated by the same reference numerals. At procedures 414, 422, and 452, various guidance information and visual aids / helpers can be displayed via a graphical user interface 500 to assist the operator in defining or adjusting the anatomical boundaries 540.

[0065] At process 414, guiding information and visual aids can be further displayed to suggest the extent or shape that the anatomical boundary 540 should cover. Therefore, extent guiding information can be displayed to improve the extent of protection provided by the anatomical boundary 540, as shown below. Figures 5A to 5B Let's discuss this in more detail.

[0066] Figure 5A and Figure 5B This is a simplified diagram illustrating the range guide 600 associated with anatomical boundaries (such as anatomical boundary 540) according to some embodiments. Figure 5C A graphical user interface 670 is shown, presenting range guidance 600 and two-dimensional image data 510 to assist and guide the user in drawing curves. As previously described, anatomical boundaries 540 typically identify surfaces 610, such as the surface of the lung pleura, which should not be punctured or otherwise contacted or crossed by medical instruments during medical procedures at the site of target 620. Figure 5A and Figure 5B As shown, the medical procedure can correspond to a biopsy procedure in which the catheter 630 is inserted near the target 620. During the biopsy procedure, the needle is aimed from the exit point 635 of the catheter 630 toward the target 620. Therefore, in a biopsy procedure (and various other types of procedures in which instruments can extend from the catheter 630 toward the target 620), the anatomical boundary 540 can be used to identify portions of the surface 610 that are behind the target 620 relative to the exit point 635 and are thus at risk of puncture if the needle (or other instrument) extends too far beyond the target 620.

[0067] like Figure 5A As shown, the three-dimensional hazardous portion 640 of surface 610 is determined based on the intersection of the anatomical surface 610 and a three-dimensional zone. This zone can be, for example, a conical projection 642 extending from exit point 635 through target 620. In some embodiments, the hazardous portion 640 may include an additional margin 644 extending directly beyond the region within the projection 642. In some embodiments, the hazardous portion 640 may be determined in a binary manner (e.g., a given portion is considered hazardous or not hazardous) or in a progressive or continuous manner to reflect changes in the level of danger at different locations.

[0068] Based on the identification of the hazardous portion 640 of surface 610, guidance information can be provided to the operator in image data 510 to ensure that the anatomical boundaries 540 defined during method 400A provide adequate protection for the hazardous portion 640 of surface 610. For example, a visual representation of the hazardous portion 640, projection 642, or both can be displayed via a graphical user interface 500.

[0069] Figure 3E An embodiment of the guidance process 414 is illustrated in more detail by showing a method 414a for providing guidance information. At process 480, a three-dimensional zone (e.g., a conical projection 642) is generated, extending from the instrument exit point 635 toward the target 620. At process 482, a two-dimensional projection of the three-dimensional zone is displayed along with image data 510. Figure 5C As shown, a two-dimensional projection region 650 of the three-dimensional zone (e.g., cone 642) is provided in an overlapping manner on the two-dimensional image data 510 illustrating the target 620. At process 484, using the projection 650 as a guide, the user can generate a curve 652 as described in processes 420 and 430. The curve 652 can be drawn to extend within the region 650 and optionally extend beyond the region 650 to define the boundary of the portion 640 in danger. As previously described, additional curves can be drawn in additional slices of the two-dimensional image data 510 to generate multiple curves for generating the anatomical border 540. In some embodiments, pixels in the region in danger can be displayed in overlays of different shades, colors, or translucent colors. The guide can be turned on or off automatically by user selection or by combination. Additionally or alternatively, an indicator of whether the anatomical border 540 fully protects the portion 640 in danger can be displayed via the graphical user interface 500 or otherwise conveyed to the operator.

[0070] like Figure 5B As shown, one factor that may cause different levels of risk is the uncertainty associated with the medical procedure (e.g., uncertainty about the location of exit point 635, uncertainty about the location of target 620, or both). Other factors that may cause different levels of risk include the distance between surface 610 and exit point 635; locations farther from exit point 635 are generally at lower risk compared to closer locations.

[0071] During the planning process, a safety score can be calculated and provided to the operator, indicating the likelihood that the instrument will breach boundary 540. Based on the score, the planned navigation path can be adjusted or modified to achieve a safer route. Various paths with different safety scores are available for the operator to choose from.

[0072] Refer again Figure 3D Additional guidance information and visual aids can be provided at process 422. Figure 3FAn embodiment of the guidance process 422 is illustrated in more detail by showing a method 422a for providing guidance information. At process 486, the projection or shadow of curve 520 may be displayed in other planes of image data 510 where curve 520 is not displayed (e.g., in CT slices of image data 510 other than those including curve 520). Therefore, when defining curve 530, the projection or shadow of curve 520 provides the operator with guidance on the characteristics of curve 520 (e.g., start point, end point, length, etc.) in the form of a cue. Without such a cue, the operator may inadvertently define curve 530 with characteristics significantly different from curve 520 (e.g., significantly different start point, end point, or length). In this case, anatomical boundary 540 may have an irregular shape or otherwise may not correspond to the desired anatomical boundary.

[0073] At process 488, the guidance information may include the start and end points of the first curve. In some embodiments, the anatomical boundary 540 may also have an irregular shape when curve 530 is unintentionally flipped relative to curve 520 (e.g., when the corresponding start and end points are at opposite ends of the curves). For example, the anatomical boundary 540 may have a distorted shape when the orientation is flipped. Thus, guidance information may be displayed to indicate which direction curve 530 should be oriented to match curve 520. For example, regarding the projection or shadow of curve 520 discussed above (or similarly, the projection of anatomical boundary 540), the start point may be displayed in a visually distinguishable manner from the end point (e.g., using a different color, pattern, texture, etc.).

[0074] Refer again Figure 3D This can provide instrument or boundary adjustment guidance information at process 452. Figure 3GAn embodiment of the guidance process 452 is illustrated in more detail by showing a method 452a for providing guidance information. For example, at an optional process 490, the projection or shadow of the anatomical boundary 540 may be extrapolated and displayed in an area outside the current extent of the anatomical boundary 540 to provide guidance to the operator as the extent of the anatomical boundary 540 is expanded. In some embodiments, the projection or shadow of the anatomical boundary 540 may be displayed in a manner visually distinguishable from the anatomical boundary 540 itself (e.g., using a different color, pattern, texture, etc.) to alert the operator whether the currently displayed cross-section is within or outside the current extent of the anatomical boundary 540. As previously described, the planning for a biopsy procedure including the anatomical boundary 540 may be saved and used by the control system to provide automated navigation of the medical device or operator navigation assistance in performing the biopsy procedure. During navigation, the boundary 540 may be displayed using a three-dimensional anatomical model of the anatomical region (e.g., view 512), an endoluminal view, or other anatomical views presented on the user's display. Boundary 540 can also or alternatively be displayed using a registered image from other imaging techniques, such as a fluorescence fluoroscopic image obtained during a medical procedure (e.g., superimposed on the registered image).

[0075] At optional procedure 491, suggested deployment locations for medical devices can be provided. For example, during a registration procedure that registers a 3D model to a patient's anatomy, points collected for the medical device can be used to access a recommended point cloud within the patient's anatomy. The recommended point cloud can be determined based on its position relative to boundary 540. For example, a point can only be recommended if it is within a threshold distance from boundary 540. Similarly, during a biopsy procedure, the recommended biopsy location can be determined based on its position relative to boundary 540. For example, a biopsy point can only be recommended if it is within a threshold distance from boundary 540.

[0076] In optional process 492, during the medical procedure, the orientation and position of the medical device relative to the anatomical boundary 540 can be monitored. The distance between the medical device and the anatomical boundary 540 can be measured, for example, the distance to the distal portion of the device or the portion of the device closest to the anatomical boundary 540. In process 493, when the distance between the device and the anatomical boundary 540 becomes less than a predetermined threshold distance value, an indicator can be provided to the operator. For example, a visual indicator on the graphical user interface 500 can be provided in the form of a color change, text alert, highlighted device, highlighted boundary 540, or other visual warning signals. The indicator can also be provided in the form of an auditory, tactile, or other operator-perceptible signal. Alternatively or additionally, in process 494, the control system 112 can monitor the distance and slow down or stop the device completely as it approaches the surface corresponding to the boundary 540. Alternatively or additionally, in process 495, the operator can provide user input (e.g., pressing a button) that will move the distal end of the medical device away from the surface corresponding to the boundary 540. Alternatively or alternatively, distance-based indicators can be used in planning procedures with virtual medical devices.

[0077] One or more elements in the embodiments of this disclosure can be implemented in software to execute on a processor of a computer system, such as a control processing system. When implemented in software, the elements of the embodiments of the invention are essentially code segments that perform necessary tasks. The program or code segment can be stored in a processor-readable storage medium or device that can be downloaded via a transmission medium or communication link in the form of computer data signals embodied in a carrier wave. The processor-readable storage device can include any medium capable of storing information including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment can be downloaded via a computer network such as the Internet, intranets, etc. Any of a variety of centralized or distributed data processing architectures can be employed. The programming instructions can be implemented as a number of individual programs or subroutines, or can be integrated into many other aspects of the system described herein. In one embodiment, the control system supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0078] Medical tools delivered via the flexible elongated devices or catheters disclosed herein may include, for example, image capture probes, biopsy instruments, laser ablation fibers and / or other surgical, diagnostic, or therapeutic tools. Medical tools may include end effectors having a single working component, such as scalpels, blunt blades, optical fibers, electrodes, and / or the like. Other end effectors may include, for example, forceps, graspers, scissors, clamp applicators, and / or the like. Other end effectors may further include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, and / or the like. Medical tools may include image capture probes comprising a stereoscopic or single-view camera for capturing images, including video images. Medical tools may additionally accommodate cables, linkages, or other actuation controls (not shown) extending between proximal and distal ends to controllably bend the distal end of the medical device 304. Detailed descriptions of steerable instruments are found in U.S. Patent No. 7,316,681 (filed October 4, 2005) (disclosing "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (disclosing "Passive Preload and Capstan Drive for Surgical Instruments"), both of which are incorporated herein by reference in their entirety.

[0079] The system described herein can be applied to the navigation and treatment of anatomical tissues via naturally or surgically created access pathways in any of a variety of anatomical systems, including the lungs, colon, intestines, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and / or the like.

[0080] Note that the presented processes and displays may be inherently unrelated to any particular computer or other device. Various general-purpose systems can be used in accordance with the teachings and programming herein, or it may be demonstrated that a more specialized device can be constructed to perform the described operations. The necessary structures for various such systems will be shown as elements in the claims. Furthermore, embodiments of the invention are described without reference to any particular programming language. It will be understood that various programming languages ​​can be used to implement the teachings of the invention as described herein.

[0081] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are illustrative only and not limiting to the broader invention, and that embodiments of the invention are not limited to the specific structures and arrangements shown and described, as various other modifications can be made by those skilled in the art.

Claims

1. A medical system, the medical system comprising: Display system; User input device; and A control system communicatively coupled to the display system and the user input device, the control system being configured to: The display system displays image data corresponding to the three-dimensional anatomical region; The user input device receives a first user input to generate a first curve in the three-dimensional anatomical region. The user input device receives a second user input to generate a second curve in the three-dimensional anatomical region. as well as The anatomical boundary defined by the first curve and the second curve is determined, and the anatomical boundary indicates the surface of the anatomical structure in the three-dimensional anatomical region.

2. The medical system of claim 1, wherein the anatomical boundary is determined from an intermediate curve between the first curve and the second curve.

3. The medical system according to claim 1, wherein the control system is further configured to: The anatomical boundaries are displayed using the image data via the display system.

4. The medical system of claim 3, wherein the anatomical boundaries are displayed on the image data in an overlay manner via the display system.

5. The medical system of claim 1, wherein the control system is further configured to: The user input device receives third user input to generate a third curve in the three-dimensional anatomical region. Adjust the anatomical boundaries defined by the first curve, the second curve, and the third curve; and The adjusted anatomical boundaries are displayed using the image data via the display system.

6. The medical system of claim 1, wherein the control system is further configured to provide user-selectable options for providing the first user input and the second user input in a hand-drawn or line-based format.

7. The medical system of claim 1, wherein the anatomical boundary corresponds to a three-dimensional surface mesh comprising a plurality of vertices.

8. The medical system of claim 7, wherein the control system is configured to determine the plurality of vertices, determining the plurality of vertices comprising: Each of the first and second curves is resampled to an equal number of sampling points; Spline fitting is performed between pairs of sampling points at matching locations along the first and second curves to generate multiple splines; as well as Each of the plurality of splines is resampled to generate the plurality of vertices.

9. The medical system of claim 7, wherein the control system is configured to determine the plurality of vertices, determining the plurality of vertices comprising: Fit the three-dimensional spline surface to the first curve and the second curve; as well as The three-dimensional spline surface is resampled to generate the plurality of vertices.

10. The medical system of claim 1, wherein the control system is configured to determine the anatomical boundaries based on an intensity gradient associated with the image data.

Images