System and method for moving medical tools with targets in visual or robotic system for higher throughput

Abstract

Visualization and robotic systems and methods utilize pre-operative three-dimensional (3D) images of patient motion and intra-operative real-time patient motion information to show a target moving relative to a medical tool or control a robotic medical tool in real-time to track the target while biopsy or treating the target. These systems and methods involve: receiving a pre-operative 3D image of patient motion; displaying a guidance or control robotic tool of a robotic tool for navigating the medical tool in the vicinity of the target based on information from a position sensor disposed on the medical tool; tracking intraoperative 3D patient motion using a motion sensor disposed on the patient; determining 3D target motion based on the preoperative 3D image and the tracked patient motion; and controlling the medical tool with the robotic tool to track the 3D target motion, or to display the 3D target motion relative to the medical tool.

Description

Technical Field

[0001] The present disclosure relates to the field of visualization and navigation of a medical tool, such as a biopsy tool or an ablation tool, relative to a target and tracking the real-time motion of the target during navigation or treatment of the target using the medical tool. Background Art

[0002] There are several commonly used medical methods (such as endoscopy or minimally invasive surgery) for treating various diseases affecting organs, including liver, brain, heart, lungs, gallbladder, kidney and bone. Typically, clinicians use one or more imaging methods such as magnetic resonance imaging (MRI), ultrasound imaging, computed tomography (CT) or fluoroscopy to identify the area of ​​interest in the patient's body and the target for biopsy or treatment and navigate to the area of ​​interest and the target. In some operations, preoperative scanning can be used for target identification and intraoperative guidance. In some cases, real-time imaging or intraoperative imaging may also be required to obtain a more accurate and current image of the target area and the intracavitary medical tool for biopsy or treatment of tissue in the target area. In addition, it may be necessary to display the current position of the medical device relative to the target and the real-time image data around it to navigate the medical device to the target in a safe and accurate manner (e.g., without causing damage to other organs or tissues). However, real-time, intraoperative imaging may expose the patient to unnecessary and / or potentially unhealthy amounts of X-ray radiation.

[0003] Endoscopic methods have been demonstrated to be useful for navigating to regions of interest within a patient's body, and in particular regions within the body's luminal network, such as the lungs. To enable endoscopic methods, and more specifically bronchoscopic methods in the lungs, endobronchial navigation systems have been developed that use previously acquired MRI data or CT image data to generate a three-dimensional (3D) rendering, model, or volume of a particular body part, such as the lungs.

[0004] The resulting volume generated from the MRI scan or CT scan is then used to create a navigation plan to facilitate advancement of a navigation catheter (or other suitable medical tool) through the bronchoscope and the branches of the patient's bronchus to the area of ​​interest. A positioning or tracking system, such as an electromagnetic (EM) tracking system, may be used in conjunction with, for example, the CT data to facilitate guiding the navigation catheter through the branches of the bronchus to the area of ​​interest. In some cases, the navigation catheter may be positioned within an airway adjacent to the area of ​​interest of the branching cavity network or within an airway within the area of ​​interest to provide access for one or more medical tools.

[0005] However, a 3D volume of a patient's lungs generated from a previously acquired scan (such as a CT scan) may not provide a sufficient basis for accurately guiding a medical device or tool to a target during a navigation procedure. In some cases, inaccuracies are caused by deformation of the patient's lungs during surgery relative to the lungs when the previously acquired CT data was acquired. This deformation (CT to body divergence) can be caused by many different factors, including, for example, changes in the body when transitioning between sedated and non-sedated states, bronchoscope changes in the patient's posture, bronchoscope pushing on tissue, different lung volumes (e.g., CT scans were acquired during inspiration, while navigation was performed during exhalation), different bed positions, different days, etc. This deformation can result in significant movement of the target, making it difficult to align the medical tool with the target in order to safely and accurately biopsy or treat the target tissue.

[0006] Therefore, systems and methods are needed to address patient motion during in vivo navigation and biopsy or treatment procedures. In addition, in order to safely and accurately navigate a medical tool to a remote target and perform a biopsy or treatment on the remote target using a medical tool by a surgical robotic system or a clinician using a guidance system, the system should track the target during the motion of the patient's body (e.g., the movement of the chest during the patient's breathing) while minimizing the patient's exposure to intraoperative X-ray radiation. Summary of the invention

[0007] The technology of the present disclosure generally relates to systems and methods for showing movement of a target relative to a medical tool or controlling a robotic medical tool in real time to track a target while biopsy or treating the target using preoperative three-dimensional (3D) images of the patient's motion and intraoperative real-time patient motion information.

[0008] In one aspect, the present disclosure provides a method of controlling a medical tool to track 3D motion of a target in a patient. The method includes receiving a preoperative three-dimensional (3D) image of a patient including motion of the target. The method also includes: navigating the medical tool near the target based on information from a position sensor disposed on the medical tool; receiving patient motion information; tracking intraoperative 3D motion of the patient based on the patient motion information, thereby generating tracked patient motion; and determining the 3D motion of the target in the patient based on the preoperative 3D image and the tracked patient motion. The method also includes controlling the medical tool to track the 3D motion of the target.

[0009] Specific implementations of the method may also include one or more of the following features. The preoperative 3D image may be a computed tomography (CT) image, a cone beam computed tomography (CBCT) image, or a magnetic resonance imaging (MRI) image. Patient motion information may be received from an electromagnetic (EM) motion sensor or an anesthesia machine. The preoperative 3D image may be captured using functional respiratory imaging (FRI). Determining the 3D motion of a target in a patient may include registering the preoperative 3D image with the tracked patient motion.

[0010] In another aspect, the present disclosure provides an intracavitary navigation method. The intracavitary navigation method includes: receiving a preoperative 3D image of a patient's motion including a target; and displaying guidance for navigating a medical tool near the target based on the preoperative 3D image and information from a position sensor disposed on the medical tool. The intracavitary navigation method also includes receiving patient motion information and tracking the patient's motion based on the patient motion information, thereby generating tracked patient motion. The intracavitary navigation method also includes determining the 3D motion of a target in the patient based on the preoperative 3D image and the tracked patient motion, and displaying the 3D motion of the target relative to the tip of the medical tool.

[0011] The specific implementation of the intracavity navigation method may also include one or more of the following features. The intracavity navigation method may include capturing a preoperative 3D image during the patient's breathing cycle. The intracavity navigation method may include displaying an indicator of at least one direction in which the medical tool is navigated to reach the target. The intracavity navigation method may include segmenting the target from the preoperative 3D image, thereby generating a segmented target, and determining the position of the target in a reference frame of the preoperative 3D image based on the segmented target. The intracavity navigation method may include registering the preoperative 3D image to the tracked patient motion.

[0012] In yet another aspect, the present disclosure provides a robotic intracavitary navigation system. The robotic intracavitary navigation system includes: a robotic arm that holds and navigates a medical tool; an electromagnetic (EM) field generator that generates an electromagnetic field; a first EM sensor that is disposed at the tip of the medical tool; and one or more second EM sensors that are disposed on a patient. The robotic intracavitary navigation system also includes a processor and a memory having instructions stored thereon, which, when executed by the processor, cause the processor to receive a preoperative 3D image of the patient's motion and track the navigation of the medical tool toward a target using the first EM sensor. The instructions, when executed by the processor, may also cause the processor to track the patient's motion intraoperatively using one or more second EM sensors disposed on the patient, thereby generating tracked patient motion. The instructions, when executed by the processor, may also cause the processor to determine the 3D motion of a target in the patient based on the preoperative 3D image and the tracked patient motion, and use the first EM sensor and the 3D motion of the target to control the alignment of the medical tool with the target during the patient's motion.

[0013] The specific implementation of the robot intracavitary navigation system may include one or more of the following features. Preoperative 3D images can be captured during at least one breathing cycle of the patient. The instructions, when executed by the processor, can cause the processor to control the robot arm to navigate the medical tool toward the target during the patient's movement. One or more second EM sensors can be disposed on the patient's chest and can track the movement of the patient's chest during at least one breathing cycle. The instructions, when executed by the processor, further cause the processor to control the robot arm to navigate the medical tool through the patient's cavity network. The medical tool can be an extended working channel or a biopsy tool.

[0014] In yet another aspect, the present disclosure provides an intracavitary navigation system. The intracavitary navigation system includes: an electromagnetic (EM) field generator that generates an electromagnetic field; a first EM sensor that is disposed at the tip of a medical tool; one or more second EM sensors that are disposed on a patient's chest; and a display. The intracavitary navigation system also includes a processor and a memory having instructions stored thereon that, when executed by the processor, cause the processor to perform the following operations: receiving a preoperative 3D image of the patient's motion; using the first EM sensor to track the navigation of the medical tool toward a target; using one or more second EM sensors disposed on the patient's chest to track the patient's motion during surgery, thereby generating tracked patient motion; determining a 3D motion of a target in the patient based on the preoperative 3D image and the tracked patient motion; and displaying the target and the tip of the medical tool relative to the target on the display using the first EM sensor and the 3D motion of the target during the patient's motion.

[0015] The specific implementation of the intracavitary navigation system may include one or more of the following features. The instructions, when executed by the processor, may cause the processor to display an indicator of at least one direction in which to navigate the medical tool to reach the target. When executed by the processor, the instructions may cause the processor to segment the target from the preoperative 3D image. When executed by the processor, the instructions may cause the processor to register the preoperative 3D image to the 3D motion of the target.

[0016] The details of one or more aspects of the present disclosure are set forth in the following drawings and description. Other features, objectives, and advantages of the techniques described in the present disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various aspects of the present invention are described below with reference to the accompanying drawings, in which:

[0018] Figure 1 is a diagram of a system for navigating to a target via a cavity network according to the present disclosure;

[0019] Figure 2 is a flow chart of an example of a method for visualizing a medical tool for tracking a target during motion of a patient according to the present disclosure;

[0020] Figure 3 is a screen shot showing an example of a navigation user interface for a medical tool tip tracking a moving target according to the present disclosure;

[0021] Figure 4 is a block diagram illustrating a robotic surgical system;

[0022] Figure 5 It is used to control Figure 4 A system block diagram of a robotic surgical control system of a robotic surgical system;

[0023] Figure 6 is a flow chart of an example of a method for controlling a robotic arm to orient and advance a catheter relative to a moving target during respiration of a patient according to the present disclosure; and

[0024] Figure 7 is a diagram of a system for visualizing or controlling a medical tool relative to a target during real-time navigation and use of the medical tool and during movement of a patient in accordance with the present disclosure. DETAILED DESCRIPTION

[0025] Clinicians can use, for example, fluoroscopic imaging systems to visualize intraoperative navigation of a medical tool (e.g., a biopsy tool) and confirm the placement of the medical tool after the medical tool has been navigated to a desired location (e.g., close to a target). However, although fluoroscopic images show high-density objects such as metal tools, bones, and large soft tissue objects such as a heart, fluoroscopic images may not clearly show small soft tissue objects of interest such as lesions. In addition, fluoroscopic images are two-dimensional projections. Therefore, X-ray volume reconstruction is required to enable identification of soft tissue objects and navigation of medical tools to these objects.

[0026] There are several solutions to provide volume reconstruction. One solution is a CT imaging system that combines multiple X-ray projections from known, calibrated X-ray source positions into a volume through an algorithm, in which soft tissue is more visible. For example, a CT imaging system can be used with iterative scanning during a procedure to provide guidance through the body until the medical tool reaches the target. This is a cumbersome procedure because it requires several complete CT scans, a dedicated CT room, and blind navigation between scans. In addition, due to high levels of ionizing radiation, each scan requires staff to leave the room and expose the patient to the radiation. Another solution is a cone beam CT imaging system. However, cone beam CT machines are expensive, and like CT imaging systems, can only provide blind navigation between scans, require multiple iterative navigations, and require staff to leave the room. In some cases, the benefits of CT imaging and fluoroscopic imaging can be combined, for example, by using preoperative CT imaging and intraoperative fluoroscopic imaging to help clinicians or surgical robots navigate medical tools to targets, including small soft tissue objects. However, ideally, the use of CT imaging and fluoroscopic imaging should be minimized to minimize human exposure to X-ray radiation. Aspects of the present disclosure are directed to minimizing human exposure to X-ray radiation.

[0027] In an electromagnetic navigation procedure, planning, registration, and navigation (which may involve navigation of a locatable guide within an extended working channel) are performed to ensure that a medical tool (e.g., a biopsy tool) reaches a target (e.g., a lesion) along a planned path so that a biopsy or treatment of the target can be performed. After the navigation phase, a fluoroscopic image can be captured and used in a local registration process to reduce CT to body divergence. After the local registration process, the locatable guide can be removed from the extended working channel, and a medical device (e.g., a biopsy tool) can be introduced into the extended working channel and navigated to the target to perform a biopsy or treatment of the target (e.g., a lesion).

[0028] In the process of navigating a medical device to a target, a clinician may use a live 2D fluoroscopic view to visualize the position of the medical tool relative to the target. Although the medical tool may be visible in the live fluoroscopic view, some targets (e.g., lesions) may not be visible in the live fluoroscopic view. In addition, the user interface used to advance or navigate the medical tool toward the target does not provide sufficient information about the medical tool relative to the moving target, including when the medical tool is close to or aligned with the target.

[0029] Biopsy yield is tracked as a metric to evaluate different biopsy systems and methods. For example, computed tomography (CT) image-guided transthoracic needle aspiration (TTNA) has a biopsy yield of 90% or higher. Intracavitary biopsies hover between approximately 75% and 90%. However, intracavitary biopsies at least provide the option of staging the patient, for example, by sampling lymph nodes. Recently, techniques such as tools in lesions, target coverage, and CT-to-body divergence correction have been used to increase biopsy yield. However, during biopsy sampling, the target remains stationary during coverage, even though the target moves in 3D space in the lung.

[0030] The present disclosure features systems and methods that move a medical tool (e.g., the distal tip of a biopsy catheter) with a target in real time on a display of a user interface or via control of a robotic arm holding the medical tool to ensure that the medical tool can safely and accurately biopsy or treat the target tissue. For example, the systems and methods of the present disclosure move the biopsy catheter with the target to ensure that the biopsy catheter collects samples of the target rather than samples of other tissues, because collecting samples of other tissues may cause complications such as pneumothorax. In the case of a biopsy procedure, moving the distal tip of the biopsy catheter with the target can achieve a higher biopsy yield.

[0031] These systems and methods relate to capturing pre-procedural or pre-operative 3D images, such as pre-operative CT images, during patient motion, such as during one or more respiratory cycles. For example, the systems and methods can use aspects of functional respiratory imaging (FRI). This allows 3D motion information of the target to be input into an intracavitary navigation system and used by the intracavitary navigation system to guide a clinician or a robotic arm to navigate and control a medical tool to safely and accurately biopsy or treat the target. Pre-procedural data with FRI together with real-time data from sensors on the patient and tracking of the respiratory cycle will allow accurate understanding of 3D target movement in the chest.

[0032] Once the intraluminal catheter is at the target location, tracking can enable the target to be shown moving relative to the catheter (e.g., a static catheter), or control of a robotic arm and / or end effector to manipulate the catheter in real time to track the target while taking a biopsy or performing treatment. In the case of a biopsy procedure, this would allow a sample of the target to be obtained for each biopsy.

[0033] According to aspects of the present disclosure, visualization and / or robotic control of in vivo navigation of a medical tool (eg, a biopsy tool) toward a moving target (eg, a lesion) may be part of a larger workflow of a navigation system (such as an electromagnetic navigation system). Figure 1 is a perspective view of an example of a system for facilitating navigation of a medical tool (e.g., a biopsy tool) to a soft tissue target via an airway of a lung. The system 100 can optionally be configured to generate a three-dimensional (3D) reconstruction of a target area from a 2D fluoroscopic image. Where the system 100 is configured to generate a 3D reconstruction, intraoperative 2D fluoroscopic images can be captured only during critical portions of the procedure, e.g., to confirm placement of the medical tool in the target in order to minimize human exposure to X-ray radiation. The system 100 can be further configured to facilitate access of the medical tool to the target area and to determine the position of the medical tool relative to the target by using electromagnetic navigation bronchoscopy (ENB).

[0034] One aspect of the system 100 is a software component for reviewing computed tomography (CT) image data that has been acquired separately from the system 100. Review of the CT image data allows a user to identify one or more targets, plan a path to the identified targets (planning phase), navigate the catheter 102 to the targets using a user interface (navigation phase), and confirm placement of the sensor 104 relative to the targets. One such EMN system is the ELECTROMAGNETIC NAVIGATION SYSTEM currently sold by Medtronic PLC. The target may be a tissue of interest identified by examining the CT image data during the planning phase. After navigation, a medical tool (such as a biopsy tool or other tool) may be inserted into the catheter 102 to obtain a tissue sample from tissue located at or near the target.

[0035] like Figure 1 As shown, catheter 102 is part of a catheter guide assembly 106. In practice, catheter 102 is inserted into a bronchoscope 108 to access the luminal network of a patient P. Specifically, catheter 102 of catheter guide assembly 106 can be inserted into a working channel of bronchoscope 108 for navigation through the luminal network of the patient. A locatable guide 110 including an electromagnetic (EM) sensor 104 is inserted into catheter 102 and locked in place so that sensor 104 extends a desired distance beyond the distal tip of catheter 102. The position and orientation of sensor 104 relative to a reference coordinate system within the electromagnetic field, and thus the distal portion of catheter 102, can be derived. Catheter guide assembly 106 is currently marketed by Medtronic under the trade name EM-110. Surgical Suite or EDGETM Surgical kits are marketed and sold and are believed to be usable with the present disclosure.

[0036] The system 100 generally includes: an operating table 112 configured to support a patient P; a bronchoscope 108 configured to be inserted into the airway of the patient P through the mouth of the patient P; a monitoring device 114 coupled to the bronchoscope 108 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 108); a positioning or tracking system 114 including a positioning module 116, a patient motion sensor 118, and a transmitter pad 120, which may include a plurality of markers; and a computer system 122 including software and / or hardware for facilitating identification of a target, path planning to a target, navigation of a medical tool to a moving target, and / or confirmation and / or determination of placement of a catheter 102 or a suitable medical tool relative to a target. The computer system 122 may be similar to Figure 7 The workstation 701 and may be configured to perform operations including Figure 2 and Figure 6 The method of the present disclosure including the method of .

[0037] An imaging system 124 capable of acquiring fluoroscopic or X-ray images or videos of patient P is optionally included in some aspects of the system 100. Images, image sequences, or videos captured by the imaging system 124 can be stored within the imaging system 124 or transmitted to the computer system 122 for storage, processing, and display. In addition, the imaging system 124 can be moved relative to the patient P so that images can be acquired from different angles or perspectives relative to the patient P to create a 2D image sequence, such as a video. The posture of the imaging system 124 relative to the patient P and when capturing the images can be estimated via markers incorporated with the transmitter pad 120. The markers are positioned under the patient P, between the patient P and the operating table 112, and between the patient P and the radiation source or sensing unit of the imaging system 124. The markers incorporated with the transmitter pad 120 can be two separate elements that can be coupled in a fixed manner or alternatively can be manufactured as a single unit. The imaging system 124 may include a single imaging system or more than one imaging system. As Figure 1 As shown, imaging system 124 may include a fluoroscopic imaging system that is used solely to confirm placement of a medical tool proximate a target prior to biopsy or treatment of the target.

[0038] Computer system 122 can be any suitable computer system including a processor and a storage medium, wherein the processor is capable of executing instructions stored on the storage medium. Computer system 122 may further include a database configured to store patient data, a preoperative CT data set (e.g., a preoperative CT image captured according to functional respiratory imaging (FRI)), a navigation plan, a fluoroscopic data set optionally including fluoroscopic images and videos, optionally fluoroscopic 3D reconstruction, and any other such data. Although not explicitly shown, computer system 122 may include input of a preoperative CT data set, optional fluoroscopic images / videos, and other data described herein, or may be otherwise configured to receive a preoperative CT data set, optional fluoroscopic images / videos, and other data described herein. In addition, computer system 122 includes a display configured to display a graphical user interface. Computer system 122 may be connected to one or more networks, through which one or more databases may be accessed.

[0039] Regarding the planning phase, the computer system 122 uses previously acquired CT image data to determine the patient's regular motion (e.g., motion caused by breathing), uses the same or different previously acquired CT image data to generate and view a three-dimensional model or rendering of the airway of the patient P, so that targets on the three-dimensional model can be identified (automatically, semi-automatically, or manually), and allows the determination of a path through the airway of the patient P to reach tissues located at and around the target. More specifically, the CT images acquired from the previous CT scans are processed and assembled into a three-dimensional CT volume, which is then used to generate a three-dimensional model of the airway of the patient P. The three-dimensional model can be displayed on a display associated with the computer system 122, or in any other suitable manner. Using the computer system 122, various views of the three-dimensional model or enhanced two-dimensional images generated by the three-dimensional model are presented. The enhanced two-dimensional images can have certain three-dimensional capabilities because they are generated from three-dimensional data. The three-dimensional model can be manipulated to facilitate the identification of targets on the three-dimensional model or two-dimensional image, and the selection of a suitable path through the airway of the patient P to enter the tissue located at the target can be performed. Once a selection is made, the path plan, the three-dimensional model, and the images derived therefrom can be saved and exported to the navigation system for use during the navigation phase. One such planning software is currently sold by Medtronic Planning kit.

[0040] With respect to the navigation phase, registration of the images and the navigation path are performed using a six-degree-of-freedom electromagnetic positioning or tracking system 114 or other suitable system for determining position, but other configurations are also contemplated. The tracking system 114 includes a tracking module 116, a patient motion sensor 118, which may also be used as a patient motion sensor, and a transmitter pad 120 (including markers). The tracking system 114 is configured for use with the locatable guide 110, in particular the EM sensor 104. As described above, the locatable guide 110 and the EM sensor 104 are configured for insertion through the catheter 102 into the airway of the patient P (with or without the bronchoscope 108), and may be selectively locked relative to each other via a locking mechanism.

[0041] The transmitter pad 120 is positioned below the patient P. The transmitter pad 120 generates an electromagnetic field around at least a portion of the patient P, within which the position of the patient motion sensor 118 and the EM sensor 104 can be determined using the tracking module 116. A second EM sensor 126 can also be incorporated into the end of the catheter 102. The second EM sensor 126 can be a five-degree-of-freedom sensor or a six-degree-of-freedom sensor. One or more of the patient motion sensors 118 are attached to the chest of the patient P or to an appropriate location on the patient's body that optimizes the sensing of the patient's motion. The six-degree-of-freedom coordinates of the patient motion sensor 118 are sent to a computer system 122 (which includes appropriate software), in which they are used to calculate a patient reference coordinate system. The six-degree-of-freedom coordinates of the patient motion sensor 118 are also sent to the computer system 122, in which they are used to track the real-time motion of the patient, which may be caused by the patient's breathing cycle (e.g., inhalation and exhalation). Registration is typically performed to coordinate the positions of the three-dimensional model and two-dimensional image from the planning phase with the airway of the patient "P" as viewed through the bronchoscope 108 and to allow accurate knowledge of the position of the EM sensor 104 when the navigation phase is performed, even in portions of the airway that cannot be reached by the bronchoscope 108.

[0042] The recording of the position of the patient P on the transmitter pad 120 can be performed by moving the EM sensor 104 through the airway of the patient P. More specifically, the transmitter pad 120, the patient motion sensor 118, and the tracking system 114 are used to record data related to the position of the EM sensor 104 as the locatable guide 110 moves through the airway. The shape caused by the position data is compared with the internal geometry of the passage of the three-dimensional model generated in the planning stage, and based on the comparison, the positional correlation between the shape and the three-dimensional model is determined, for example, using software on the computer system 122. Among other things, the software identifies non-tissue spaces (e.g., cavities filled with air) in the three-dimensional model. The software aligns or registers an image representing the position of the sensor 104 with the three-dimensional model and / or a two-dimensional image generated by the three-dimensional model, based on the recorded position data and the assumption that the locatable guide 110 is still positioned in the non-tissue space in the airway of the patient P. Alternatively, manual registration techniques may be employed by navigating the bronchoscope 108 with the EM sensor 104 to a pre-specified location in the lungs of the patient P and manually associating the images from the bronchoscope with the model data of the three-dimensional model.

[0043] Although described herein with respect to an EMN system using EM sensors, the present disclosure is not limited thereto and may be used in conjunction with flexible sensors, ultrasonic sensors, or other suitable motion sensors. Additionally, the methods described herein may be used in conjunction with a robotic system such that a robotic actuator drives a catheter 102, a bronchoscope 108, or other medical tool near a target. Examples of robotic systems are described in Figure 4 and Figure 5 Shown in the middle.

[0044] After registering the patient P to the image data and the path planning, a user interface is displayed in the navigation software, which sets the path that the clinician should follow to reach the target. As depicted in the user interface, once the catheter 102 has been successfully navigated to the vicinity of the target, the locatable guide 110 can be unlocked and removed from the catheter 102, thereby leaving the catheter 102 in place as a guide channel for guiding medical tools (including but not limited to optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation tools), laser probes, cryogenic probes, sensor probes, and aspiration needles) to the target. The medical tool can then be inserted through the catheter 102 and navigated to the target or a specific area adjacent to the target.

[0045] Prior to inserting the medical tool through the catheter 102, a local registration process may optionally be performed for each target to reduce CT to body divergence. During the capture phase of the local registration process, a sequence of fluoroscopic images may be captured and acquired via the imaging system 124, optionally by the user and according to directions displayed via the computer system 122. A fluoroscopic 3D reconstruction may then be generated via the computer system 122. The generation of the fluoroscopic 3D reconstruction is based on the sequence of fluoroscopic images and the projection of the structure of the markers incorporated with the transmitter pad 120 onto the sequence of fluoroscopic images. One or more slices of the 3D reconstruction may then be generated based on the preoperative CT scan and via the computer system 122. The 3D reconstruction and one or more slices of the fluoroscopic 3D reconstruction may then optionally be displayed to the user on a display simultaneously via the computer system 122. The slices of the 3D reconstruction may be presented on a user interface in a scrollable format, wherein the user is able to scroll through the slices continuously.

[0046] During the marking phase of the local registration process, the clinician may be guided to identify and mark the target while using the slices of the 3D reconstruction as a reference. The user may also be guided to identify and mark the navigation catheter tip in the sequence of fluoroscopic 2D images. The offset between the target position and the navigation catheter tip may then be determined or calculated via the computer system 122. The offset may then be used via the computer system 122 to correct the position and / or orientation of the navigation catheter on the display relative to the target, and / or to correct the registration between the three-dimensional model in the target area and the tracking system 114, and / or to generate a local registration between the three-dimensional model in the target area and the fluoroscopic 3D reconstruction.

[0047] In the optional confirmation stage of the local registration process, the fluoroscopic 3D reconstruction is displayed in a confirmation screen. The confirmation screen may include a slider that can be selected and moved by the user to review a video loop of the fluoroscopic 3D reconstruction that shows the marked target and navigation catheter tip from different viewing angles. After confirming that there are marks on the target and navigation catheter tip throughout the video, the clinician can select the "Accept" button, at which point the local registration process ends and the position of the navigation catheter is updated. The clinician can then use, for example, the navigation view in the peripheral navigation screen to fine-tune the alignment of the navigation catheter with the target before starting the endoscopic procedure.

[0048] After the local registration process, the clinician or the robotic arm can insert the medical tool into the catheter 102 and advance the medical tool toward the target. When advancing the medical tool toward the target, the clinician can view a user interface screen of a 3D medical tool tip view including a 3D model of the target, wherein the 3D model of the target is moved according to the movement of the target determined from the preoperative CT scan of the patient's movement and the movement sensed by the patient's motion sensor. This user interface screen not only allows the clinician to view the medical tool in real time, but also allows the clinician to view whether the medical tool is aligned with the moving target. The user interface screen can also provide a graphical indication of whether the medical tool is aligned with the target in three dimensions. For example, when the medical tool is aligned with the target in three dimensions, the user interface shows the target coverage in a first color (e.g., green). On the other hand, when the medical tool is not aligned with the target in three dimensions, the user interface shows the target coverage in a second color (e.g., orange or red) different from the first color.

[0049] Figure 2 Flowchart of an example of a method for visualizing a medical tool (e.g., a catheter) relative to a moving target during a patient's breathing cycle. At box 202, a preoperative 3D image of the patient's motion is received. The preoperative 3D image can be a computed tomography (CT) image, a cone beam computed tomography (CBCT) image, or a magnetic resonance imaging (MRI) image. These images can be captured based on functional respiratory imaging (FRI). FRI can include acquiring a low-dose, high-resolution CT scan of the patient's lungs, segmenting the CT scan to obtain a 3D geometry, and performing functional simulations, for example, using computational fluid dynamics (CFD) to quantify airflow through the lungs, which provides detailed information about the movement of all parts of the lungs during one or more breathing cycles, etc. At box 204, an EM sensor disposed on the medical tool (e.g., disposed on the tip of the medical tool as described herein) is used to track the navigation of the medical tool toward the target.

[0050] At block 206, the patient's motion is tracked intraoperatively in real time using the patient motion information. The patient motion information may be received from one or more motion sensors disposed on and / or in the patient's chest, from an anesthesia machine, or from any suitable system for tracking patient motion or breathing cycles. The one or more motion sensors may be provided by Figure 1 The tracking system 114 shown tracks EM sensors in real time, for example, Figure 1 The patient motion sensor 118 of the system 100 is used. At block 208, a 3D motion of the object is determined based on the preoperative 3D image and the tracked patient motion. Determining the 3D motion of the object in the patient may include registering the preoperative 3D image with the tracked patient motion.

[0051] After determining the 3D motion of the target at block 208, at block 210, the tip of the medical tool is displayed relative to the moving target during the patient's motion using the EM sensor disposed on the medical tool and the 3D motion of the target. In various aspects, method 200 may include displaying in the user interface an indicator of at least one direction to navigate the medical tool to reach the target. In one example, the tip of the medical tool and the target may be displayed in a Figure 3 In the user interface 300.

[0052] Figure 3 Shown with Figure 3 The peripheral navigation screen 301 is associated with a "Peripheral Navigation" tab of the user interface 300 of the embodiment of the present invention. The peripheral navigation screen 301 includes a local CT view 302, a 3D navigation catheter tip view 304, a 3D map view 306, and a bronchoscope view 308. The peripheral navigation screen 301 also includes a local registration user control 303 that enables a user to apply local registration and / or restart local registration. The user interface 300 also includes a "Central Navigation" tab 311 and a "Target Alignment" tab 312, which can be selected separately to perform central navigation or target alignment, respectively. The tip portion 305 of the medical tool is displayed relative to a moving target 307. In various aspects, the tip portion 305 can be displayed as stationary, while the target 307 can move according to the determined 3D motion of the target.

[0053] Figure 4 is a block diagram illustrating a robotic surgical system 400 according to aspects of the present disclosure. The robotic surgical system 400 includes a first robotic arm 402 and a second robotic arm 404 attached to robotic arm bases 406 and 408, respectively. The first robotic arm 402 and the second robotic arm 404 include a first end effector 416 and a second end effector 418, respectively. The end effectors 416, 418 may include robotic manipulators or grippers suitable for operating endoscopic catheters and medical tools of the present disclosure. The first end effector 416 operates one or more tools 412, including a biopsy tool and / or a flexible endoscope or bronchoscope. The second end effector 418 operates a sheath or catheter 410, which may include one or more channels for receiving and guiding one or more medical tools 412. The robotic surgical system 400 may further include an electromagnetic (EM) generator 414 configured to generate an EM field that is sensed by an EM sensor incorporated into or disposed on the medical tool, and an EM patient motion sensor 421 disposed on and / or within the patient. In various aspects, the EM generator 414 may be embedded in the operating table 415 or may be incorporated into a pad that may be placed between the operating table 415 and the patient 411.

[0054] The first and second robotic arms 402 , 404 can be controlled to align the end effectors 416 and 418 so that the proximal end portion of the catheter 410 is distal to the proximal end portion of the one or more tools 412 , and so that the one or more tools 412 remain axially aligned with the catheter 410 .

[0055] In one aspect, the first robotic arm 402 inserts the catheter 410 through an endotracheal tube (not shown) in, for example, the mouth of the patient 411, and into the bronchial system of the patient 411. The second robotic arm 404 then inserts one or more tools 412 (e.g., a biopsy tool) through the catheter 102 to a target within the bronchial system of the patient 411. The first robotic arm 402 and the second robotic arm 404 can move the catheter 410 and the one or more tools 412 (e.g., a biopsy tool) axially relative to each other and into and out of the patient 411 under the control of a surgeon (not shown) at a console (not shown).

[0056] The navigation phase may include advancing the catheter 410 into the patient 411 along with one or more tools 412, and then advancing the one or more tools 412 beyond the distal end of the catheter 410 to reach a desired destination, such as a target. Other navigation modes may be used, such as by using a guide wire through the working channel of the catheter 410. The surgeon may use a visual guidance modality or a combination of visual guidance modalities to assist navigation and perform biopsy procedures, such as fluoroscopy, video, computed tomography (CT) or magnetic resonance imaging (MRI). In various aspects, one or more tools 412 are deployed through the longitudinally aligned working channel within the catheter 410 to perform biopsy procedures and any other desired procedures. In various aspects, the robot arms 402, 404 may include three joints 401 and three arm segments 405. In other aspects, the robot arms 402, 404 may include more or less than three joints 401 and three arm segments 405.

[0057] Figure 5 It is used to control Figure 4 The robotic control system 500 includes a control system 510 that controls the robotic surgical system 400. For example, the control system 510 may perform the operations described herein. Figure 6Method 600. The control system 510 can interface with a display 522, a user controller 525, and an endoscopic or bronchoscope camera 526. The control system 510 can be coupled to the robotic surgical system 400 directly or indirectly, for example, via wireless communication. The control system 510 includes a processor 512, a memory 514 coupled to the processor 512, a random access memory (RAM) 516 coupled to the processor 512, and a communication interface 518 coupled to the processor 512. The processor 512 may include one or more hardware processors. The control system 510 can be a fixed computer such as a personal computer, or a portable computer such as a tablet computer. Alternatively, the control system 510 can be incorporated into one of the robotic arm bases 406, 408. The control system 510 can also interface with a user controller 525, which can be used by a surgeon to control the robotic arm system 524 to perform a biopsy procedure.

[0058] Those skilled in the art will appreciate that the memory 514 may be a computer-readable storage medium that is accessible by the processor 512. That is, the computer-readable storage medium may include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, the computer-readable storage medium may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the processor 512.

[0059] The application stored in the memory 514 may cause the display 522 to present a user interface (not shown) when executed by the processor 512. The user interface may be configured to present the bronchoscopic image from the bronchoscope camera 526 to the user. Optionally, the user interface may be further configured to guide the user in selecting a target by identifying and marking the target in the image data, etc.

[0060] The communication interface 518 can be configured to connect to a network, such as a local area network (LAN) composed of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. The communication interface 518 can be used to connect between the control system 510 and the bronchoscope camera 526. The communication interface 518 can also be used to receive image data and path planning data from the memory 514. The communication interface 518 can also be coupled or communicated with one or more patient motion sensors 421 and / or anesthesia machine 530 to receive patient motion information, such as the patient's breathing cycle. The control system 510 may also include an input device (not shown), which can be any device with which a user can interact with the control system 510, such as a mouse, a keyboard, a foot pedal, a touch screen, and / or a voice interface. The control system 510 may also include an output module (not shown), which may include any connection port or bus, such as a parallel port, a serial port, a universal serial bus (USB), or any other similar connection port known to those skilled in the art.

[0061] Figure 6 is used to control one or more robot arms (e.g. Figure 4 Flowchart of an example of a method for directing and advancing a catheter relative to a moving target during a patient's breathing by using a robotic arm 402, 404) to direct and advance a catheter relative to a moving target during a patient's breathing. At box 602, a preoperative 3D image of the patient's motion is received, for example, by a computer system 122. The preoperative 3D image can be captured by a 3D imaging system (e.g., a CT imaging system) during one or more breathing cycles of the patient. At box 604, a first EM sensor is used to track the navigation of a medical tool toward a target, and the first EM sensor can be disposed on the tip of the medical tool. The medical tool can be an extended working channel or a biopsy tool. At box 606, the patient's motion information is used to track the patient's motion during the operation. Patient motion information can be received from one or more second EM sensors disposed on and / or in the patient, from an anesthesia machine, or from any suitable system for tracking patient motion or breathing cycles. In various aspects, one or more second EM sensors are disposed on and / or in the patient's chest and are configured to track the motion of the patient's chest during one or more breathing cycles. At box 608, the 3D motion of the target is determined based on the preoperative 3D image and the tracked patient motion. The 3D motion of the object may be represented by a functional model and may be determined by registering the pre-operative 3D images to the 3D motion of the object.In various aspects, method 600 may include segmenting the object from a sequence of pre-operative 3D images.

[0062] After determining the 3D motion of the target at box 608, at box 610, during patient movement using the first EM sensor and the 3D motion of the target, the orientation of the tip of the catheter is controlled, for example via a robotic arm, to align with or track the target. In various aspects, the computer system 122 can determine whether a medical tool (e.g., an extended working channel or a biopsy catheter) is not aligned with the target. If the computer system 122 determines that the medical tool is not aligned with the target, the computer system 122 can generate an alarm. In various aspects, the computer system 122 can control the robotic arm to navigate the medical tool through the patient's cavity network.

[0063] Reference now Figure 7 , which is configured to include Figure 2 and Figure 6 Schematic diagram of a system 700 for use in conjunction with the methods of the present disclosure, including methods of the present disclosure. System 700 may include a workstation 701 and optionally include an imaging system 715, such as a fluoroscopic imaging system and / or a CT imaging system for capturing preoperative 3D images. In some aspects, workstation 701 may be coupled to imaging system 715 directly or indirectly (e.g., via wireless communication). Workstation 701 may include memory 702, processor 704, display 706, and input device 710. Processor 704 may include one or more hardware processors. Workstation 701 may optionally include an output module 712 and a network interface 1008. Memory 702 may store application programs 718 and image data 714. Application programs 718 may include instructions executable by processor 704 for performing operations including Figure 2 and Figure 6 The method of the present disclosure including the method of .

[0064] Application 718 may further include user interface 716. Image data 714 may include preoperative CT image data, fluoroscopic image data, or fluoroscopic 3D reconstruction data. Processor 704 may be coupled to memory 702, display 706, input device 710, output module 712, network interface 708, and imaging system 715. Workstation 701 may be a fixed computer system such as a personal computer, or a portable computer system such as a tablet computer. Workstation 701 may be embedded in multiple computers.

[0065] The memory 702 may include any non-transitory computer-readable storage medium for storing data and / or software including instructions executable by the processor 704 and controlling the operation of the workstation 701, and in some aspects, may also control the operation of the imaging system 715. The imaging system 715 may be used to capture a sequence of preoperative CT images of a portion of the body (e.g., lungs) when the portion of the patient's body (e.g., lungs) moves, such as when the lungs move during a respiratory cycle. Optionally, the imaging system 715 may include a fluoroscopic imaging system that captures a sequence of fluoroscopic images, generates a fluoroscopic 3D reconstruction based on the sequence of fluoroscopic images, and captures a live 2D fluoroscopic view to confirm the placement of a medical tool. In one aspect, the memory 702 may include one or more storage devices, such as solid-state storage devices, such as flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, the memory 702 may include one or more mass storage devices connected to the processor 704 via a mass storage controller (not shown) and a communication bus (not shown).

[0066] Although the description of computer-readable media contained herein refers to solid-state storage devices, it should be understood by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 704. That is, computer-readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and can be accessed by the workstation 701.

[0067] When the application 718 is executed by the processor 704, the display 706 may be caused to present a user interface 716. The user interface 716 may be configured to present a single screen to the user that includes a three-dimensional (3D) view of a 3D model of a target from the perspective of the tip of a medical tool, a real-time two-dimensional (2D) fluoroscopic view showing the medical tool, and a target marker corresponding to the 3D model of the target overlaid on the real-time 2D fluoroscopic view. The user interface 716 may be further configured to display the target marker in different colors depending on whether the tip of the medical tool is aligned with the target in three dimensions.

[0068] The network interface 708 can be configured to connect to a network (such as a local area network (LAN) composed of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet). The network interface 708 can be used to connect between the workstation 701 and the imaging system 715. The network interface 708 can be further used to receive image data 714. The input device 710 can be any device through which a user can interact with the workstation 701, such as, for example, a mouse, a keyboard, a foot pedal, a touch screen, and / or a voice interface. The output module 712 may include any connection port or bus, such as, for example, a parallel port, a serial port, a universal serial bus (USB), or any other similar connection port known to those skilled in the art. Based on the above and with reference to the various drawings, it will be understood by those skilled in the art that certain modifications may also be made to the present disclosure without departing from the scope of the present disclosure.

[0069] Although detailed aspects are disclosed herein, the disclosed aspects are merely examples of the present disclosure that may be embodied in various forms and in various aspects. For example, aspects of visualization and robotic systems in conjunction with electromagnetic navigation systems are disclosed herein; however, the visualization and robotic systems and methods may also be applied to other navigation or tracking systems or methods known to those skilled in the art. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the present disclosure in almost any suitable specific structure.

[0070] Although various aspects of the disclosure have been shown in the accompanying drawings, it is not intended to limit the disclosure thereto, as it is intended that the disclosure be as broad as the art allows and that the specification should be read in the same manner. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various aspects. Those skilled in the art will be able to envision other modifications within the scope and spirit of the claims appended hereto.

Claims

1. A method, comprising: receiving a preoperative three-dimensional (3D) image of the patient's motion including the target; navigating the medical tool in the vicinity of the target based on information from a position sensor disposed on the medical tool; Receive patient movement information; tracking intraoperative 3D motion of the patient based on the patient motion information, thereby generating tracked patient motion; determining a 3D motion of the object in the patient based on the preoperative 3D image and the tracked patient motion; as well as The medical tool is controlled to track the 3D movement of the target.

2. The method according to claim 1, wherein: The preoperative 3D image is a computed tomography (CT) image, a cone beam computed tomography (CBCT) image, or a magnetic resonance imaging (MRI) image.

3. The method according to claim 1, wherein: The patient motion information is received from an electromagnetic (EM) motion sensor or an anesthesia machine. 4 . The method of claim 1 , further comprising wherein the pre-operative 3D image is captured using functional respiratory imaging (FRI).

5. The method according to claim 1, wherein: Determining the 3D motion of the target in the patient includes registering the pre-operative 3D image with the tracked patient motion.

6. An intracavity navigation method, the intracavity navigation method comprising: receiving a preoperative 3D image of the patient's motion including the target; displaying guidance for navigating a medical tool near the target based on the preoperative 3D image and information from a position sensor disposed on the medical tool; Receive patient movement information; tracking the patient's movement based on the patient motion information, thereby generating tracked patient motion; determining a 3D motion of the object in the patient based on the preoperative 3D image and the tracked patient motion; as well as The 3D motion of the target relative to the tip of the medical tool is displayed. 7 . The intracavitary navigation method of claim 6 , further comprising capturing the preoperative 3D image during a breathing cycle of the patient.

8. The intracavitary navigation method according to claim 6, further comprising displaying an indicator of at least one direction in which the medical tool is navigated to reach the target.

9. The intracavity navigation method according to claim 6, further comprising: segmenting the object from the preoperative 3D image, thereby generating a segmented object; as well as A position of the object in a reference frame of the pre-operative 3D image is determined based on the segmented object.

10. The intracavitary navigation method of claim 6, further comprising registering the preoperative 3D image with the tracked patient motion.

11. A robot intracavity navigation system, the robot intracavity navigation system comprising: a robotic arm configured to hold and navigate a medical tool; an electromagnetic (EM) field generator configured to generate an electromagnetic field; a first EM sensor disposed at a tip of the medical tool; one or more second EM sensors disposed on the patient; processor; as well as A memory having instructions stored therein, wherein when the instructions are executed by the processor, the processor is caused to perform the following operations: receiving a preoperative 3D image of the patient's motion; tracking navigation of the medical tool toward a target using the first EM sensor; intraoperatively tracking movement of the patient using the one or more second EM sensors disposed on the patient, thereby generating tracked patient movement; determining a 3D motion of the object in the patient based on the preoperative 3D image and the tracked patient motion; as well as The medical tool is controlled using the first EM sensor and the 3D motion of the target to align with the target during movement of the patient.

12. The robotic intracavity navigation system according to claim 11, wherein: The pre-operative 3D images are captured during at least one breathing cycle of the patient.

13. The robotic intracavity navigation system according to claim 11, wherein: The instructions, when executed by the processor, further cause the processor to control the robotic arm to navigate the medical tool toward the target during patient movement.

14. The robotic intracavity navigation system according to claim 11, wherein: The one or more second EM sensors are disposed on the chest of the patient and are configured to track the movement of the chest of the patient during at least one breathing cycle.

15. The robotic intracavity navigation system according to claim 11, wherein: The instructions, when executed by the processor, further cause the processor to control the robotic arm to navigate the medical tool through the patient's luminal network.

16. The robotic intracavity navigation system according to claim 11, wherein: The medical tool is an extended working channel or a biopsy tool.

17. An intracavity navigation system, comprising: an electromagnetic (EM) field generator configured to generate an electromagnetic field; a first EM sensor disposed at a tip of a medical tool; one or more second EM sensors disposed on the chest of the patient; monitor; processor; as well as A memory having instructions stored therein, wherein when the instructions are executed by the processor, the processor is caused to perform the following operations: receiving a preoperative 3D image of the patient's motion; tracking navigation of the medical tool toward a target using the first EM sensor; tracking movement of the patient intraoperatively using the one or more second EM sensors disposed on the chest of the patient, thereby generating tracked patient movement; determining a 3D motion of the object in the patient based on the preoperative 3D image and the tracked patient motion; as well as The target and the tip of the medical tool relative to the target are displayed on the display during movement of the patient using a first EM sensor and the 3D movement of the target.

18. The intracavitary navigation system according to claim 17, wherein: The instructions, when executed by the processor, further cause the processor to display an indicator of at least one direction in which to navigate the medical tool to reach the target.

19. The intracavitary navigation system according to claim 17, wherein: The instructions, when executed by the processor, further cause the processor to segment the object from the pre-operative 3D image.

20. The intracavitary navigation system according to claim 17, wherein: The instructions, when executed by the processor, further cause the processor to register the pre-operative 3D image to the 3D motion of the target.

Images