System and method for determining one or more phases of anatomical movement

Abstract

Systems and methods for determining one or more stages of anatomical movement are provided. A catheter positioned in an anatomical element may be tracked using a tracker configured to track a pose of the catheter. Sensor data corresponding to a plurality of pose information of the catheter may be received from the tracker. One or more stages of motion of the anatomical element may be determined based on the plurality of pose information. An image may be received at a target phase of the one or more phases.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 468,976, filed May 25, 2023, and U.S. Non-Provisional Patent Application No. 18 / 656,752, filed May 7, 2024, the entire disclosure of which is incorporated herein by reference. Background Technology

[0003] This disclosure relates in general to tracking the movement of anatomical elements, and more specifically to tracking the movement of anatomical elements to determine one or more stages of anatomical movement.

[0004] Surgical robots can assist surgeons or other healthcare providers in performing surgical procedures, or they can autonomously complete one or more surgical procedures. Imaging can be used by healthcare providers for diagnostic and / or therapeutic purposes. Patient anatomy can change over time, especially after medical implants are placed within the patient's anatomy. Summary of the Invention

[0005] Examples of aspects of this disclosure include:

[0006] A system for determining one or more stages of anatomical movement according to at least one embodiment of the present disclosure includes: a catheter positioned in an anatomical element; a tracker for tracking the pose of the catheter; a processor; and a memory storing data for processing by the processor, the data causing the processor, when processed, to: track the catheter; receive sensor data from the tracker corresponding to a plurality of pose information of the catheter; determine one or more stages of movement of the anatomical element based on the plurality of pose information; receive an image at a target stage in the one or more stages; and display the image.

[0007] Any aspect of this article, wherein tracking the catheter includes tracking the speed of the catheter.

[0008] In any aspect of this article, at least one of the tracker, the catheter, or a combination of the tracker and the catheter is configured such that at least one of the motion caused by the fluid flow in the anatomical element is enhanced, reduced, or increased.

[0009] In any aspect of this document, the memory stores additional data for processing by the processor, which, when processed, causes the processor to: receive bioelectrical signals, wherein the determination of the one or more phases of motion is further based on the bioelectrical signals.

[0010] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: receive a respiration cycle signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0011] Any of the aspects herein, wherein the anatomical element comprises a heart, and the one or more phases correspond to one or more phases of a heartbeat.

[0012] Any of the aspects herein, wherein the one or more phases of the heartbeat comprise at least a diastolic motion of the heart and a systolic motion of the heart.

[0013] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: average the sensor data for a target phase of the one or more phases; and apply the average to subsequent tracking at the target phase.

[0014] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to determine a patient state based on the motion of the anatomical element.

[0015] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: determine movement and velocity of the catheter; and determine an environmental state of the catheter based on the movement and the velocity.

[0016] Any of the aspects herein, wherein the tracker comprises at least one of: an inertial measurement unit tracker, an electromagnetic tracker, or a combination of the inertial measurement unit tracker and the electromagnetic tracker.

[0017] Any of the aspects herein, wherein the tracker comprises an imaging device, and the sensor data comprises image data, wherein the plurality of pose information is derived from the image data.

[0018] A system for determining one or more phases of anatomical movement in accordance with at least one embodiment of the present disclosure comprises: a catheter positioned in an anatomical element; a tracker for tracking a pose of the catheter; a processor; and a memory storing data for processing by the processor that, when processed, causes the processor to: track the catheter; receive sensor data corresponding to a plurality of pose information of the catheter from the tracker; and determine one or more phases of motion of the anatomical element based on the plurality of pose information.

[0019] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to receive an image at a stage of the one or more stages and display the image.

[0020] Any of the aspects herein, wherein the tracker comprises at least one of an inertial measurement unit tracker or an electromagnetic tracker.

[0021] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to determine a patient state based on the motion of the anatomical element.

[0022] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: determine a movement and a velocity of the catheter; and determine an environmental state of the catheter based on the movement and the velocity.

[0023] A system for associating one or more images with one or more cycles of motion of an anatomical element, in accordance with at least one embodiment of the present disclosure, comprises: a processor; and a memory storing data for processing by the processor that, when processed, causes the processor to: receive a plurality of images over a period of time; receive sensor data corresponding to a plurality of pose information of a catheter, the plurality of pose information corresponding to pose information of the catheter over the period of time to produce; determine one or more first motion phases of a first motion of the anatomical element based on the plurality of pose information; determine one or more second motion phases of a second motion of the anatomical element based on the plurality of pose information; and determine, from the plurality of images, a subset of images corresponding to a target first motion phase of the one or more first motion phases of the first motion and a target second motion phase of the one or more second motion phases of the second motion.

[0024] Any of the aspects herein, wherein the subset of images comprises a first subset of images, and wherein the memory stores further data for processing by the processor that, when processed, causes the processor to determine, from the plurality of images, a second subset of images corresponding to another target first motion phase of the one or more first motion phases of the first motion and another target second motion phase of the one or more second motion phases of the second motion.

[0025] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to reconstruct a three-dimensional model based on an intersection of the first subset of images and the second subset of images.

[0026] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to display at least one of: the first subset of images, the second subset of images, or a combination of the first subset of images and the second subset of images.

[0027] Any of the aspects herein, wherein the first motion corresponds to a cardiac cycle and the second motion corresponds to a respiratory cycle.

[0028] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to receive respiratory information of the patient over the time period, wherein determining the one or more second motion phases of the second motion is also based on the respiratory information.

[0029] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to obtain and display a subsequent plurality of images at the target first motion phase and the target second motion phase over a subsequent time period.

[0030] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: average sensor data at a target phase of the one or more phases; and apply the average to subsequent tracking at the target phase.

[0031] Any of the aspects herein, wherein the sensor data comprises at least one of image data, accelerometer data, pose data, or motion data.

[0032] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: receive a bioelectric signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0033] Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: receive a respiratory cycle signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0034] Any aspect in combination with any one or more other aspects.

[0035] Any one or more of the features disclosed herein.

[0036] Any one or more of the features generally disclosed herein.

[0037] Any one or more of the features generally disclosed herein can be combined with any one or more of the other features generally disclosed herein.

[0038] Any of the aspects / features / embodiments in combination with any one or more other aspects / features / embodiments.

[0039] Any one or more of the aspects or features disclosed herein are used.

[0040] It should be understood that any of the features described herein can be claimed in combination with any other feature as described herein, regardless of whether the features are from the same described embodiment.

[0041] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and in the description below. Other features, objects, and advantages of the technologies described in this disclosure will be apparent from the description and drawings, and from the claims.

[0042] The phrases "at least one", "one or more", and "and / or", as used herein, are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C" and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together. When each of the above expressions is recited with respect to elements such as X, Y, and Z, or classes of elements such as X1-Xn, Y1-Ym, and Z1-Zo, the phrase is intended to mean individual elements from the set of X, Y, and Z, combinations of elements from the same class (e.g., X1and X2), as well as combinations of elements from two or more classes (e.g., Y1and Zo).

[0043] The term "a" or "an" entity refers to one or more of that entity. As such, the terms "a" (or "an"), "one or more" and "at least one" can be used interchangeably herein. It is also to be noted that the terms "comprising", "comprise" and "comprised of" can be used interchangeably.

[0044] The foregoing is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. The summary is neither an extensive overview of the disclosure nor an introduction to the disclosure, its various aspects, embodiments and configurations. Its sole purpose is to present some select concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later. As will be appreciated, other aspects, embodiments and configurations of the disclosure can utilize one or more of the features set forth above or in the detailed description below.

[0045] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided herein. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings incorporated herein and forming a part of the specification, illustrate several aspects of the present disclosure. Together with the description, the drawings explain the principles of the present disclosure. In the drawings, like reference numerals indicate like elements through the several figures. Additional features and advantages will be apparent to those of ordinary skill in the art upon examining the following drawings, detailed description, and claims.

[0047] Figure 1 is a block diagram of a system in accordance with at least one embodiment of the present disclosure;

[0048] Figure 2 is a schematic diagram of a system in accordance with at least one embodiment of the present disclosure;

[0049] Figure 3 is a flow diagram in accordance with at least one embodiment of the present disclosure;

[0050] Figure 4 is a flow diagram in accordance with at least one embodiment of the present disclosure;

[0051] Figure 5 is a flow diagram in accordance with at least one embodiment of the present disclosure; and

[0052] Figure 6 is a flow diagram in accordance with at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] It should be understood that various aspects disclosed herein can be combined in different combinations than the combinations specifically presented in the descriptions and accompanying drawings. It should also be understood that certain actions or events can be performed in different sequences than as described, and / or omitted, combined, or added to as appropriate, e.g., depending on the example or embodiment. Furthermore, although certain aspects of the present disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of the present disclosure can be performed by a combination of units or modules associated with, e.g., computing devices and / or medical devices.

[0054] In one or more examples, the described methods, processes, and techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively, or in addition, the functions can be implemented using a machine learning model, neural network, artificial neural network, or a combination thereof (instructions, alone or in combination). The computer-readable medium can include a non-transitory computer-readable medium that corresponds to a tangible medium such as a data storage medium (e.g., a RAM, a ROM, an EEPROM, a flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0055] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000 series processors, Nvidia GeForce RTX 3000 series processors, AMD Radeon RX 5000 series processors, AMD Radeon RX 6000 series processors, or any other graphics processing unit), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein can refer to any of the foregoing structures or any other physical structure suitable for implementing the described techniques. Additionally, the techniques can be fully implemented in one or more circuits or logic elements.

[0056] Before any embodiments of the present disclosure are explained in detail, it is to be understood that the present disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the accompanying drawings. The present disclosure is capable of other embodiments and of being practiced or being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure can be illustrated using examples. Unless otherwise expressly stated, the use of one or more examples (which can be indicated by, for example, "for example," "by way of illustration," "such as," or similar language) is not intended to limit the scope of the present disclosure.

[0057] The terms proximal and distal are used in the present disclosure in their conventional medical meaning, proximal being closer to an operator or user of the system and further away from a surgical area of interest in or on a patient's body, and distal being closer to the surgical area of interest in or on a patient's body and further away from an operator or user of the system.

[0058] The use of a catheter (or any other device or component for placement) to place an implantable device, such as a cardiac device, requires accurate positioning of the catheter within the patient's anatomy. O-arm or computed tomography (CT) imaging can be used to help confirm placement of an implantable device (e.g., a cardiac device) and a catheter, however, O-arm or CT imaging devices can be cumbersome and also expose the patient and surgical personnel to radiation. A physician placing a catheter would benefit from a method that is quick and easy to use to gather information about the position and / or orientation of the catheter without exposing the patient and surgical personnel to radiation.

[0059] At least one embodiment of the present disclosure provides systems and methods for providing clinical information about the position of a catheter using spatial temporal information from a catheter tracked using a tracker, such as, for example, an electromagnetic (EM) tracker or an inertial measurement unit (IMU) tracker. EM and / or IMU sensors are placed on, for example, a cardiac catheter to provide a constant stream of information about the position and orientation of the heart. More specifically, as the heart beats in a cyclical manner, the position of the catheter changes (as the catheter moves), and this motion can be recorded and plotted. The motion can then be mapped to the cardiac cycle, similar to an electrocardiogram (EKG or ECG) that can be mapped to the cardiac cycle. In addition, the velocity of the device can be derived from the tracker. Thus, knowing the velocity and position of the device can be used to improve placement of an implantable device, such as a cardiac device.

[0060] Systems and methods using position and orientation (e.g., pose information) and / or velocity information from a tracked catheter can also be used to improve EM and / or IMU tracking accuracy by timing the acquisition of EM and / or IMU signals to specific times (e.g., phases) in the cardiac signal. For example, EM trackers can be most accurate during diastole and can have too much motion during systole to accurately navigate the EM tracker. Timing the EM and / or IMU data acquisition to match the less motioned levels of the cycle can improve the accuracy of the tracking. In another example, as the EM tracker measures the heartbeat cycle, the tracking or navigation system can average, correlate, or otherwise combine the EM and / or IMU samples to accurately navigate the tracker throughout the cycle, including during the systolic motion.

[0061] Systems and methods can also be used to replace or improve information retrieved from an EKG by showing the systolic and diastolic movement of the heart measured at a specific anatomical location. The provided systems can be simplified and use fewer components because the catheter does not need to be connected to an EKG to obtain heart information because the heart information can be determined using EM and / or IMU tracking and software to derive the heart information from the EM and / or IMU data.

[0062] Systems and methods can also be used to time the acquisition of images, such as ultrasound images or CT images, at the target phase of the heartbeat cycle to reduce motion artifacts.

[0063] Systems and methods can also be used to correlate or identify specific anatomical locations. For example, a catheter will experience more motion in the center of the blood flow of the aorta than near the sidewall of the aorta. In another example, the motion in the ventricle should be different than the motion in the atrium. Thus, specific regions of the heart wall can have different motion patterns that can help confirm that the tip of the catheter is at the target location.

[0064] Systems and methods can also be used to identify or correlate specific disease states. For example, low motion or velocity can indicate low blood flow rates.

[0065] Systems and methods can also be used to ensure that the motion of an implantable device or cardiac device does not exceed the known limits of the device. For example, EM and / or IMU data can be used to determine if force and / or torque limits are exceeded on a cardiac device (or any implantable device) and, if so, the user can be warned. Similarly, the environment or motion of a catheter can be checked or confirmed before a cardiac device, such as, for example, a micro device or a heart valve, is delivered. Normal values of the motion of a tracked catheter can be confirmed before a cardiac device is delivered and the user can be warned if the value is abnormally large or small.

[0066] In addition to the heartbeat cyclic motion, the shape, size, and location of the heart and other organs dynamically change with the beating of the heart and the breathing of the lungs. In other words, the cardiac cycle and the respiratory cycle can affect the heart and / or other organs. This movement can affect patient imaging, such as for example during ultrasound imaging, as sequential or consecutive images correspond to different time periods / stages of the cardiac cycle and the respiratory cycle. This correspondence results in a mismatch between consecutive images during image modeling (e.g., ultrasound 2D, 3D, 4D, 2D+Doppler) as the location / size / shape of the organs will not match through consecutive images. Registration can also be affected as the location / size / shape of the organs does not match from consecutive images to volumetric images. Additionally, planning and execution of a surgical procedure using image-guided surgery can be affected as the anatomy from previous imaging can not match the current location / size / shape of the organs.

[0067] At least one embodiment of the present disclosure provides systems and methods for dynamically synchronizing acquired images with the cardiac cycle and the respiratory cycle to create multiple sets of non-consecutive images and image models with matching stages (e.g., matching organ location, size, and shape). These systems and methods can be applied to, for example, images (such as ultrasound images) that are captured in real-time and from different perspectives / locations relative to the organs.

[0068] The cardiac cycle and the respiratory cycle can be measured using, for example, current patient monitoring (e.g., respiratory monitoring / control, cardiac monitoring / control, ECG, EKG, etc.), patient and / or instrument tracking using EM tracking (e.g., magnetic inductive coils, etc.), patient and / or instrument tracking using IMU tracking (e.g., accelerometers, gyroscopes, etc.), a computing device for interpreting an ECG graph dynamically shown on a screen of some ultrasound imaging devices, or a combination thereof. In some cases, a combination of respiratory monitoring (e.g., tidal volume) and patient electromagnetic or inertial tracking or both (e.g., chest or abdominal movement) can be used to determine movement during the respiratory cycle. Similarly, a combination of cardiac monitoring (e.g., ECG) and patient electromagnetic or inertial tracking or both (e.g., anterior / posterior lateral movement) can be used to determine movement during the cardiac cycle. In other cases, a combination of cardiac monitoring (e.g., ECG) and instrument electromagnetic or inertial tracking or both (e.g., a tracked catheter within the heart) can be used to determine movement during the cardiac cycle.

[0069] The cardiac cycle and the respiratory cycle can be synchronized with imaging, such as for example, ultrasound imaging using direct communication between multiple systems (e.g., navigation systems, imaging devices, computing devices, etc.), indirect communication between multiple systems, and / or correlation between multiple systems.

[0070] Systems and methods for dynamically synchronizing acquired images with the cardiac and respiratory cycles can be used to improve imaging, modeling, registration, and image guidance. For example, imaging, modeling, registration, and image guidance can be improved using: average images with selected phases, correlated images throughout the cycle, display images at selected phases (e.g., strobing or freezing ultrasound images, dynamic ultrasound strobing effects for freezing images at a particular point in the cardiac cycle (e.g., a target phase) while allowing dynamic movement of instruments relative to the anatomy, hiding ultrasound images taken outside of the target phase, highlighting selected phase ultrasound images and darkening out-of-phase ultrasound images, matching phases of a phased CT or MRI, etc.), reconstructed images and models with selected phases or through the cycle motion (e.g., reconstructing a 3D heart model at a selected heart phase, reconstructing a region of interest cycle motion using speckle ultrasound tracking, etc.), and / or adjusted regions of interest or targets for cycle motion (e.g., synchronizing electronic control of ultrasound imaging fan direction or depth or both with heart motion, etc.).

[0071] Systems and methods for dynamically synchronizing acquired images with the cardiac and respiratory cycles can also be used to improve EM and / or IMU tracking. EM and / or IMU tracking can be improved using, for example: average samples within the same phase, correlated samples throughout the cardiac and respiratory cycles, display of tracking at selected target phases (e.g., strobing or freezing tracked position and orientation, dynamic ultrasound strobing effects for freezing tracking at a particular point in the cardiac cycle while allowing dynamic movement of structures relative to, for example, instruments, hiding tracking performed outside of the selected phase, highlighting selected phase tracking and darkening out-of-phase tracking, etc.), tracking reconstructed with selected phases or through the cycle motion (e.g., reconstructing tracked heart wall motion throughout the cardiac cycle, etc.), and / or adjusted regions of interest or targets for cycle motion.

[0072] Embodiments of the present disclosure provide technical solutions to one or more of the following problems: (1) determining cardiac and / or respiratory motion, (2) accurately placing implantable devices in a patient, (3) determining environmental conditions of implantable devices, and (4) improving displayed images and image reconstruction.

[0073] Turning first to Figure 1FIG. 1 shows a block diagram of a system 100 in accordance with at least one embodiment of the present disclosure. The system 100 can be used to track movement of anatomical elements and determine one or more phases of motion of the anatomical elements, and / or perform one or more other aspects of the methods disclosed herein. The system 100 includes a computing device 102, one or more imaging devices 112, a catheter 136 (which can be tracked with a catheter tracker 138), a robot 114, a navigation system 118, a database 130, and / or a cloud or other network 134. Systems in accordance with other embodiments of the present disclosure can include more or fewer components than the system 100. For example, the system 100 can not include one or more of the imaging devices 112, the robot 114, the navigation system 118, one or more components of the computing device 102, the database 130, and / or the cloud 134.

[0074] The computing device 102 includes a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices in accordance with other embodiments of the present disclosure can include more or fewer components than the computing device 102.

[0075] The processor 104 of the computing device 102 can be any processor described herein or any similar processor. The processor 104 can be configured to execute instructions stored in the memory 106 that can cause the processor 104 to implement one or more computational steps with or based on data received from the imaging devices 112, the catheter tracker 138, the robot 114, the navigation system 118, the database 130, and / or the cloud 134.

[0076] The memory 106 can be or include RAM, DRAM, SDRAM, other solid state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer-readable data and / or instructions. The memory 106 can store information or data that can be used to complete any step of, for example, the methods 300, 400, 500, and / or 600 or any other method described herein. The memory 106 can store, for example, instructions that support one or more functions of the robot 114 and / or machine learning models. For example, the memory 106 can store content (e.g., instructions and / or machine learning models) that, when executed by the processor 104, enable image processing 120, tracking 122, registration 124, and / or reconstruction 128.

[0077] Image processing 120 enables the processor 104 to process image data of an image (received from, for example, the imaging device 112, an imaging device of the navigation system 118, or any imaging device) in order to, for example, identify information about anatomical elements and / or objects depicted in the image. This information can include, for example, identification of hard tissue and / or soft tissue, a boundary between hard tissue and soft tissue, a boundary of hard tissue and / or soft tissue, identification of a surgical tool or reference marker 140, etc. The image processing 120 can identify hard tissue, soft tissue, and / or a boundary of hard tissue and / or soft tissue, for example, by determining a difference in color or grayscale of image pixels or a contrast between color or grayscale of image pixels. For example, a boundary between hard tissue and soft tissue can be identified as a contrast between lighter pixels and darker pixels.

[0078] Tracking 122 enables the processor 104 to track the catheter tracker 138, which can include, for example, an EM and / or IMU tracker. The tracking 122 can enable the processor 104 to receive sensor data from the catheter tracker 138, for example, and determine pose information (e.g., position and orientation) of the catheter 136 from the sensor data. The tracking 122 can also enable the processor 104 to determine a velocity of the catheter 136 from the sensor data, for example.

[0079] Registration 124 enables the processor 104 to correlate an image with another image. The registration 124 can enable the processor 104 to also correlate identified anatomical elements and / or objects in one image with identified anatomical elements and / or objects in another image. The registration 124 can enable obtaining and measuring information about anatomical elements and objects (e.g., the catheter 136).

[0080] Reconstruction 128 enables the processor 104 to generate a three-dimensional (3D) representation of one or more anatomical elements and / or one or more objects based on one or more images received by the processor 104. Generating the 3D representation by the processor 104 can include determining a surface representation or a virtual boundary of the one or more anatomical elements and / or the one or more objects depicted in the images received by the processor 104 and based on corresponding image pose information. More specifically, in some embodiments, each image can be positioned adjacent to another image based on the respective corresponding image pose information, and a surface representation can be formed based on the relative positions of surfaces depicted in each image. In some embodiments, the surface representation can be a virtual mesh. The virtual mesh can include, for example, a set of polygonal faces that, when brought together, form a surface covering of the virtual object. The set of polygonal faces can be connected at their edges and vertices to define the shape of the virtual object.

[0081] In some embodiments, if provided as instructions, such content can be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, memory 106 can store other types of content or data that can be processed by processor 104 to carry out the various methods and features described herein (e.g., machine learning models, artificial neural networks, deep neural networks, etc.). Thus, although various content of memory 106 can be described as instructions, it should be understood that the functionality described herein can be implemented using instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions can cause processor 104 to manipulate data stored in memory 106 and / or received from or via imaging device 112, catheter tracker 138, robot 114, database 130, and / or cloud 134.

[0082] Computing device 102 can also include a communication interface 108. Communication interface 108 can be used to receive image data or other information from external sources, such as imaging device 112, catheter tracker 138, robot 114, navigation system 118, database 130, cloud 134, and / or any other system or component that is not part of system 100, and / or to send instructions, images, or other information to external systems or devices (e.g., another computing device 102, imaging device 112, catheter tracker 138, robot 114, navigation system 118, database 130, cloud 134, and / or any other system or component that is not part of system 100). Communication interface 108 can include one or more wired interfaces (e.g., USB ports, Ethernet ports, Firewire ports) and / or one or more wireless transceivers or interfaces configured to send and / or receive information via, for example, one or more wireless communication protocols such as 802.11a / b / g / n, Bluetooth, NFC, ZigBee, etc. In some embodiments, communication interface 108 can be used to enable device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to complete computationally intensive tasks or for any other reason.

[0083] The computing device 102 can also include one or more user interfaces 110. The user interface 110 can be or include a keyboard, a mouse, a trackball, a monitor, a television, a screen, a touchscreen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface 110 can be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Nonetheless, any required input for any step of any method described herein can be automatically generated by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 can be used to allow a surgeon or other user to modify instructions to be executed by the processor 104 in accordance with one or more embodiments of the present disclosure, and / or to modify or adjust settings of other information displayed on or corresponding to the user interface 110.

[0084] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 can utilize a user interface 110 housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 can be located proximate to one or more other components of the computing device 102, while in other embodiments, the user interface 110 can be located remote from one or more other components of the computing device 102.

[0085] The imaging device 112 can be operable to image anatomical features (e.g., bones, veins, tissue, etc.) and / or other aspects of a patient’s anatomy to produce image data (e.g., image data that depicts or corresponds to bones, veins, tissue, etc.). “Image data,” as used herein, refers to data generated or captured by the imaging device 112 that contains data in machine-readable form, graphical / visual form, and in any other form. In various examples, the image data can include data corresponding to an anatomical feature of a patient or a portion thereof. The image data can be or include pre-operative images, intra-operative images, post-operative images, or images taken independent of any surgical procedure. In some embodiments, a first imaging device 112 can be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device 112 can be used to obtain second image data (e.g., a second image) at a second time after the first time. The imaging device 112 can be capable of taking 2D images or 3D images to produce image data. The imaging device 112 can be or include, for example, an ultrasound scanner (which can include, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device that utilizes X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermal imaging camera (e.g., an infrared camera), a radar system (which can include, for example, a transmitter, a receiver, a processor, and one or more antennas), or any other imaging device 112 suitable for obtaining images of anatomical features of a patient. The imaging device 112 can be entirely contained within a single housing, or can include a transmitter / emitter and a receiver / detector located in separate housings or otherwise physically separated.

[0086] In some embodiments, the imaging device 112 can include more than one imaging device 112. For example, a first imaging device can provide first image data and / or a first image, and a second imaging device can provide second image data and / or a second image. In yet other embodiments, the same imaging device can be used to provide both first image data and second image data and / or any other image data described herein. The imaging device 112 can be used to generate a stream of image data. For example, the imaging device 112 can be configured to operate using an open shutter, or using a shutter that continuously alternates between open and closed, in order to capture successive images. For purposes of the present disclosure, image data can be considered to be successive and / or provided as a stream of image data if the image data represents two or more frames per second, unless otherwise indicated.

[0087] The catheter 136 can be any catheter 136 that can be inserted and positioned within a patient, and more particularly, can be inserted into an anatomical element of a patient. For example, the catheter 136 can be inserted into the heart of a patient, although it should be understood that the catheter 136 can be positioned in any anatomical element or region of a patient (e.g., intrathecally positioned in a spinal region of a patient, in a brain region of a patient, etc.). In some cases, the catheter 136 can be implantable. In some embodiments, the catheter 136 can enable fluid sampling from and / or delivery of a therapeutic agent to a patient.

[0088] The catheter tracker 138 can include one or more sensors disposed on the catheter 136 and configured to track a pose and / or movement of the catheter 136. The catheter tracker 138 can include, for example, EM and / or IMU sensors. In other embodiments, the catheter tracker 138 can include one or more of the following or any combination thereof: electrical components, mechanical components, electromechanical components, magnetic components, electromagnetic components, etc. For example, the catheter tracker 138 can include infrared tracking, light diffraction tracking (e.g., optical fiber), and / or impedance tracking. The catheter tracker 138 can be positioned at a tip of the catheter 136, although in other embodiments, the catheter tracker 138 can be positioned on any portion of the catheter 136. The catheter tracker 138 can be shaped to enhance, reduce, and / or increase motion caused by fluid flow in the anatomical element. For example, the catheter tracker 138 can include fins and / or baffles to alter the motion of the fluid flow.

[0089] The catheter tracker 138 can include multiple sensors, and each sensor can be positioned at the same location or a different location than any other sensor. In some embodiments, the catheter tracker 138 can include a memory for storing sensor data. In yet other examples, the catheter tracker 138 can output a signal (e.g., sensor data) to one or more sources (e.g., the computing device 102, the navigation system 114, and / or the robot 116). For example, when the catheter tracker 138 detects movement of the catheter 136, the catheter tracker 138 can transmit data to the computing device 102. Further, in some embodiments, the catheter tracker 138 can transmit data to the computing device 102 to display on the user interface 110 or otherwise notify the surgical physician or operator of a change in a property. In other embodiments, the catheter tracker 138 can alert the surgeon or operator of the change in property through an alarm, such as but not limited to a sound or a signal light display.

[0090] The robot 114 can be any surgical robot or surgical robotic system. The robot 114 can be or include, for example, the Mazor X ™Stealth® system. The robot 114 can be configured to position the imaging device 112 at one or more precise locations and orientations, and / or to return the imaging device 112 to the same location and orientation at a later point in time. The robot 114 can additionally or alternatively be configured to manipulate surgical tools (whether or not based on guidance from the navigation system 118) to accomplish or assist with surgical tasks. In some embodiments, the robot 114 can be configured to hold and / or manipulate anatomical elements during or in conjunction with a surgical procedure. The robot 114 can include one or more robotic arms 116. In some embodiments, the robotic arms 116 can include a first robotic arm and a second robotic arm, although the robot 114 can include more than two robotic arms. In some embodiments, one or more of the robotic arms 116 can be used to hold and / or manipulate the imaging device 112. In embodiments in which the imaging device 112 includes two or more physically separate components (e.g., a transmitter and a receiver), one robotic arm 116 can hold one such component, and another robotic arm 116 can hold the other such component. Each robotic arm 116 can be capable of being positioned independently of the other robotic arms. The robotic arms 116 can be controlled in a single shared coordinate space or in separate coordinate spaces. The robotic arms 116 can include one or more sensors that enable the processor 104 (or a processor of the robot 114) to determine the precise pose of the robotic arm (and of any objects or elements held by or fixed to the robotic arm) in space.

[0091] In some embodiments, reference markers (i.e., navigation markers) 140 can be placed on the robot 114 (including, e.g., on the robotic arms 116), on the imaging device 112, on the patient, or on any other objects in the surgical space. The reference markers 140 can be tracked by the navigation system 118, and the results of the tracking can be used by the robot 114 and / or by an operator of the system 100 or any of its components. In some embodiments, the navigation system 118 can be used to track other components of the system (e.g., the imaging device 112), and the system can operate without using the robot 114 (e.g., the surgeon manually manipulates the imaging device 112 and / or one or more surgical tools, e.g., based on information and / or instructions generated by the navigation system 118).

[0092] During operation, the navigation system 118 can provide navigation for the surgeon and / or the surgical robot. The navigation system 118 can be any now known or future developed navigation system, including, e.g., the Medtronic StealthStation® system. ™S8 surgical navigation system or any successor system thereto. The navigation system 118 can include one or more cameras or other sensors for tracking one or more reference markers 140, navigation trackers, or other objects positioned within the surgical theater or other room in which some or all of the systems 100 are positioned. The one or more cameras can be optical cameras, infrared cameras, or other cameras. In some embodiments, the navigation system 118 can include one or more electromagnetic sensors or trackers or inertial measurement unit trackers (e.g., catheter tracker 138). In various embodiments, the navigation system 118 can be used to track the position and orientation (e.g., pose) of the catheter 136 via the catheter tracker 138, the imaging device 112, the robot 114, and / or the robotic arm 116 and / or one or more surgical tools. More specifically, the navigation system 118 can be used to track the pose of a navigation tracker that is directly or indirectly attached in a fixed relationship to one or more of the foregoing. The navigation system 118 can include a display for displaying one or more images from an external source (e.g., the computing device 102, the imaging device 112, or other source) or for displaying images and / or video streams from the one or more cameras or other sensors of the navigation system 118. The navigation system 118 can be configured to provide guidance to a surgeon or other user of the system 100 or components thereof, to the robot 114, or to any other element of the system 100 regarding, for example, the pose of one or more anatomical elements, whether a tool is in a proper trajectory, and / or how to move the tool into a proper trajectory to perform a surgical task in accordance with a preoperative or other surgical plan.

[0093] The database 130 can store information relating one coordinate system to another (e.g., relating one or more robotic coordinate systems to a patient coordinate system and / or a navigation coordinate system). The database 130 can additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information regarding a target and / or image information regarding the anatomy of a patient at and / or proximal to a surgical site for use by the robot 114, the navigation system 118, and / or the computing device 102 or a user of the system 100); one or more images that can be used in connection with a surgical procedure completed by or with the assistance of one or more other components of the system 100; and / or any other useful information. The database 130 can be configured to provide any such information to the computing device 102 or any other device of the system 100 or any other device external to the system 100, whether directly or via the cloud 134. In some embodiments, the database 130 can be or include a portion of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0094] The cloud 134 can be or represent the Internet or any other wide area network. The computing device 102 can connect to the cloud 134 using a wired connection, a wireless connection, or both, via the communication interface 108. In some embodiments, the computing device 102 can communicate with the database 130 and / or external devices (e.g., computing devices) via the cloud 134.

[0095] The system 100, or a similar system, can be used to, for example, carry out one or more aspects of any of the methods 300, 400, 500, and / or 600 described herein. The system 100, or a similar system, can also be used for other purposes.

[0096] Turning to Figure 2 , a display 200 is shown displaying a first screen 202 and a second screen 204. The display 200 can be the same as or similar to the user interface 110. The first screen 202 displays an image of an anatomical element 212, which includes, for example, a heart 206 of a patient. The second screen 204 displays a detailed view of the heart 206 and a catheter, such as the catheter 136 positioned in the heart 206. As shown, the catheter tracker 138 includes an EM coil wrapped around a tip of the catheter 136. It should be understood that, in other embodiments, the catheter tracker 138 can include, for example, an IMU tracker or a combination of an EM tracker and an IMU tracker. It should also be understood that, in other embodiments, the catheter tracker 138 can be positioned at any location on the catheter 136. As the heart 206 beats, the catheter 136 and the catheter tracker 138 move with the heart 206, and the movement of the catheter tracker 138 can be recorded (as represented by arrows 208). A valve annulus displacement is also shown (as represented by arrows 210) and the valve annulus displacement is related to the relative displacement of the valve annulus structure as it moves during a heartbeat. Information that can be obtained from the catheter tracker 136, such as, for example, pose information, movement, velocity of the catheter tracker 136, can be useful in many situations, such as, for example, improved placement, tracking, and imaging of devices, reconstruction, display of real-time or near real-time imaging, etc., as will be discussed in detail below.

[0097] Figure 3 The method 300 is depicted that can be used, for example, to track movement of an anatomical element and determine one or more phases of anatomical movement.

[0098] The method 300 (and / or one or more steps thereof) can be carried out or otherwise performed by, for example, at least one processor. The at least one processor can be the same as or similar to the processor 104 of the computing device 102 described above. The at least one processor can be part of a robot, such as the robot 114, or part of a navigation system, such as the navigation system 118. The method 300 can also be performed using a processor other than any processor described herein. The at least one processor can perform the method 300 by executing elements stored in a memory, such as the memory 106. The elements stored in the memory and executed by the processor can cause the processor to perform one or more steps of the functions as shown in the method 300. One or more portions of the method 300 can be performed by a processor that executes any of the contents of the memory, such as the image processing 120, the tracking 122, the registration 124, and / or the reconstruction 128.

[0099] The method 300 includes tracking a catheter (step 304). The catheter can be the same as or similar to the catheter 136 and include a catheter tracker, such as the catheter tracker 138. The catheter can be positioned or disposed in an anatomical element of a patient, such as, for example, a heart, such as the heart 206. In other embodiments, the catheter can be positioned in any portion or anatomical element of a patient. The catheter tracker can be used to track the catheter as it moves, whether due to a user moving the catheter and / or due to a patient’s circulating heartbeat and / or breathing movements. The catheter tracker can include one or more sensors to sense movements and / or poses of the catheter. In some embodiments, the catheter tracker includes an EM tracker and / or an IMU tracker. The catheter tracker 138 can be tracked by, for example, a navigation system, such as the navigation system 118. In some embodiments, a processor, such as the processor 104 or a processor of the navigation system, can perform a tracking, such as the tracking 122, to track the catheter tracker.

[0100] The method 300 also includes receiving sensor data corresponding to a plurality of pose information of the catheter (step 308). The sensor data is received from the catheter tracker and can be stored in, for example, a memory, such as the memory 106, and / or a database, such as the database 130. The sensor data can be received wirelessly from the catheter tracker by, for example, a navigation system. The sensor data can be received continuously in real-time or near real-time. The sensor data can also be received at predetermined intervals or based on user input. The sensor data can be used by, for example, a processor to obtain the plurality of pose information. The pose information can include information such as a position and an orientation of the catheter, coordinates of the catheter, and the like. The sensor data can also be used to obtain other information, such as, for example, a velocity of the catheter.

[0101] The method 300 also includes determining one or more phases of motion of the anatomical element based on the plurality of pose information (step 312). The one or more phases of motion can be determined by, for example, a processor mapping or plotting movement of the catheter and identifying a repeating pattern of movement. The processor can then correlate the pattern of movement to the one or more phases of motion. For example, in embodiments in which the anatomical element comprises a heart or an organ adjacent or proximate to the heart, the repeating pattern can correspond to one or more phases of a heartbeat or cardiac cycle. More specifically, the one or more phases can include at least a diastolic motion and a systolic motion, which can be identified and correlated to the pattern of movement. The processor can also use artificial intelligence or machine learning to identify the one or more phases. In such embodiments, the processor can be provided with a plurality of training poses, and each set of training poses can be annotated to include identifying information regarding a corresponding one or more phases.

[0102] It should be appreciated that in additional or alternative embodiments, the one or more phases of motion can correspond to respiratory motion caused by a patient breathing. In such embodiments, the respiratory motion can be determined from the plurality of pose information and / or from tracking a reference marker, such as the reference marker 140, which can be external to the patient.

[0103] Determining the one or more phases (e.g., diastolic motion and / or systolic motion) is beneficial because images can be taken during a target phase (or multiple target phases) that results in higher quality images than images taken during an undesired phase, as will be described in detail below.

[0104] The method 300 also includes receiving an image at a target phase of the one or more phases (step 316). The target phase can be automatically selected by, for example, a processor of a computing device or navigation system. In other cases, the target phase can be selected by a user, such as a surgeon or other medical provider. The target phase can be selected to reduce artifacts in the image. For example, in embodiments in which the anatomical element comprises a heart, the target phase can comprise a heart diastolic phase because there is less movement associated with the heart diastolic phase than the heart systolic phase. It can also be desirable to receive one or more images at the target phase to update a 3D model at the target phase or to display the image for consistency.

[0105] The images can be received or obtained from an imaging device, such as imaging device 112, and can be stored in a memory of a computing device, such as computing device 102. The imaging device can be any imaging device, such as an MRI scanner, a CT scanner, any other X-ray based imaging device, or an ultrasound imaging device. The images can also be generated by and / or uploaded to any other component of the system, such as system 100. In some embodiments, the images can be received indirectly by any other component of the system or a node of a network to which the system is connected.

[0106] The images can be 2D images or 3D images or a set of 2D images and / or 3D images. The images can depict the patient’s anatomy or a portion thereof and can include at least a portion of an anatomical element and / or a catheter. It will be appreciated that in some embodiments, the images can also depict a reference marker, such as reference marker 140. In some embodiments, the images can also depict a plurality of anatomical elements associated with the patient’s anatomy, including incidental anatomical elements (e.g., vertebrae or other anatomical objects that will not have a surgical procedure performed thereon) in addition to the target anatomical element (e.g., a rib or other anatomical object on which a surgical procedure or surgical procedure will be performed). The images can include various features corresponding to the patient’s anatomy and / or anatomical elements (and / or portions thereof), including gradients corresponding to boundaries and / or contours of various anatomical elements depicted, different intensity levels corresponding to different surface textures of various anatomical elements depicted, combinations thereof, and the like. The images can depict any portion or region of the patient’s anatomy and can include, but are in no way limited to, one or more vertebrae, ribs, lungs, soft tissue (e.g., skin, tendons, muscle fibers, etc.), patella, clavicle, scapula, combinations thereof, and the like.

[0107] Each image can be processed by the processor using image processing, such as image processing 120, to identify anatomical elements and / or reference markers in the image. In some embodiments, feature recognition can be used to identify features of the anatomical elements and / or reference markers. For example, the contours of vertebrae, femurs, or other bones can be identified in the image. In other embodiments, the image processing algorithm can use artificial intelligence or machine learning to identify the anatomical elements and / or reference markers. In such embodiments, a plurality of training images can be provided to the processor, and each training image can be annotated to include identifying information about the reference markers and / or anatomical elements in the image.

[0108] The method 300 also includes registering the images (step 320). The processor can use a registration, such as the registration 124, to transform, map, or create a correlation between the images and / or components thereof received in step 320, which can then be used by a system, such as the system 100 or a navigation system, to convert one or more coordinates in a patient coordinate space to one or more coordinates in a robot coordinate space of a robot (e.g., the robot 114) and / or vice versa. The registration can include registering between 2D images and / or vice versa and / or between a 3D image (e.g., a CT scan) and one or more 2D images (e.g., ultrasound images) and / or vice versa.

[0109] In some embodiments, the registration can be performed one or more times intraoperatively (e.g., during a surgical procedure) to update, adjust, and / or refresh a current registration. For example, new 3D images and / or new multiple 2D images can be captured intraoperatively, and a new registration can be completed therefrom (e.g., using the preoperative 3D image and new multiple intraoperative 2D images, a new intraoperative 3D image, and new multiple 2D images, or otherwise). For example, an updated registration can be needed if the patient’s posture changes or is changed during a surgical procedure. The registration can also be performed and / or repeated for a target phase or updated target phase. For example, the registration can be completed for a first phase, and later updated for a second phase.

[0110] The method 300 also includes displaying the images (step 324). In some embodiments, after step 320, the images can be displayed on a user interface, such as the user interface 110, or a display, such as the display 200. The images can be displayed at the target phase, such as, for example, by strobing or freezing the images (which can be, for example, ultrasound images) at the target phase, using a dynamic strobing effect to freeze the images at the target phase in the heart rhythm while allowing dynamic movement of an implantable device or instrument (e.g., a catheter, a cardiac device, etc.) relative to the anatomical elements, hiding images taken outside of the target phase, highlighting the images of the target phase and darkening the images outside of the target phase, and / or matching the target phase of a 3D image (e.g., a CT or MRI) at the target phase. This selective display of the images can visually assist a user (such as a surgeon or other medical provider) during placement of, for example, an implantable device (such as a cardiac device, a stent, etc.). More specifically, by displaying the images at the target phase, the images of the anatomical elements can appear nearly stationary, such that the user can more easily maneuver the implantable device throughout the anatomical elements. Without the selective display at the target phase, the anatomical elements would appear to constantly move due to the circulatory motion, which can make it difficult to visually maneuver the implantable device through the constantly moving anatomical elements.

[0111] It should be appreciated that steps 316, 320, and / or 324 can be repeated such that more than one image can be displayed throughout a surgical procedure. In some embodiments, images can be received and displayed continuously in order to provide real-time or near real-time imaging of anatomical elements, catheters, and / or implantable devices. In other embodiments, images can be updated at predetermined time periods. In other embodiments, images can be updated based on user input.

[0112] Method 300 also includes reconstructing a 3D model (step 328). The 3D model can be reconstructed or updated based on the images received in step 316. The processor can use a reconstruction, such as reconstruction 128, to reconstruct the 3D model. As previously described, generating a 3D representation by the processor 104 can include determining a surface representation or virtual boundary of one or more anatomical elements and / or one or more objects depicted in the images received in, for example, step 316 and based on corresponding image pose information. The image pose information can be obtained from, for example, a robot supporting the imaging device, such as robot 114. In some embodiments, each image can be positioned adjacent to another image based on the respective corresponding image pose information, and a surface representation can be formed based on the relative positions of the surfaces depicted in each image. In some embodiments, the surface representation can be a virtual mesh. The virtual mesh can include, for example, a set of polygonal faces that, when brought together, form a surface covering of the virtual object. The set of polygonal faces can be connected at their edges and vertices to define the shape of the virtual object.

[0113] Similar to step 324, the 3D model can be reconstructed using images from a target phase. For example, a first image from a first angle and a second image from a second angle can be taken at the same target phase such that portions of the anatomical element reconstructed from the first image and the second image have similar positions, sizes, and shapes. In other words, if the first image and the second image were taken at different phases, the portions of the anatomical element depicted in the first image and the second image can have different positions, sizes, and / or shapes and can not match each other. Thus, it is beneficial to obtain images from the same target phase to reconstruct the 3D model. It should be appreciated that images from more than one phase can be used to reconstruct the 3D model in multiple phases.

[0114] The present disclosure encompasses embodiments of method 300 that include more or fewer steps than those described above and / or one or more steps that are different than those described above.

[0115] Figure 4 A method 400 that can be used, for example, to track movement of anatomical elements is depicted.

[0116] Method 400 (and / or one or more steps thereof) can be carried out or otherwise performed by, for example, at least one processor. The at least one processor can be the same as or similar to processor 104 of computing device 102 described above. The at least one processor can be part of a robot, such as robot 114, or part of a navigation system, such as navigation system 118. Method 400 can also be performed using a processor other than any processor described herein. The at least one processor can perform method 400 by executing elements stored in a memory, such as memory 106. The elements stored in the memory and executed by the processor can cause the processor to perform one or more steps of the functions as shown in method 400. One or more portions of method 400 can be performed by a processor that executes any of the contents of the memory, such as image processing 120, tracking 122, registration 124, and / or reconstruction 128.

[0117] Method 400 includes receiving a plurality of images over a time period (step 404). Step 404 can be the same as or similar to step 316 of method 300 described above, in which a plurality of images are obtained over a time period.

[0118] Method 400 also includes receiving sensor data corresponding to a plurality of pose information of heart motion over the time period (step 408). In embodiments in which the plurality of pose information is received from a catheter tracker, such as catheter tracker 138, of a catheter, such as catheter 136, step 408 can be the same as or similar to step 308 of method 300 described above, except that sensor data corresponding to the plurality of pose information is received over the time period.

[0119] In other embodiments, the sensor data can include data from, for example, an EKG or ECG from which one or more phases of heart motion can be determined. In some embodiments, the plurality of pose information can be determined from a combination of sensor data from a catheter tracker and / or EKG or ECG data. Sensor data and / or pose information can also be derived from image data (received from, for example, an imaging device) that includes one or more catheters within an imaging field of view of the image data.

[0120] Method 400 also includes receiving respiration information over the time period (step 412). Respiration motion can be determined from the plurality of pose information received in step 408 and / or from tracking a reference marker, such as reference marker 140 positioned on a patient. The reference marker can be external to the patient. In other embodiments, sensors, such as EM and / or IMU sensors, can be placed on or positioned in patient anatomy to track movement of, for example, chest or abdominal muscles that can correspond to respiration movement.

[0121] The method 400 also includes determining one or more phases of a first motion of the anatomical element based on the plurality of pose information (step 416). Step 416 is the same as or similar to step 312 of method 300 described above, except that the first motion can correspond to a heartbeat or cardiac cycle.

[0122] The method 400 also includes determining one or more phases of a second motion of the anatomical element based on the respiration information (step 420). Step 420 is the same as or similar to step 312 of method 300 described above, except that the second motion can correspond to a respiration or respiratory cycle. Further, the one or more phases can correspond to an inhalation or exhalation during the respiratory cycle.

[0123] The method 400 also includes determining, from the plurality of images, a first subset of images corresponding to a first phase of the first motion and a first phase of the second motion (step 424). The first subset of images can be determined by, for example, the processor determining each image corresponding to a first phase of the first motion (e.g., a first phase of a cardiac cycle) and a first phase of the second motion (e.g., a first phase of a respiratory cycle). More specifically, the processor can correlate the plurality of images taken over a time period with the plurality of pose information and respiration information over that time period. Thus, each image can be related to a phase of a cardiac cycle and a phase of a respiratory cycle based on the corresponding pose information and respiration information. Then, images of the same phase of the cardiac cycle and the respiratory cycle can be grouped together. It should be appreciated that, in some cases, the cardiac cycle and the respiratory cycle can be synchronized. In other cases, the cardiac cycle and the respiratory cycle can occur at different frequencies and can be unsynchronized. In other words, the images can correspond to different combinations of the cardiac cycle and the respiratory cycle at different image subsets.

[0124] The method 400 also includes determining, from the plurality of images, a second subset of images corresponding to a second phase of the first motion and a second phase of the second motion (step 428). Step 428 can be the same as or similar to step 424, except that the second subset of images is different than the first subset of images.

[0125] The method 400 also includes reconstructing a 3D model based on the first image subset and / or the second image subset (step 432). Step 432 can be the same as or similar to step 328 of method 300 described above. It should be appreciated that in some embodiments, the 3D model can be reconstructed based on the first image subset for the first phase of the first motion and the first phase of the second motion and the second image subset for the second phase of the first motion and the second phase of the second motion. In some embodiments, the first image subset can correspond to images collected during a particular phase of the cardiac cycle and the second image subset can correspond to images collected during a particular phase of the respiratory cycle. For example, the first image subset can correspond to a first image subset collected in a diastolic position of the heart during the cardiac cycle and the second image subset can correspond to a second image subset collected in an expiratory position of the respiratory cycle. The 3D model can then be reconstructed based on a combination of the first image subset and the second image subset.

[0126] The method 400 also includes displaying the first image subset and / or the second image subset (step 436). Step 436 can be the same as or similar to step 324 of method 300 described above. Similar to step 436, the first image subset for the first phase of the first motion and the first phase of the second motion can be displayed and the second image subset for the second phase of the first motion and the second phase of the second motion, respectively, can be displayed. In other words, multiple versions of the 3D model can be reconstructed for different phases of the first motion and / or the second motion.

[0127] It should be appreciated that any step and any combination of steps of the method 400 can be repeated. For example, steps 424 or 428 can be repeated for a third image subset or any subsequent image set. Likewise, steps 432 and / or 436 can be repeated based on a third image subset or any subsequent image set.

[0128] Embodiments of the method 400 are encompassed by the present disclosure that include more or fewer steps than those described above and / or one or more steps that are different than those described above.

[0129] Figure 5 A method 500 is depicted that can be used, for example, to improve tracking of a catheter tracker.

[0130] The method 500 (and / or one or more steps thereof) can be carried out or otherwise performed by, for example, at least one processor. The at least one processor can be the same as or similar to the processor 104 of the computing device 102 described above. The at least one processor can be part of a robot, such as the robot 114, or part of a navigation system, such as the navigation system 118. The method 500 can also be performed using a processor other than any processor described herein. The at least one processor can perform the method 500 by executing elements stored in a memory, such as the memory 106. The elements stored in the memory and executed by the processor can cause the processor to perform one or more steps of the functions as shown in the method 500. One or more portions of the method 500 can be performed by a processor that executes any of the contents of the memory, such as the image processing 120, the tracking 122, the registration 124, and / or the reconstruction 128.

[0131] The method 500 includes tracking the catheter (step 504). The step 504 can be the same as or similar to the step 304 of the method 300 described above.

[0132] The method 500 also includes receiving sensor data corresponding to a plurality of pose information of the catheter (step 508). The step 408 can be the same as or similar to the step 308 of the method 300 described above. Similarly, the sensor data and / or the pose information can also be derived from image data (received from, for example, an imaging device) that includes one or more catheters within an imaging field of view of the image data.

[0133] The method 500 also includes determining one or more phases of motion of the anatomical element based on the plurality of pose information (step 512). The step 512 can be the same as or similar to the step 312 of the method 300 described above.

[0134] The method 500 also includes averaging or combining the sensor data for the target phase (step 516). The sensor data for the target phase can be averaged by, for example, a processor, such as the processor 104. In other embodiments, the sensor data can be correlated or combined for the target phase or a plurality of target phases. By combining or averaging the sensor data for the target phase, variations or noise in the sensor data can be reduced, thereby increasing the accuracy of the subsequent tracking of the target phase.

[0135] The method 500 also includes applying the average or combination to the subsequent tracking (step 520). By applying the average or combination of the sensor data to the subsequent tracking, the accuracy of the tracking can be improved because variations in the individual sensor data can be reduced. It should be understood that the steps 516 and 520 can be repeated and act as a feedback loop to increase the accuracy of the tracking of the catheter device, for example, throughout a surgical procedure.

[0136] The present disclosure encompasses embodiments of the method 500 that include more or fewer steps than those described above and / or one or more steps that are different than those described above.

[0137] Figure 6 A method 600 is depicted that can be used, for example, to determine a patient state or a patient state of an environment of a catheter.

[0138] The method 600 (and / or one or more steps thereof) can be carried out or otherwise performed by, for example, at least one processor. The at least one processor can be the same as or similar to the processor 104 of the computing device 102 described above. The at least one processor can be part of a robot, such as the robot 114, or part of a navigation system, such as the navigation system 118. The method 600 can also be performed using a processor other than any processor described herein. The at least one processor can perform the method 600 by executing elements stored in a memory, such as the memory 106. The elements stored in the memory and executed by the processor can cause the processor to perform one or more steps of the functions as shown in the method 600. One or more portions of the method 600 can be performed by a processor that performs any of the contents of the memory, such as the image processing 120, the tracking 122, the registration 124, and / or the reconstruction 128.

[0139] The method 600 includes tracking the catheter (step 604). The step 604 can be the same as or similar to the step 304 of the method 300 described above.

[0140] The method 600 also includes receiving sensor data corresponding to a plurality of pose information of the catheter (step 608). The step 608 can be the same as or similar to the step 308 of the method 300 described above.

[0141] The method 600 also includes determining a velocity of the catheter (step 612). The velocity of the catheter can be determined by a processor, such as the processor 104, using the plurality of pose information of the catheter over a period of time. It should be appreciated that the movement or velocity of the catheter can have different characteristics based on which portion of the anatomical element the catheter is positioned in. For example, in embodiments in which the anatomical element is a heart, the catheter can have more or less movement in the center of the aorta as compared to near the aortic wall. In another example, the motion of the catheter in a ventricle should be different than in an atrium. Such information can be used to confirm the position of the catheter within the anatomical element and, thus, the placement of an implantable device, such as a cardiac device, in a desired location. The information can also be used to determine whether the environmental state of the catheter is suitable for an implantable device and / or a patient state, as will be described below.

[0142] The method 600 also includes determining an environment state (step 616). The environment state can be determined using, for example, the velocity of the catheter determined in step 612. In some embodiments, the environment state can also be determined using, for example, pressure that can be measured by, for example, a catheter tracker. The environment state can be related to the environment state of the catheter positioned in the anatomical element. The environment state can be used to determine whether the location of the catheter positioning is suitable for, for example, a cardiac device or any other implantable device. For example, an abnormal velocity of the catheter can indicate motion that can damage or interfere with the implantable device. On the other hand, a velocity related to an expected velocity indicates that the placement is adequate for the implantable device. The velocity of the catheter can also be used to determine whether the motion of the catheter (and thus the implantable device) can exceed force and / or torque limits on the implantable device. Thus, if the environment state is not adequate, the implantable device can be implanted in another location, or a different implantable device can be selected.

[0143] The method 600 also includes determining a patient state (step 620). The patient state can be determined using, for example, the velocity of the catheter determined in step 612. The patient state can be related to the state of the patient and whether the patient can, for example, have any disease states. For example, a low velocity or motion of the catheter can indicate a low blood flow rate. In another example, a pathology can be determined based on movement of the catheter in a particular location.

[0144] In embodiments in which two or more catheter trackers are positioned on the catheter, the difference between the two catheter trackers (e.g., the relative spacing and length of the flow path between the two catheter trackers) can be used to determine a pulse wave velocity. The pulse wave velocity can be related to the stiffness of the vessel wall of the anatomical element and, more specifically, can be used to estimate the degree of arterial disease. For example, an increased pulse wave velocity can indicate a plaque in the vessel wall, as the vessel wall will be thicker and thus increase the pulse wave velocity.

[0145] The method 600 also includes generating a notification (step 624). The notification can be a visual notification, an audible notification, or any type of notification communicated to a user. The notification can be communicated to the user via a user interface, such as the user interface 110. In some embodiments, the notification can be automatically generated by a processor, such as the processor 104. In other embodiments, the notification can be automatically generated by any component of the system, such as the system 100.

[0146] The notification can be based on the speed (e.g., as determined in step 612) exceeding or being less than a predetermined threshold. The notification can alternatively or additionally be based on the environmental state and / or the patient state being an undesirable environmental state and / or patient state. For example, a notification can be generated when the environmental state is determined to be insufficient for the implantable device. In another example, a notification can be generated when the patient state indicates that the patient can have a disease. Thus, the notification can alert the surgeon or user of an expected speed, environmental state, and / or patient state that the surgeon or other user can wish to avoid or otherwise mitigate.

[0147] The present disclosure encompasses embodiments of the method 600 that include more or fewer steps than those described above and / or one or more steps that are different than those described above.

[0148] As described above, the present disclosure encompasses methods having fewer steps than all of the steps identified in Figure 3 , Figure 4 , Figure 5 and Figure 6 as well as methods (and corresponding descriptions of the methods 300, 400, 500, and 600) that include additional steps beyond those identified in Figure 3 , Figure 4 , Figure 5 and Figure 6 . The present disclosure also encompasses methods that include one or more steps from one method described herein and one or more steps from another method described herein. Any dependency described herein can be or include a registration or any other dependency.

[0149] The foregoing is not intended to limit the disclosure to one or more forms disclosed herein. In the foregoing DETAILED DESCRIPTION, for purposes of simplicity, the various features of the present disclosure are grouped together in one or more aspects, embodiments, and / or configurations. Features of the aspects, embodiments, and / or configurations of the present disclosure can be combined in alternative aspects, embodiments, and / or configurations other than those discussed above. The methods of the present disclosure are not to be construed as reflecting an intention that the claims require more features than are explicitly recited in each claim. Rather, the aspects of the invention are directed to each individual feature or combination of features. Accordingly, the claims are hereby incorporated into this DETAILED DESCRIPTION by reference, with each claim standing on its own as a separate preferred embodiment of the present disclosure.

[0150] Moreover, although the foregoing has been described in some detail for purposes of clarity and the known scope thereof, it should be various modifications, combinations and alterations understood that certain changes in form and detail can be made to the description and illustrations herein, without departing from the spirit, scope and underlying principles of the disclosure. It is therefore intended that the application be interpreted as including all such variations as fall within the scope of the invention. It is applicant's intention that successful patenting include "means plus function" claims unless the claim language does not permit such claims. The application further reserves the right to any patentable matter described in this document.

[0151] Embodiment 1. A system for determining one or more phases of an anatomical movement, the system comprising: a catheter positioned in an anatomical element; a tracker for tracking a pose of the catheter; a processor; and a memory storing data for processing by the processor, the data when processed causing the processor to: track the catheter; receive sensor data corresponding to a plurality of pose information of the catheter from the tracker; determine one or more phases of a motion of the anatomical element based on the plurality of pose information; receive an image at a target phase of the one or more phases; and display the image.

[0152] Embodiment 2. The system of embodiment 1, wherein tracking the catheter comprises tracking a velocity of the catheter.

[0153] Embodiment 3. The system of embodiment 1, wherein at least one of the tracker, the catheter, or a combination of the tracker and the catheter is shaped such that at least one of an enhancement, a reduction, or an increase in motion caused by a fluid flow in the anatomical element.

[0154] Embodiment 4. The system of embodiment 1, wherein the memory stores further data for processing by the processor, the further data when processed causing the processor to: receive a bioelectric signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0155] Embodiment 5. The system of embodiment 1, wherein the memory stores further data for processing by the processor, the further data when processed causing the processor to: receive a respiration cycle signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0156] Embodiment 6. The system of embodiment 1, wherein the anatomical element comprises a heart, and the one or more phases correspond to one or more phases of a heartbeat.

[0157] Embodiment 7. The system of embodiment 6, wherein the one or more phases of the heartbeat include at least a diastolic motion and a systolic motion of the heart.

[0158] Embodiment 8. The system of embodiment 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: average the sensor data for a target phase of the one or more phases; and apply the average to a subsequent tracking at the target phase.

[0159] Embodiment 9. The system of embodiment 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to determine a patient state based on the motion of the anatomical element.

[0160] Embodiment 10. The system of embodiment 1, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: determine a movement and a velocity of the catheter; and determine an environmental state of the catheter based on the movement and the velocity.

[0161] Embodiment 11. The system of embodiment 1, wherein the tracker comprises at least one of: an inertial measurement unit tracker, an electromagnetic tracker, or a combination of the inertial measurement unit tracker and the electromagnetic tracker.

[0162] Embodiment 12. The system of embodiment 1, wherein the tracker comprises an imaging device and the sensor data comprises image data, wherein the plurality of pose information is derived from the image data.

[0163] Embodiment 13. A system for determining one or more phases of anatomical movement, the system comprising: a catheter positioned in an anatomical element; a tracker for tracking a pose of the catheter; a processor; and a memory storing data for processing by the processor that, when processed, causes the processor to: track the catheter; receive sensor data corresponding to a plurality of poses of the catheter from the tracker; and determine one or more phases of motion of the anatomical element based on the plurality of poses.

[0164] Embodiment 14. The system of embodiment 13, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to receive an image at a phase of the one or more phases and display the image.

[0165] Example 15. The system of Example 13, wherein the tracker comprises at least one of an inertial measurement unit tracker or an electromagnetic tracker.

[0166] Example 16. The system of Example 13, wherein the memory stores further data for processing by the processor, which when processed causes the processor to determine a patient state based on the motion of the anatomical element.

[0167] Example 17. The system of Example 13, wherein the memory stores further data for processing by the processor, which when processed causes the processor to: determine a movement and a velocity of the catheter; and determine an environmental state of the catheter based on the movement and the velocity.

[0168] Example 18. A system for associating one or more images with one or more cycles of motion of an anatomical element, the system comprising: a processor; and a memory storing data for processing by the processor, which when processed causes the processor to: receive a plurality of images over a time period; receive sensor data corresponding to a plurality of pose information of a catheter, the plurality of pose information corresponding to pose information of the catheter over the time period to produce; determine one or more first motion phases of a first motion of the anatomical element based on the plurality of pose information; determine one or more second motion phases of a second motion of the anatomical element based on the plurality of pose information; and determine, from the plurality of images, a subset of images corresponding to a target first motion phase of the one or more first motion phases of the first motion and a target second motion phase of the one or more second motion phases of the second motion.

[0169] Example 19. The system of Example 18, wherein the subset of images comprises a first subset of images, and wherein the memory stores further data for processing by the processor, which when processed causes the processor to determine, from the plurality of images, a second subset of images corresponding to another target first motion phase of the one or more first motion phases of the first motion and another target second motion phase of the one or more second motion phases of the second motion.

[0170] Example 20. The system of Example 19, wherein the memory stores further data for processing by the processor, which when processed causes the processor to reconstruct a three-dimensional model based on an intersection of the first subset of images and the second subset of images.

[0171] Example 21. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to display at least one of: the first subset of images, the second subset of images, or a combination of the first subset of images and the second subset of images.

[0172] Example 22. The system of Example 18, wherein the first motion corresponds to a cardiac cycle and the second motion corresponds to a respiratory cycle.

[0173] Example 23. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to receive respiratory information of a patient for the period of time, wherein determining the one or more second motion phases of a second motion is also based on the respiratory information.

[0174] Example 24. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to obtain and display a subsequent plurality of images at the target first motion phase and the target second motion phase for a subsequent period of time.

[0175] Example 25. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: average sensor data at a target phase of the one or more phases; and apply the average to subsequent tracking at the target phase.

[0176] Example 26. The system of Example 18, wherein the sensor data comprises at least one of image data, accelerometer data, pose data, or motion data.

[0177] Example 27. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: receive a bioelectric signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

[0178] Example 28. The system of Example 18, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: receive a respiratory cycle signal, wherein determining the one or more phases of motion is further based on the bioelectric signal.

Claims

1. A system for determining one or more phases of anatomical movement, the system comprising: A catheter (136) is positioned within an anatomical element (212); Tracker (138) for tracking the pose of the catheter; Processor (104); and Memory (106), the memory storing data for processing by the processor, the data causing the processor to: Track the catheter; Receive sensor data from the tracker corresponding to multiple pose information of the catheter; Based on the multiple pose information, one or more stages of motion of the anatomical element are determined; Receive an image at a target stage in one or more of the stages; as well as The image is displayed.

2. The system of claim 1, wherein at least one of the tracker, the catheter, or a combination of the tracker and the catheter is configured such that at least one of the motion caused by the fluid flow in the anatomical element is enhanced, reduced, or increased.

3. The system of claim 1 or 2, wherein the memory stores additional data for processing by the processor, the additional data causing the processor to: Receive bioelectric signals, wherein the determination of one or more phases of motion is further based on the bioelectric signals.

4. The system according to any of the preceding claims, wherein the memory stores additional data for processing by the processor, the additional data causing the processor, when processed, to: Receive respiratory and circulatory signals, wherein the determination of one or more phases of movement is further based on the bioelectrical signals.

5. The system according to any of the preceding claims, wherein the anatomical element comprises a heart, and the one or more phases correspond to one or more phases of a heartbeat.

6. The system of claim 5, wherein the one or more phases of the heartbeat include at least diastolic and systolic movements of the heart.

7. The system according to any of the preceding claims, wherein the memory stores additional data for processing by the processor, the additional data causing the processor, when processed, to: The sensor data will be averaged for a target stage within one or more of the stages; and The average is applied to subsequent tracking at the target stage.

8. The system according to any of the preceding claims, wherein the memory stores additional data for processing by the processor, the additional data, when processed, enabling the processor to determine the patient's state based on the movement of the anatomical elements.

9. The system according to any preceding claim, wherein the memory stores additional data for processing by the processor, the additional data causing the processor, when processed, to: Determine the movement and speed of the catheter; and The environmental state of the catheter is determined based on the movement and the speed.

10. The system according to any of the preceding claims, wherein the tracker comprises at least one of: an inertial measurement unit tracker, an electromagnetic tracker, or a combination of the inertial measurement unit tracker and the electromagnetic tracker.

11. The system according to any of the preceding claims, wherein the tracker includes an imaging device, and the sensor data includes image data, wherein the plurality of pose information is derived from the image data.

12. A system for determining one or more phases of anatomical movement, the system comprising: A catheter (136) is positioned within an anatomical element (212); Tracker (138) for tracking the pose of the conduit (136); Processor (104); and Memory (106), the memory storing data for processing by the processor, the data causing the processor to: Track the catheter; Receive sensor data from the tracker corresponding to multiple pose information of the catheter; as well as Based on the multiple pose information, one or more phases of motion of the anatomical element are determined.

13. The system of claim 12, wherein the memory stores additional data for processing by the processor, the additional data, when processed, causing the processor to receive and display an image at one of the one or more stages.

14. The system of claim 12 or 13, wherein the memory stores additional data for processing by the processor, the additional data, when processed, enabling the processor to determine the patient's state based on the movement of the anatomical elements.

15. The system according to any preceding claim, wherein the memory stores additional data for processing by the processor, the additional data causing the processor, when processed, to: Determine the movement and speed of the catheter; and The environmental state of the catheter is determined based on the movement and the speed.

Images