System and method for performing intraoperative guidance

Abstract

Systems and methods are provided in which intraoperative video transmission is enhanced using one or more virtual display features based on the detected orientation and position of a tracked medical device. In some exemplary embodiments, the virtual display features can be used to present information related to the previously detected intraoperative position and orientation of a tracked medical device in an enhanced video transmission. The virtual display elements can be used, for example, to facilitate alignment of an untracked medical device using the previously determined intraoperative position and orientation of a tracked medical device. In other exemplary embodiments, the virtual display elements can be dynamically displayed in a fixed spatial relationship relative to the tracked medical device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 655,102, entitled “SYSTEMS AND METHODS FOR PERFORMING INTRAOPERATIVE GUIDANCE,” filed on April 9, 2018, the entire contents of which are incorporated herein by reference, and to U.S. Provisional Patent Application No. 62 / 736,348, entitled “SYSTEMS AND METHODS FOR PERFROMING INTRAOPERATIVE GUIDANCE,” filed on September 25, 2018, the entire contents of which are incorporated herein by reference. Background Art

[0003] The present disclosure relates to guided medical procedures.

[0004] Guidance enables the surgeon to visualize the position and orientation of the medical instrument relative to the patient's anatomy. To visualize the position and orientation of the medical instrument, fiducial markers are attached to the medical instrument so that signals from the fiducial markers can be detected by a camera or other sensing device (e.g., an electromagnetic tracking device), and the position and orientation of the medical instrument can be determined by triangulation. SUMMARY OF THE INVENTION

[0006] Provided herein are systems and methods in which one or more virtual display features are utilized to enhance intraoperative video transmission based on a detected orientation and position of a tracked medical device. In some exemplary embodiments, the virtual display feature may be used to present information related to a previously detected intraoperative position and orientation of a tracked medical device with enhanced video transmission. Virtual display elements may be used, for example, to facilitate alignment of an untracked medical device utilizing a previously determined intraoperative position and orientation of a tracked medical device. In other exemplary embodiments, virtual display elements may be dynamically displayed in a fixed spatial relationship relative to a tracked medical device.

[0007] Thus, in a first aspect, provided herein is a method of providing intraoperative guidance for aligning a non-trackable medical device during a medical procedure, the method comprising:

[0008] Using a tracking system to detect signals from fiducial markers associated with a traceable medical device;

[0009] Processing the signals to determine the intraoperative orientation and position of the trackable medical device in a reference frame relative to the tracking system;

[0010] Using a video camera to obtain intraoperative video feed;

[0011] Display intraoperative video feeds; and

[0012] In subsequent stages of a medical procedure involving a non-trackable medical device, coordinate transformations between a reference frame associated with a camera and a reference frame associated with a tracking system are employed to enhance intraoperative video transmission using virtual display features that identify a previously determined intraoperative orientation and position of a trackable medical device, thereby enabling the non-trackable medical device to be visually aligned relative to a previously determined intraoperative orientation and position of the trackable medical device in subsequent stages of the medical procedure.

[0013] In another aspect, provided herein is a method of providing intraoperative guidance for aligning a non-trackable medical device during a medical procedure, the method comprising:

[0014] Using a surface detection system to detect signals from one or more surface reference features associated with the traceable medical device;

[0015] Processing signals to determine intraoperative orientation and location of trackable medical devices;

[0016] A camera using a surface detection system is used to obtain intraoperative video feeds;

[0017] Display intraoperative video feeds; and

[0018] In a subsequent phase of a medical procedure involving a non-trackable medical device, intraoperative video transmission is enhanced using virtual display features associated with a previously determined intraoperative orientation and position of a traceable medical device, thereby enabling the non-trackable medical device to be visually aligned relative to the previously determined intraoperative orientation and position of the traceable medical device during the subsequent phase of the medical procedure.

[0019] In another aspect, provided herein is a method of providing intraoperative guidance for aligning a non-trackable medical device during a medical procedure, the method comprising:

[0020] Using a tracking system to detect signals from fiducial markers associated with a traceable medical device;

[0021] Processing signals to determine intraoperative orientation and location of trackable medical devices;

[0022] A camera with a tracking system was used to obtain intraoperative video feeds;

[0023] Display intraoperative video feeds; and

[0024] In a subsequent phase of a medical procedure involving a non-trackable medical device, intraoperative video transmission is enhanced using virtual display features associated with a previously determined intraoperative orientation and position of a traceable medical device, thereby enabling the non-trackable medical device to be visually aligned relative to the previously determined intraoperative orientation and position of the traceable medical device during the subsequent phase of the medical procedure.

[0025] In another aspect, provided herein is a method of aligning a non-traceable medical device during a medical procedure, the method comprising:

[0026] Intraoperatively orient and position trackable medical devices in relation to the patient’s intraoperative orientation and position;

[0027] Use guidance and tracking systems to:

[0028] detecting signals from fiducial markers associated with traceable medical devices;

[0029] Processing the signals to determine the intraoperative orientation and position of the trackable medical device in a reference frame relative to the tracking system;

[0030] A video camera was used to obtain intraoperative video transmission;

[0031] Display intraoperative video transmission

[0032] Enhance the intraoperative video feed with virtual display features associated with the determined intraoperative position and orientation of the trackable medical instrument using coordinate transformations between a reference frame associated with the camera and a reference frame associated with the tracking system; and

[0033] During subsequent stages of a medical procedure involving an untracked medical device, the intraoperative video feed is viewed and virtual display features are employed to locate and orient the untracked medical device.

[0034] In another aspect, provided herein is a method of providing intraoperative guidance during a surgical procedure, the method comprising:

[0035] Using a tracking system to detect signals from fiducial markers associated with a traceable medical device;

[0036] Processing the signals to determine the intraoperative orientation and position of the trackable medical device in a reference frame relative to the tracking system;

[0037] A video camera was used to obtain intraoperative video transmission;

[0038] Display intraoperative video feeds; and

[0039] Coordinate transformations between a reference frame associated with a camera and a reference frame associated with a tracking system are employed to enhance intraoperative video delivery in real time using virtual display features associated with trackable medical devices.

[0040] In another aspect, provided herein is a system for providing intraoperative guidance for aligning a non-trackable medical device during a medical procedure, the system comprising:

[0041] Tracking Subsystem

[0042] Display; and

[0043] Computer hardware operably coupled to the tracking subsystem, wherein the computer hardware includes a memory coupled to the one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

[0044] detecting, using the tracking subsystem, a signal from a fiducial marker associated with a trackable medical device;

[0045] Processing the signals to determine the intraoperative orientation and position of the trackable medical device in a reference frame relative to the tracking system;

[0046] A video camera was used to obtain intraoperative video transmission;

[0047] Displaying the intraoperative video feed on a monitor; and

[0048] In subsequent stages of a medical procedure involving a non-trackable medical device, coordinate transformations between a reference frame associated with a camera and a reference frame associated with a tracking system are employed to enhance intraoperative video transmission using virtual display features that identify a previously determined intraoperative orientation and position of a trackable medical device, thereby enabling the non-trackable medical device to be visually aligned relative to a previously determined intraoperative orientation and position of the trackable medical device in subsequent stages of the medical procedure.

[0049] In another aspect, provided herein is a system for providing intraoperative guidance during a surgical procedure, the system comprising:

[0050] Tracking subsystem;

[0051] Display; and

[0052] Computer hardware operably coupled to the tracking subsystem, wherein the computer hardware includes a memory coupled to one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

[0053] employing the tracking subsystem to detect signals from fiducial markers associated with a trackable medical device;

[0054] Processing the signals to determine the intraoperative orientation and position of the trackable medical device in a reference frame relative to the tracking system;

[0055] A video camera was used to obtain intraoperative video transmission;

[0056] Displaying the intraoperative video feed on a monitor; and

[0057] Coordinate transformations between a reference frame associated with a camera and a reference frame associated with a tracking system are employed to enhance intraoperative video delivery in real time using virtual display features associated with trackable medical devices.

[0058] A further understanding of the functionality and advantages of the present disclosure may be realized by referring to the following detailed description and accompanying drawings.

[0059] BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:

[0061] Figure 1 An exemplary system for providing intraoperative guidance of trackable and non-trackable medical devices is shown.

[0062] Figure 2 The flowchart of FIG. 1 illustrates an exemplary method for providing intraoperative guidance in which an intraoperative video feed is augmented with virtual display features associated with a detected intraoperative orientation and position of a trackable medical instrument.

[0063] Figure 3A An example of a guidance user interface showing different intraoperative views of a tracked medical instrument relative to a volumetric image is shown, wherein a pedicle probe is shown inserted into a pedicle.

[0064] Figure 3B Shows Figure 3A The guidance user interface displays linear virtual features related to the detected position and orientation of the pedicle probe.

[0065] Figure 4A An exemplary frame of an intraoperative video feed acquired using two stereo cameras of a structured light surface detection system is shown. The tracked pedicle probe can be seen in the video frame.

[0066] Figure 4B An exemplary frame of an intraoperative video feed obtained after the pedicle probe is removed from the surgical field is shown. The intraoperative video feed is enhanced with a linear virtual display feature indicating the intraoperative position and orientation of a previously determined tracked medical instrument.

[0067] Figure 4C An exemplary frame of an intraoperative video feed obtained after removal of the pedicle probe from the surgical field is shown, wherein a non-trackable medical instrument is displayed aligned with a linear virtual display feature such that the non-trackable medical instrument is aligned with a previously determined intraoperative position and orientation of a tracked medical instrument.

[0068] Figure 5A-Figure 5C An example situation is shown where degeneracy exits in aligning linear annotations and untracked medical devices.

[0069] Figure 6 The flowchart illustrates an exemplary process for generating and enhancing a video feed from a pair of stereo cameras.

[0070] Figure 7 The flowchart illustrates an exemplary process for generating and enhancing a video feed from a single camera.

[0071] Figure 8 The flowchart of FIG. 1 illustrates an exemplary method for providing intraoperative guidance in which an intraoperative video feed is augmented in real-time with virtual display features associated with a current intraoperative orientation and position of a trackable medical instrument.

[0072] Figure 9A-9D An exemplary embodiment is shown in which a displayed video feed can be optionally rotated so that the action in the real world is displayed in the correct orientation in the video feed. DETAILED DESCRIPTION OF THE INVENTION

[0074] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below. The following description and accompanying drawings are illustrations of the present disclosure and should not be construed as limiting the present disclosure. Many specific details are described to provide a thorough understanding of the various embodiments of the present disclosure. However, in some cases, well-known or conventional details are not described in order to provide a brief discussion of the embodiments of the present disclosure.

[0075] As used herein, the terms "comprises" and "comprising" should be interpreted as inclusive and open-ended rather than exclusive. Specifically, when used in the specification and claims, the terms "comprises" and "comprising" and variations thereof mean including the specified features, steps, or components. These terms should not be interpreted to exclude the presence of other features, steps, or components.

[0076] As used herein, the term "exemplary" means "serving as an example, instance, or illustration," and should not be construed as preferred or advantageous over other configurations disclosed herein.

[0077] As used herein, the terms "about" and "approximately" are intended to encompass variations that may exist in the upper and lower limits of a range of values, such as variations in properties, parameters, and dimensions. Unless otherwise indicated, the terms "about" and "approximately" mean plus or minus 25% or less.

[0078] It should be understood that, unless otherwise specified, any specified range or group is intended as a shorthand way of referring to each and every member of the range or group, and to every possible subrange or subgroup contained therein, and similarly to any subrange or subgroup therein, respectively. Unless otherwise indicated, the present disclosure is directed to and specifically incorporates every specific member and combination of subranges or subgroups.

[0079] As used herein, the term "on the order of" when used in connection with a quantity or parameter, refers to a range spanning from about one-tenth to ten times the quantity or parameter.

[0080] Most surgical procedures typically utilize a large number of different medical instruments to implement the surgical plan. Although some medical instruments can be tracked by a tracking system via detection of signals from fiducial markers (fiducials), it is often the case that only a portion of the medical instruments employed are tracked by a guidance system during the procedure. For example, during a given medical procedure, medical instruments (e.g., surgical instruments) from a variety of different manufacturers are often employed, and only some of the medical instruments can be tracked using a given tracking system. Therefore, there exists a technical problem in the art that only a portion of the medical instruments employed can be tracked during the medical procedure and are suitable for presentation on a guidance display (e.g., a guidance user interface window).

[0081] Various aspects of the present disclosure provide a solution to this problem by enhancing intraoperative video transmission using one or more virtual display features based on the detected direction and position of the tracked medical device. In some exemplary embodiments, the virtual display feature can be used to present information related to the intraoperative position and orientation of the previously detected tracked medical device in an enhanced video transmission. The virtual display element can be used, for example, to facilitate the alignment of an untracked medical device using the previously determined intraoperative position and orientation of the tracked medical device. In another exemplary application, a user or clinician can first position and align the untracked medical device at a given position and orientation believed to be the correct intraoperative position and orientation, and then observe the position and orientation of the untracked medical device relative to the virtual display element in the enhanced video transmission to determine whether the current position and orientation is indeed aligned relative to the previously detected intraoperative position and orientation of the tracked medical device. In other exemplary embodiments, the virtual display element can be dynamically displayed in a fixed spatial relationship relative to the tracked medical device.

[0082] Now for Figure 1 , an exemplary system for determining the intraoperative direction of a tracked medical device and enhancing the intraoperative video transmission based on the detected direction and position of the tracked medical device is shown. Although the exemplary system includes a surface detection system to facilitate the registration of the patient's volumetric image data in the intraoperative reference system, other systems and methods may be used alternatively. For example, in an optional embodiment, an intraoperative CT scan can be performed using a reference array of fiducial marker locations attached to the patient to facilitate direct intraoperative registration of the patient's volumetric image data. The exemplary system includes a surface detection system 10 that is operably coupled to the control and processing hardware 100 interface. The surface detection system 10 can be any suitable system for detecting, measuring, imaging, or otherwise determining the surface topography of one or more targets (such as, but not limited to, an exposed patient spinal region 50). According to this non-limiting exemplary embodiment, an intraoperative video transmission (or multiple intraoperative video transmissions) is obtained from one or more cameras of the surface detection system 10. Non-limiting examples of suitable surface detection systems include photogrammetry systems, stereo imaging systems, and structured light detection systems.

[0083] Figure 1 The exemplary system shown in also includes a tracking system 20 that can be used to track the position and orientation of one or more trackable medical devices 40. The trackable medical device 40 is shown to have a fiducial marker 45 attached thereto, and passive or active signals emitted or reflected (e.g., scattered) from the fiducial marker 45 are detected by the tracking system 20 (e.g., a stereoscopic tracking system using two tracking cameras).

[0084] In an exemplary embodiment, the optical tracking subsystem 20 may include a stereo camera with integrated infrared illumination. Due to the high reflectivity of the fiducial markers 45 to infrared light, they can be easily located in each image of the two cameras. These image positions can be used to calculate the three-dimensional position of each fiducial marker 45 by geometric triangulation. For example, the triangulation process can be performed by first calculating the center of mass of each detection marker in the two camera views of the stereo calibrated camera system. This produces a set of marker points in the two camera views, from which the difference between corresponding points in the two views can then be calculated. This difference can then be converted to three-dimensional space coordinates (in the coordinate system of the tracking system 20) using a perspective transformation together with the x and y pixel positions of each marker in one camera view. If at least three fiducial markers 45 are rigidly attached to the medical device 40, it is possible to calculate its position and orientation (six degrees of freedom).

[0085] In some exemplary illustrations provided herein, the fiducial markers 45 for the optical tracking system are shown as reflective spheres, which are typically used for passive optical tracking. However, depending on the tracking system used, any other type of marker or marker attribute may be used, such as but not limited to active markers (i.e., LEDs that do not require integration of additional lighting and electromagnetic markers) and passive markers (e.g., glyphs, different marker colors, different marker sizes, different marker shapes). It should be understood that in some embodiments, less than three markers may be used for position and location tracking. For example, a single marker may be provided for position and location tracking, as long as the single marker includes sufficient spatial structure and / or content. An example of such a single marker is a glyph, including coplanar spatial features such as corner or edge features.

[0086] although Figure 1 An exemplary system with two subsystems (tracking and surface detection) is shown, but it is noted that alternative system configurations may be used to perform simultaneous tracking and acquisition of anatomical surfaces utilizing an integrated system, such as that described in international patent application PCT / CA2011 / 050257, which is hereby incorporated by reference in its entirety.

[0087] Figure 1Also shown is an exemplary implementation of control and processing hardware 100, which includes one or more processors 110 (e.g., CPU / microprocessors), a bus 105, memory 115, which may include random access memory (RAM) and / or read-only memory (ROM), a data acquisition interface 120, a display 125, external memory 130, one or more communication interfaces 135, a power supply 140, and one or more input / output devices and / or interfaces 145 (e.g., speakers, user input devices such as a keyboard, keys, a mouse, a position-tracking pin, a position-tracking probe, a foot switch, and / or a microphone for capturing voice commands).

[0088] It should be understood that Figure 1 The exemplary system shown is an illustration of a non-limiting exemplary embodiment and is not intended to be limited to the components shown. In addition, one or more components of the control and processing hardware 100 may be provided as external components that interface with a processing device. For example, as shown, one or both of the surface detection system 10 and the tracking system 20 may be included as components of the control and processing hardware 100, or may be provided as one or more external devices.

[0089] Despite Figure 1 Only one of each component is shown, but any number of various components may be included in the control and processing hardware 100. For example, a computer typically contains multiple different data storage media. In addition, although the bus 105 is depicted as a single connection between all components, it should be understood that the bus 105 can represent one or more circuits, devices, or communication channels connecting two or more components. For example, in a personal computer, the bus 105 typically includes a mainboard or motherboard. The control and processing hardware 100 may include more or fewer components than those shown.

[0090] The control and processing hardware 100 may be implemented as one or more physical devices coupled to the processor 110 through one of a plurality of communication channels or interfaces. For example, the control and processing hardware 100 may be implemented using application specific integrated circuits (ASICs). Alternatively, the control and processing hardware 100 may be implemented as a combination of hardware and software, where the software is loaded into the processor from a memory or over a network connection.

[0091] Some aspects of the present disclosure may be embodied at least in part in software. In other words, the technology may be performed in a computer system or other data processing system, which responds to its processor, such as a microprocessor, to execute a sequence of instructions contained in a memory, such as a ROM, a volatile RAM, a non-volatile memory, a cache, a disk, and an optical disk, or a remote storage device. In addition, instructions may be downloaded to a computing device over a data network in the form of a compiled and linked version. Alternatively, the logic for performing the processing described above may be implemented in another computer and / or machine-readable medium, such as a discrete hardware component, such as a large-scale integrated circuit (LSI's), an application-specific integrated circuit (ASIC's), or firmware, such as an electrically erasable programmable read-only memory (EEPROM's) and a field programmable gate array (FPGAs).

[0092] Computer readable media can be used to store software and data that, when executed by a data processing system, cause the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. A portion of the software and / or data can be stored in any of these storage devices. Typically, machine-readable media include any mechanism that provides (i.e., stores and / or transmits) information in a form accessible to a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors).

[0093] Examples of computer-readable media include, but are not limited to, recordable and non-recordable types of media, such as volatile and non-volatile storage devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), etc. The instructions may be embodied in digital and analog communication links for electrical, optical, acoustic or other forms of propagated signals such as carrier waves, infrared signals, digital signals, etc. As used herein, the phrases "computer-readable material" and "computer-readable storage medium" refer to all computer-readable media, except for transient propagated signals themselves.

[0094] The embodiments of the present disclosure may be implemented via the processor 110 and / or the memory 115. For example, the functions described below may be implemented in part via hardware logic in the processor 110 and may be implemented in part using instructions stored in the memory 115. Some embodiments are implemented using the processor 110 without storing additional instructions in the memory 115. Some embodiments are implemented using instructions stored in the memory 115 to be executed by one or more microprocessors, which may be general-purpose processors or special-purpose processors. Therefore, the present disclosure is not limited to a specific configuration of hardware and / or software.

[0095] The control and processing hardware 100 is programmed with subroutines, applications or modules 150 including executable instructions that, when executed by one or more processors 110, cause the system to perform one or more methods described in the present disclosure. Such instructions may be stored, for example, in the memory 115 and / or other internal storage.

[0096] For example, as described in detail below, the augmented reality module 155 of the control and processing hardware 100 can be used to enhance one or more intraoperative video transmissions obtained from video data obtained from one or more cameras of the surface detection system 10 or one or more additional cameras having one or more virtual display elements based on the intraoperative orientation and position of a trackable medical probe determined via the tracking system 20.

[0097] The guidance user interface module 165 includes executable instructions for displaying a user interface to perform, for example, image-guided surgical procedures and to display one or more intraoperative video feeds.

[0098] Furthermore, in the exemplary embodiment shown, the registration module 160 includes executable instructions to register the segment surface data (data obtained from the volume image data 30) with the intraoperative surface data obtained by the surface detection system 10 as described above. For example, the volume image data 30 can be provided to the control and processing hardware 100 for registration to the intraoperatively acquired surface data.

[0099] The registration module 160 performs image registration between the segment surfaces generated from the volume image data 30 and the intraoperative surface data. Non-limiting examples of surface segmentation methods include non-template-based methods and methods using anatomical shape models. Non-template-based methods can use geometric properties such as connectivity, surface normals and curvature to determine the boundaries of the segmented region, or statistical properties such as differences with nearby adjacent points on the surface. Methods based on anatomical shape models can use pre-calculated atlases (e.g., vertebrae) as templates to perform segmentation. These two types of methods can also be used in combination. In all of these methods, one or more volume reference points can be used as seed points to initialize the segmentation process. Optionally, for fully automatic segmentation methods that operate on the entire volume data (which are typically based on anatomical atlases), one or more volume references can be used to mark the target level.

[0100] Surface registration can be performed as an initial registration based on the consistency between volumetric reference points defined in the volumetric image data 30 and respective intraoperative reference points identified on the segmented surface. After the initial registration is generated, a surface-to-surface registration can then be performed between the segmented surface data and the intraoperative surface data to obtain a registration transformation. The registration transformation maps the segmented surface in the volumetric reference system to the intraoperative surface data. It will be understood that when performing methods according to the exemplary embodiments disclosed herein, any suitable surface registration method can be used to perform registration between surfaces. Non-limiting examples of suitable registration methods include iterative closest point algorithms in which the distance between points from different surfaces is minimized.

[0101] In the exemplary case where the surface detection system 10 is a structured light detection system, a projection device is used to project surface topography detection light onto a target area, and one or more cameras detect the surface topography light scattered or reflected from the target area. The detected optical signal can be used to generate a surface topography data set consisting of a point cloud or a mesh. More specifically, the projection device projects time- and / or spatially modulated light onto the surface to be imaged, and the camera captures an image of the illuminated surface. This active illumination enables robust and effective identification of pixel correspondences between a calibrated camera-projector (the projector can be regarded as an inverse camera) or a calibrated camera-camera system. The correspondence (difference) data can then be converted into real-space coordinate data in the coordinate system of the calibrated camera and / or projector by geometric triangulation. During a surgical procedure, the structured light detection system can be positioned to obtain a three-dimensional surface of the surgical site (e.g., an exposed bone surface of the spine). Then, the virtual representation of the created three-dimensional surface is registered to the volume image data 30 (e.g., CT, MRI, US, PET, etc.) by the registration module 160, using, for example, the method described in International Patent Application No. PCT / CA2011 / 050257. The volume image data 30 can be acquired before the operation, but not necessarily before the operation. For example, in some applications, the volume image data 30 can also be acquired during the operation.

[0102] In order to present the trackable medical device in the guidance image, a calibration transformation is determined between the reference frame of the surface detection system 10 and the reference frame of the tracking system 20. If the relative positions of the tracking system and the surface imaging system are fixed, this calibration can be performed by obtaining the positions of at least 3 points of a calibration object from both systems and aligning these points to obtain the calibration transformation, as described in International Patent Application No. PCT / CA2011 / 050257.

[0103] In an optional embodiment, as disclosed in International Patent Application No. PCT / CA2011 / 050257, the surface detection device can have a fiducial marker attached thereto, and the fiducial marker can be tracked by the tracking system. In this configuration, a calibration procedure can be used to obtain a calibrated transformation from the reference system of the surface detection system to the reference system of the tracking system using the attached fiducial marker. The calibrated transformation between the coordinate systems of the tracking system and the surface imaging system is then continuously updated as the position of the surface imaging device changes.

[0104] After calibration is performed, the calibration transformation between the coordinate systems of the tracking system and the surface imaging system is known. The registered surface data set and the volume image data are therefore equivalent to identifying the location of the volume image data in the coordinate system of the tracking system. Thus, any trackable medical device 40 that is subsequently tracked using the tracking subsystem can be presented to the surgeon as an overlay of the medical device 40 on the registered 3D image data on a display or other visualization device.

[0105] To compensate for patient or system motion, it may also be advantageous to use a tracking device that is attached to the patient's anatomy (e.g., to a bony feature of the patient's anatomy). Figure 1 As shown, the position of the tracking marker support structure 55 is recorded by the tracking system simultaneously with the acquisition of the surface data set. The surface data set is transformed into the coordinate system of the tracking system (using the previously acquired calibration transformation) and then registered to the volume image data. Subsequent tracking of the medical device relative to the volume image data can be performed based on the tracked tracking marker support structure and compensated for patient or system motion without the need to continuously acquire surface data.

[0106] In one exemplary embodiment, as described in International PCT Patent Application No. PCT / CA2015 / 050939, entitled “TRACKING MARKER SUPPORT STRUCTURE AND SURFACE REGISTRATION METHODS EMPLOYING THE SAME FOR PERFORMING NAVIGATED SURGICAL PROCEDURES” and filed on September 23, 2015, which is hereby incorporated by reference in its entirety, the tracking marker support structure 55 is used to calculate a real-time calibration transformation between a tracking system and a surface imaging system, for example, to evaluate the validity of a previously determined calibration transformation. As described below, this can be achieved by performing surface probing to determine the position and orientation of the tracking marker support structure in the reference frame of the surface imaging system, and comparing that position to the position of the tracking marker support structure determined by the tracking system based on signal detection from the marker, wherein the comparison employs the last calibration transformation (the previously determined calibration transformation). Thus, the validity of the last calibration transformation can be evaluated by determining whether the calculated position and orientation are within a specified tolerance range.

[0107] The method can be performed at any time before or during a surgical procedure, such as each time a registration is performed, and optionally each time a tracking marker support structure is attached to a new bone feature of the patient. For example, in the case of a spinal surgical procedure, the method can be performed or repeated when a tracking marker support structure (or additional tracking marker support structures) is attached to a new spinal level. The method will be referred to herein as "active calibration".

[0108] Again for Figure 1 , a typical medical procedure may involve the use of both a trackable medical device, such as a trackable medical device 40, and a non-trackable medical device, such as a non-trackable medical device 60. Although the tracking system 20 is capable of tracking the trackable medical device 40 so that the trackable medical device 40 can be displayed on the guidance image relative to the volumetric image data associated with the patient, the system effectively ignores the presence of the non-trackable medical device 60.

[0109] Although the system cannot track the untracked instrument 60, the untracked instrument may be visible to the clinician or operator in the intraoperative video feed. Recognizing that the intraoperative position and orientation of the untracked medical instrument is often related to the intraoperative position and orientation of the tracked medical instrument during a previous (early) stage or step of the medical procedure, the inventors realized that if the intraoperative video feed is generated using a camera having a reference frame that is related to the reference frame of the tracking system (via a known coordinate transformation), it is possible to enhance the intraoperative video feed based on the detected orientation and position of the tracked medical instrument using one or more virtual display features, such that the one or more virtual display features displayed in the intraoperative video feed can be used to facilitate subsequent alignment of the untracked medical instrument.

[0110] Thus, in some exemplary embodiments, one or more virtual display features can be used to present information related to a previously detected intraoperative position and orientation of a tracked medical instrument in an enhanced video feed, where the coordinate transformation between the reference frames of the camera and the tracking system used to generate the intraoperative video feed is known. As described below, virtual display elements can be employed, for example, to facilitate alignment of an untracked medical instrument.

[0111] Figure 2A flow chart example illustrating such an exemplary embodiment is provided. In step 200, a tracking system is used to detect a signal from a fiducial marker associated with a traceable medical device. This step can be performed when the medical device is in a specific intraoperative direction and position that defines a reference intraoperative direction and position associated with subsequent steps during a medical procedure. As described in further detail below, and an example of such a reference intraoperative direction and position is the direction and position of a pedicle probe during a cannula insertion operation, which defines an intraoperative axis associated during subsequent stages of a medical procedure, such as tapping a cylindrical track and inserting a pedicle screw. In step 205, the signal is processed to determine the intraoperative direction and position of the traceable medical device in a reference frame associated with the tracking system.

[0112] For example, as described above, images of detected fiducial marker signals can be used to calculate the three-dimensional position of each fiducial marker 45 by geometric triangulation, and because the fiducial markers are arranged in a fixed spatial relationship relative to the traceable medical device, the position and orientation of the traceable medical device can be determined (e.g., using a known geometric model of at least a portion of the traceable medical device and the relative positions of the fiducial markers).

[0113] In step 210, the camera is used to generate an intraoperative video feed, which is then displayed intraoperatively (eg, on a display window of a guidance user interface). The camera is configured so that the coordinate transformation between the camera's reference frame and the tracking system's reference frame is known.

[0114] In step 215, the intraoperative orientation and position of the trackable medical instrument as determined in step 205 are transformed into the reference frame of the camera using the known coordinate transformation between the reference frame of the camera and the reference frame of the tracking system. After the intraoperative orientation and position of the trackable medical instrument are transformed into the reference frame of the camera, the intraoperative video transmission can be enhanced with display information related to the intraoperative orientation and position of the trackable medical instrument, such as virtual display features. For example, the intraoperative video transmission can be enhanced with virtual display features that identify the previously determined intraoperative orientation and position of the trackable medical instrument, as shown at step 220.

[0115] Utilizing the intraoperative orientation and position of a trackable medical instrument at an earlier stage in a medical procedure, the previous methods can be used to facilitate alignment of an untracked medical instrument at a later stage (stage, time, step, or event) in a medical procedure. By augmenting the intraoperative video feed with a virtual display feature that indicates the orientation and position of the previously determined trackable medical instrument, the surgeon or other medical practitioner can view the intraoperative video feed and align the untrackable medical instrument relative to the virtual display feature.

[0116] For example, as described in the examples provided below, the present exemplary embodiment can be used to help locate one or more non-trackable medical instruments during a medical procedure involving implantation of pedicle screws. For example, the intraoperative direction and position of a tracked pedicle probe can be detected and stored using a tracking system. In a subsequent stage of the medical procedure, an intraoperative video transmission can be enhanced using a virtual display feature indicating the intraoperative direction and position of a previously determined pedicle probe according to the exemplary method described above. Subsequently, while observing the enhanced intraoperative video transmission, one or more non-tracked medical instruments such as a tap or a screwdriver can be visually aligned with the virtual display feature.

[0117] Now for Figure 3A-3B and Figure 4A-4C , and for clinical application of pedicle screw placement, it is shown Figure 2 An exemplary embodiment of the method shown in During medical procedures involving pedicle screw placement, trackable medical instruments such as a trackable awl and a trackable pedicle probe are often used to develop an initial trajectory for placement of the pedicle screw using a surgical guide system, wherein the trackable awl is used to create an entry point and the trackable pedicle probe is used for cannulation along an appropriate trajectory. Figure 3A An exemplary surgical guidance user interface is shown in which a pedicle probe 300 is illustrated relative to a pre-operative volumetric image 305 of a patient.

[0118] Figure 3A and Figure 3BThe exemplary guidance window shown in FIG. 1 includes spatial annotations that can be used to perform measurements to determine an appropriate screw diameter and length, as disclosed in U.S. Patent Publication No. 20170202626, entitled “SYSTEMS AND METHODS FOR DISPLAYING GUIDANCE IMAGES WITH SPATIAL ANNOTATIONS DURING A GUIDED MEDICAL PROCEDURE” and filed on January 12, 2017, which is hereby incorporated by reference in its entirety. In brief, the guidance image displays a two-dimensional cross-section through the spine, wherein the two-dimensional image includes and illustrates the longitudinal axis 310 of the pedicle probe 300. A plurality of spatial annotations 315 are generated along the longitudinal axis 310 of the pedicle probe 300, wherein each spatial annotation is centered on the longitudinal axis 310. In this exemplary embodiment, the spatial annotations 315 are generated so that the image region (linear segmentation) associated with each spatial annotation extends relative to the longitudinal axis 310 in a direction perpendicular to the longitudinal axis. Each spatial annotation depicts (indicates) an image region that has a known length metric associated with it (in this case, a linear image segmentation). Thus, each spatial annotation provides a hierarchy that allows the operator / user to obtain a visual metric of the relative size of anatomical and / or functional features in the guidance image.

[0119] like Figure 3A As shown in , tracking of the pedicle probe during the cannula insertion operation, and the known transformation between the tracking system and the intraoperative patient reference frame (e.g., using surface probing as described above, or facilitated via fiducial picking using a tracking tool), enables determination of the intraoperative orientation and position of the pedicle probe 300. Figure 3A A schematic diagram of a pedicle probe 300 is shown indicating the intraoperative orientation and position of the pedicle probe 300. The intraoperative orientation and position of the pedicle probe 300 may be specified, stored, and displayed in any suitable manner, such as, for example, via characterizing a longitudinal axis 310 of the intraoperative orientation of the pedicle probe and a distal position 320 of the distal tip of the pedicle probe 300. Figure 3B A guide image is shown showing the intraoperative longitudinal axis 310 of the pedicle probe 300 extending in proximal and distal directions relative to the distal tip of the pedicle probe.

[0120] After determining the intraoperative orientation and position of the pedicle probe 300 during cannulation and identifying the intraoperative axis associated with the cannulation procedure, one or more intraoperative video feeds can be enhanced to display one or more virtual display features associated with the intraoperative orientation and position of the pedicle probe. As explained above, a coordinate transformation is used to transform the intraoperative orientation and position as defined in the reference system of the tracking system into the reference system of the camera. The intraoperative video feed can then be enhanced with one or more virtual display features to assist the clinician by presenting information identifying the intraoperative orientation and position of the pedicle probe in the intraoperative video feed.

[0121] Reference Figure 4A , showing an exemplary embodiment in which a pair of intraoperative video feeds are generated using two cameras of a surface detection system. Each of the two intraoperative video feeds displays the intraoperative orientation and position of a pedicle probe 300 during cannulation, and the intraoperative video feeds have been enhanced using a virtual display feature that identifies an intraoperative axis associated with the intraoperative orientation of the pedicle probe. As explained in further detail below, because known coordinate transformations exist between the reference system of the surface detection system and the reference system of the tracking system, the intraoperative orientation and position of the tracked pedicle probe as determined in the reference system of the tracking system can be transformed and presented in the reference system of the surface detection system. This enables enhancement of the intraoperative video feeds obtained from the cameras of the surface detection system. An exemplary method for processing the intraoperative video feeds to perform enhancements is described in detail below.

[0122] exist Figure 4A In the exemplary embodiment shown in , the virtual display feature is displayed as a linear annotation 330 indicating the intraoperative direction of the longitudinal axis of the pedicle probe 300. It can be seen that the linear annotation 330 is coaxially aligned with the image of the pedicle probe 330. Figure 4B An exemplary embodiment is shown in which the intraoperative video feed is enhanced with linear annotations 330 (virtual display elements) after the pedicle probe is removed from the surgical field. Figure 4C As shown in , when an untrackable screwdriver 340 is introduced into the surgical field during a subsequent stage of the medical procedure, a clinician observing the displayed enhanced video feed can visually align the axis of the screwdriver 340 relative to the linear annotation 330, thereby effectively employing the previously tracked medical instrument to support subsequent intraoperative guidance of the untrackable or untracked medical instrument.

[0123] In some exemplary embodiments, a virtual display element may be displayed based on input from an operator, such as, but not limited to, voice commands, gestures, or foot switches. In some exemplary embodiments, one or more virtual display elements associated with a given traceable medical device may be displayed only when the traceable medical device is not in the surgical field. In some exemplary embodiments, one or more virtual display elements associated with a given traceable medical device may be displayed according to a preselected medical procedure stage.

[0124] although Figure 3A , Figure 3B and Figure 4A-4C An exemplary virtual display element in the form of a linear annotation is shown, but it should be understood that the virtual display element can have many different forms. For example, in an exemplary embodiment, a virtual display element can be rendered to represent a medical device, such as a traceable medical device associated with the intraoperative direction and position detected, an untraceable device to be adopted in the subsequent stage of the medical procedure, and / or the shape of a surgical implant associated with a traceable medical device or an untraceable medical device. In another exemplary embodiment, a virtual display element can take the form of a group of collinear shapes discretely displayed along the intraoperative axis, such as but not limited to a circular linear array. In another exemplary embodiment, a virtual display element can include a shape (e.g., a circle or an arrow) indicating the far end of a traceable medical device. In yet another exemplary embodiment, a virtual display element can include a directional feature (e.g., an arrow) indicating a direction associated with a traceable medical device, such as a distal direction. It should be understood that a virtual display element can provide a visual indication of an intraoperative position, an intraoperative direction, and / or an intraoperative direction and position.

[0125] In some exemplary embodiments, multiple virtual display elements can be displayed with enhanced video transmission, wherein at least one subset of the multiple virtual display elements is associated with different intraoperative directions and positions of a traceable medical probe. In an exemplary embodiment, two or more virtual display elements associated with different intraoperative directions and positions of a traceable medical probe can be displayed simultaneously. In some exemplary embodiments, different intraoperative directions and positions can be associated with different surgical operations involving a traceable medical instrument. In other exemplary embodiments, different intraoperative directions and positions can be associated with different time points, while a traceable medical instrument is moved from one position to another. In other exemplary embodiments, two or more virtual display elements can be associated with different traceable medical instruments, respectively.

[0126] In some exemplary embodiments, such as Figure 4A-4C In the exemplary embodiment shown in , multiple annotated intraoperative video transmissions can be generated and displayed, wherein the multiple video transmissions involve different perspectives (different camera alignments) relative to the surgical field. The display of multiple enhanced video transmissions may be advantageous because they provide multi-directional perspectives that help align one or more non-trackable medical devices relative to a virtual display element. As described above, in the case where a surface detection system with a stereo camera for surface detection is employed, the intraoperative video transmission can be generated using two cameras of the surface detection system. Alternatively, a single intraoperative video transmission can be generated based on video transmissions obtained from one camera or based on video transmissions obtained from two cameras.

[0127] In an exemplary embodiment, wherein the tracking system includes two cameras defining a stereo pair, the ability of an observer to unambiguously determine the intraoperative orientation and position of a trackable medical device based on the position and orientation of a virtual display element in an augmented video transmission may be impaired due to the proximity of the trackable medical device relative to a plane including orientation axes respectively associated with the cameras (axes defined along the respective viewing directions of the cameras). In other words, if an intraoperative axis associated with the intraoperative orientation of the trackable medical device is located on, or close to, the baseline of the stereo camera system, the degenerate axis prevents the observer from uniquely determining the intraoperative orientation of the trackable medical device based on the display of the virtual display element in the augmented video transmission.

[0128] An example of this is in Figure 5A-Figure 5C , where a virtual display element associated with a previously determined intraoperative position and orientation of a tracked medical instrument is displayed at 330 in the form of a linear annotation along an intraoperative axis associated with the tracked medical instrument. Figure 5A-Figure 5C Each of the figures also shows the alignment of the untracked medical instrument with the linear annotation 330, where the vertical tilt angle of the untracked medical instrument is different in each figure. Despite the change in the tilt angle of the untracked medical instrument, the alignment degeneracy in the intraoperative video feed causes each different orientation of the untracked medical instrument to appear to be aligned with the linear annotation 330. This degeneracy is caused by the alignment of the tracked medical instrument in the plane formed by the direction axis of the camera.

[0129] In some exemplary embodiments, this situation can be prevented by detecting degeneracy and providing instructions to the operator. The instructions can instruct the operator to move and / or rotate the tracking system. For example, the instructions may include a displayed or audible warning or alarm. In another exemplary embodiment, the system can automatically rotate the tracking system to eliminate such degeneracy, or prevent it from occurring. In another exemplary embodiment, this degeneracy can be prevented or corrected by automatically repositioning the stereo camera system, such as by controlling one or more motors capable of translating and / or rotating the stereo camera system. Degeneracy can be detected according to many different exemplary embodiments, which involve the position and orientation of the intraoperative axis of the tracked medical tool relative to the camera pair. For example, when the position along the instrument's central axis associated with the functional end of the tracked medical instrument is within a preselected spatial offset relative to the plane defined by the camera direction axis, and when the angle between the intraoperative axis and the plane defined by the camera direction axis is within a predetermined angle range, it can be determined that there is degeneracy. Optionally, the criteria associated with degeneracy can be specified based on the projection of the intraoperative axis onto a plane that includes the camera baseline and is perpendicular to the plane that includes the camera direction axis.

[0130] In some exemplary embodiments where at least one intraoperative video feed is generated and enhanced based on video feeds obtained from one or more cameras of the tracking system, one or more additional cameras (cameras external to the tracking system) may be used to generate one or more additional enhanced intraoperative video feeds, provided the coordinates are transformed. Figure 1 An exemplary embodiment is shown in which an additional tracked camera 70 having a set of fiducial markers 75 attached thereto is used to obtain additional intraoperative video transmission. Tracking of the additional camera 70 helps determine additional coordinate transformations between a reference frame associated with the additional camera and a reference frame associated with the tracking system, thereby allowing the additional intraoperative video transmission to be enhanced with additional virtual display features associated with the intraoperative orientation and position of the trackable medical device. The additional camera can be oriented in a non-head-up configuration. In some exemplary embodiments, one or more additional cameras can be located at the head and / or foot of a patient support (e.g., a treatment couch or bed).

[0131] In another exemplary embodiment, an intraoperative video feed may be generated by one or more tracked cameras, independent of the use of cameras associated with a tracking system or camera of a surface detection system.

[0132] In yet another exemplary embodiment, one or more intraoperative video feeds may be generated based on video data obtained from one or more cameras of the tracking system. In such cases, because the cameras of the tracking system are used to track and generate the one or more intraoperative video feeds, the one or more intraoperative video feeds and the tracked medical device are located in a common reference frame, and no coordinate transformation is required to enhance the one or more intraoperative video feeds with one or more virtual display elements associated with the tracked medical device.

[0133] In another exemplary embodiment, a surface detection system may be used to detect intraoperative surface data and track a medical device, and one or more intraoperative video feeds may be generated based on video data obtained from one or more cameras of the surface detection system. For example, a medical device may be tracked by a surface detection system based on one or more topographical surface reference features, such as described in International Patent Application No. PCT / CA2011 / 050257 and International PCT Patent Application No. PCT / CA2015 / 050939. In such cases, because the cameras of the surface detection system are used to track and generate one or more intraoperative video feeds, the one or more intraoperative video feeds and the tracked medical device are located in a common reference frame, and no coordinate transformation is required to enhance the one or more intraoperative video feeds with one or more virtual display elements associated with the tracked medical device.

[0134] Figure 6An embodiment is shown in which a stereo-rectified camera system and a structured light scanner of a tracking system are used together for registration and guidance purposes, as described, for example, in International Patent Application No. PCT / CA2011 / 050257. Stereo camera calibration data 400 including internal and external parameters is used to generate perspective transformations 420 and 425 to map 3D points to the structured light imaging space on the left and right camera 2D images. The step of generating perspective transformation 410 can actually be completed in the common computer vision SDK OpenCV using the stereoRectify() function. In addition, stereo correction maps 430 and 435 for left and right cameras can also be generated from the calibration data 400 in step 410 to correct and undistort the real-time feeds 490 and 495 displayed to the user. In OpenCV, the generation of stereo correction maps can be performed using the initUndistortRectifyMap() function. The 3D position data 440 from the tracking system can be transferred to the 3D coordinate system of the structured light scanner via calibration transformations 450, 460, for example, using methods described in International Patent Application Nos. PCT / CA2011 / 050257 and PCT Patent Application Nos. PCT / CA2015 / 050939. Once the position data is in the coordinate system of the structured light system, the left and right camera perspective transformations 420 and 425 can be used to project the tracked instrument 3D position into a 2D rendering 480 and 485 within the left and right camera 2D image space. The transformed left and right camera 2D position data 480 and 485 can then be directly superimposed onto the corrected and non-distorted video feed data 510 and 515 from the left and right cameras, respectively. Again for common implementations found in openCV, the correction steps 500 and 505 can be accomplished by calling the remap() function with inputs 430 and 435. The synthesized images 520 and 525 can then be displayed to the user to help orient the untracked instrument.

[0135] Figure 7 The single camera embodiment shown utilizes a tracking system consisting of one or more cameras / sensors to track the position of the medical instrument. The tracking system outputs position information 600 of the tracked instrument. The tracking system is calibrated to the camera, which outputs a 2D image 630 of the surgical field so that the known 3D position in the tracking system coordinate space can be projected into the 2D camera image space. The transformation 610 can generally be described as a perspective transformation for mapping the 3D tool position to coordinates 620 in the 2D camera image space. It can be determined by using a set of known correspondences between points in the tracking system 3D space and corresponding points in the 2D image space. Once the position information has been projected into the 2D image space, it can be directly overlaid or rendered into the 2D image frame 640 for display to the user.

[0136] Now for Figure 8 , a flow chart is provided that illustrates an alternative exemplary method for enhancing intraoperative video transmission based on intraoperative detection of the orientation and position of a trackable medical device. Unlike the above-described method involving enhancing intraoperative video transmission with a virtual display element associated with a previously determined intraoperative position and orientation of a tracked medical probe, the present exemplary embodiment involves enhancing intraoperative video transmission with a virtual display element associated with the current (e.g., present or real-time) orientation and position of the tracked medical device in real time. Thus, the virtual display element can be dynamically displayed so that it can be seen moving in a fixed spatial relationship relative to the tracked medical device in the intraoperative video transmission.

[0137] In step 800, the tracking system is used to detect signals from fiducial markers associated with a trackable medical device while the camera is used to generate an intraoperative video feed that is subsequently displayed intraoperatively (e.g., on a display window of a guidance user interface). The camera is configured such that a coordinate transformation between the camera's reference frame and the tracking system's reference frame is known. The signals are processed in step 805 to determine an intraoperative orientation and position of the trackable medical device in a reference frame associated with the tracking system, e.g., as described above with reference to Figure 2 described.

[0138] In step 810, the current intraoperative orientation and position of the trackable medical device as determined in step 805 is transformed into the reference frame of the camera using a known coordinate transformation between the reference frame of the camera and the reference frame of the tracking system. After the current intraoperative orientation and position of the trackable medical device is transformed into the reference frame of the camera, the intraoperative video feed may be enhanced with virtual display features associated with the current intraoperative orientation and position of the trackable medical device, as shown in 815.

[0139] For example, in one exemplary embodiment, an intraoperative video feed may be augmented (e.g., annotated) at a specified spatial location, or offset relative to a tracked medical probe, a virtual display element presenting a tool or implant associated with the tracked medical instrument to be used during the medical procedure. A virtual display element may be dynamically displayed in the intraoperative video feed or at a fixed spatial relationship relative to the tracked medical instrument. In one exemplary embodiment in which the virtual display element is an implant (or implantable device), the trackable medical instrument may include an elongated rod having a longitudinal axis, and a virtual display of the surgical implant may be displayed relative to the distal end of the elongated rod along the longitudinal axis in a distal direction.

[0140] Although the examples provided herein refer to medical instruments for performing spinal procedures, it should be understood that the methods and systems described herein can be widely applied to a variety of medical procedures, i.e., a variety of traceable and non-traceable medical instruments. For example, the methods and systems described herein can be used for guided medical procedures, such as but not limited to medical procedures involving biopsy, deep brain stimulation electrode placement, robotic surgery, multi-electrode placement for epilepsy, and laser interstitial therapy. Examples of tracked medical instruments include, but are not limited to, probes, aspirators, drills, taps, screwdrivers, electrocautery, endoscopes, laparoscopes, pedicle explorers, needles, and examples of non-traceable medical instruments include, but are not limited to, taps, screwdrivers, k-wires, pliers, scissors, scalpels, clamps, retractors, curettes, specula, needle drivers, specula, punches, and rongeurs.

[0141] Reference Fig. 9A , shows a top view of a surgical operating room, wherein a surgeon 750 is shown standing to the left of a patient 700 who is in a prone position. A guidance camera system 740 is located above the patient 700. A guidance viewing screen 730 is located to the right of the patient, with the screen facing the surgeon. During the augmented reality display, a video feed of the patient 770 is shown, wherein the patient is oriented in a vertical orientation, and as a result, the horizontal hand movements of the surgeon 750 are displayed in a vertical orientation on the view screen 730. This configuration can be confusing and frustrating to the surgeon.

[0142] In order to properly present the surgeon's movements, the user interface may be configured to facilitate optional rotation of the displayed video feed. Fig. 9B As shown, the user interface may include a button 760 that may be selected to produce a rotation of the displayed video feed. In various exemplary embodiments, the video feed may be rotated continuously or by fixed rotation increments, such as 45 degrees, 90 degrees, or 180 degrees, in response to user input to align the direction of the patient as seen in the video feed with the direction of the patient in the direction of the operating room.

[0143] Fig. 9C An example is shown where the guide viewing screen 730 is located near the patient's head. During the augmented reality display, the patient's video feed 770 can be rotated so that the patient is vertically aligned, thereby aligning the surgeon's real movements with the movements displayed on the video feed. In the exemplary embodiment shown, the video feed can be rotated (e.g., via user interface button 760) 90 degrees to align the patient's orientation as seen in the video feed with the patient's orientation in the operating room.

[0144] Fig.9DAnother exemplary embodiment is shown in which the guide viewing screen 730 is positioned at an inclination angle α relative to the patient's superior-inferior axis. During the augmented reality display, the video feed of the patient 770 can be rotated (e.g., via user interface button 761) α degrees to align the orientation of the patient seen in the video feed with the orientation of the patient in the operating room.

[0145] The above specific embodiments have been shown by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the specific forms disclosed, but to cover all modifications, equivalent forms and alternative forms that fall within the spirit and scope of the present disclosure.

Claims

1. A system for providing intraoperative guidance for aligning an untracked medical instrument during a medical procedure, the system comprising: Tracking subsystem; monitor; a plurality of cameras configured to generate intraoperative video feeds involving different viewing angles by different camera aiming; as well as Computer hardware operably coupled to the tracking subsystem and the camera, wherein the computer hardware includes a memory coupled to one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: detecting, using the tracking subsystem, a signal from a fiducial marker associated with a trackable medical device; processing the signal to determine an intraoperative orientation and position of the trackable medical device in a reference frame associated with the tracking subsystem; Using the camera to obtain intraoperative video transmission; displaying the intraoperative video feed on the display; and At a subsequent stage of the medical procedure, a coordinate transformation between a reference frame associated with the camera and a reference frame associated with the tracking subsystem is employed to enhance the intraoperative video transmission, such that each intraoperative video transmission includes a virtual display feature that identifies the previously determined intraoperative orientation and position of the trackable medical device, thereby providing a multi-directional viewing angle, so that at a subsequent stage of the medical procedure, an untracked medical device can be visually aligned relative to the previously determined intraoperative orientation and position of the trackable medical device.

2. The system of claim 1, wherein the tracking subsystem comprises a first camera and a second camera, the first camera and the second camera defining a stereo camera pair, the system further comprising: If the accuracy of achieving visual alignment of the intraoperative orientation and position of the untracked medical device relative to the trackable medical device based on the displayed virtual display elements is deemed to be compromised due to alignment degeneracy, instructions are provided to the operator directing the operator to move and / or rotate the tracking subsystem.

3. The system of claim 1, wherein the computer hardware is configured such that the virtual display feature includes a virtual display of the trackable medical device.

4. The system of claim 1, wherein the computer hardware is configured such that the virtual display feature includes a virtual display of the untracked medical device.

5. The system of claim 1, wherein the computer hardware is configured such that the virtual display feature includes a virtual display of a surgical implant.

6. The system of claim 1, wherein the computer hardware is configured to perform additional operations comprising: determining a distal position corresponding to a distal tip of the trackable medical instrument based on the intraoperative direction and position of the trackable medical instrument; and The virtual display feature is rendered such that at least a portion of the virtual display feature indicates the distal location.

7. The system of claim 1, wherein the computer hardware is configured such that the intraoperative direction and position are determined upon receiving input from an operator.

8. The system of any one of claims 1-6, wherein the computer hardware is configured to augment the intraoperative video feed with the virtual display feature when the trackable medical device disappears from the field of view of the tracking subsystem.

9. The system of any one of claims 1-6, wherein the computer hardware is configured such that the virtual display feature is selectively displayed based on input received from an operator.

10. The system of any one of claims 1-6, wherein the intraoperative direction and position is a first intraoperative direction and position, and the virtual display feature is a first virtual display feature, wherein the computer hardware is configured to perform additional operations comprising: determining additional intraoperative orientation and position of the trackable medical device; and In subsequent stages of the medical procedure, the intraoperative video feed is augmented with additional virtual display features associated with additional intraoperative orientations and positions of the trackable medical instrument.

11. The system of claim 10, wherein the computer hardware is configured such that the first virtual display feature and the additional virtual display feature are rendered simultaneously during a subsequent phase of the medical procedure.

12. The system of any one of claims 1-6, wherein the trackable medical instrument is a first trackable medical instrument, and the intraoperative direction and position are a first intraoperative direction and position, and the virtual display feature is a first virtual display feature, wherein the computer hardware is configured to perform additional operations comprising: determining additional intraoperative orientation and position of additional trackable medical devices; and The intraoperative video feed is enhanced with additional virtual display features associated with additional intraoperative orientations and positions of the additional trackable medical instrument.

13. The system of claim 12, wherein the computer hardware is configured such that the first virtual display feature and the additional virtual display feature are selectively displayed during one or more subsequent stages of the medical procedure based on input received from a user.

14. The system of claim 1, wherein the computer hardware is configured to perform additional operations further comprising: processing an intraoperative orientation and position of the trackable medical instrument to determine an intraoperative axis based on a model of the trackable medical instrument.

15. The system of claim 14, wherein the computer hardware is configured such that the intraoperative axis is determined by a longitudinal axis of the elongated portion of the trackable medical device defined by the model.

16. The system of claim 14 or 15, wherein the tracking subsystem comprises a first camera and a second camera, the first camera and the second camera defining a stereo camera pair, wherein the computer hardware is configured to perform additional operations comprising: determining whether the intraoperative axis lies within a plane including directional axes respectively associated with the first camera and the second camera; and When the intraoperative axis is within a plane including directional axes respectively associated with the first camera and the second camera, instructions are provided to the operator to direct the operator to move and / or rotate the tracking subsystem to remove degeneracy in determining the intraoperative axis.

17. The system of claim 14 or 15, wherein the computer hardware is configured such that the virtual display feature is a linear annotation indicating the position and orientation of the intraoperative axis.

18. The system of claim 17, wherein the computer hardware is configured such that the linear annotation includes a directional marker indicating a direction relative to a distal tip of the trackable medical device.

19. The system of claim 17, wherein the computer hardware is configured to perform additional operations comprising: determining a distal position corresponding to a distal tip of the trackable medical instrument based on the intraoperative direction and position of the trackable medical instrument; and The virtual display feature is rendered such that the linear annotation overlaps the distal location or extends from a location adjacent to the distal location.

20. The system of any one of claims 1-6, wherein the camera is an integral part of a surface detection system, and wherein the coordinate transformation is obtained based on a calibration between the surface detection system and the tracking subsystem.

21. The system of claim 20, wherein the surface detection system is a structured light surface detection system.

22. The system of claim 20, wherein the computer hardware is configured to perform additional operations comprising: acquiring and displaying an additional intraoperative video feed using an additional camera, wherein the additional camera includes additional fiducial markers, and wherein the position and orientation of the additional camera is tracked by the tracking subsystem; and Additional coordinate transformations between a reference frame associated with the additional camera and a reference frame associated with the tracking subsystem are employed to enhance the additional intraoperative video feed with additional virtual display features associated with the intraoperative orientation and position of the trackable medical instrument.

23. The system of claim 22, wherein the additional camera is located at the head end or the foot end of the patient support.

24. The system of any one of claims 1-6, wherein the camera includes additional fiducial markers, and wherein the position and orientation of the camera is tracked by the tracking subsystem.

25. The system of claim 24, wherein the camera is movable to allow viewing of the intraoperative video feed annotated with the virtual display features from multiple viewing directions.

26. The system of any one of claims 1-6, wherein the tracking subsystem is rigidly fixed relative to the camera.

27. The system of any one of claims 1-6, wherein the trackable medical instrument is a pedicle probe and the untracked medical instrument is a tap or a screwdriver.

28. The system of claim 1, wherein the tracking subsystem is an electromagnetic tracking system.

29. A system for providing intraoperative guidance for aligning an untracked medical device during a medical procedure, the system comprising: Surface detection subsystem; monitor; as well as Computer hardware operably coupled to the surface detection subsystem, wherein the computer hardware includes a memory coupled to one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: using the surface detection subsystem to detect a signal from a surface reference feature associated with a traceable medical device; processing the signal to determine an intraoperative orientation and position of the trackable medical device in a reference frame relative to the surface detection subsystem; Using multiple cameras of the surface detection subsystem to obtain intraoperative video transmission involving different viewing angles due to different camera alignments; displaying the intraoperative video feed on the display; and In subsequent stages of the medical procedure, the intraoperative video transmission is enhanced so that each intraoperative video transmission includes virtual display features that identify the previously determined intraoperative orientation and position of the trackable medical device, thereby providing a multi-directional viewing angle so that in subsequent stages of the medical procedure, an untracked medical device can be visually aligned relative to the previously determined intraoperative orientation and position of the trackable medical device.

30. A system for providing intraoperative guidance for aligning an untracked medical device during a medical procedure, the system comprising: a tracking subsystem, the tracking subsystem comprising a plurality of cameras having different viewing angles due to different camera alignments; monitor; and Computer hardware operably coupled to the tracking subsystem, wherein the computer hardware includes a memory coupled to one or more processors to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: detecting, using the tracking subsystem, a signal from a fiducial marker associated with a trackable medical device; processing the signal to determine an intraoperative orientation and position of the trackable medical device in a reference frame associated with the tracking subsystem; Using cameras to obtain multiple intraoperative video feeds; displaying the intraoperative video feed on the display; and In subsequent stages of the medical procedure, the intraoperative video transmission is enhanced so that each intraoperative video transmission includes virtual display features that identify the previously determined intraoperative orientation and position of the trackable medical device, thereby providing a multi-directional viewing angle so that in subsequent stages of the medical procedure, an untracked medical device can be visually aligned relative to the previously determined intraoperative orientation and position of the trackable medical device.

Images