Surgical systems and methods using intraoperative video

Abstract

Systems and methods incorporate both tracker-based navigation and intraoperative video to track surgical objects during a procedure. A tracker is disposed relative to a surgical instrument for treating patient tissue at a surgical site, and another tracker is disposed relative to a video device for capturing realtime video of the surgical site. A localizer cooperates with the trackers to generate data indicative of poses of the instrument and video device in a known coordinate system. A controller defines a virtual object in the known coordinate system and associated with the patient tissue based on the tracked pose of the video device and the realtime video. Based on the tracked pose of the instrument relative to the virtual object in the known coordinate system, the controller at least one of controls operation of the instrument and switches between a virtual model display of the patient tissue and the realtime video.

Description

SURGICAL SYSTEMS AND METHODS USING INTRAOPERATIVE VIDEORELATED APPLICATION(S)

[0001] This application claims priority to and all the benefits of U.S. Prov. Patent Appl. No. 63 / 661,882 filed June 19, 2024, the entire contents of which is hereby incorporated by reference herein.BACKGROUND

[0002] Surgical navigation systems assist surgeons by tracking the positions of various objects during surgical procedures. Typical surgical navigation systems are configured to track the poses of surgical devices, such as instruments and implants, relative to patient tissue of interest to a surgical procedure, and display information based on the tracked poses to guide the surgeon in performing the procedure. Some surgical navigation systems are configured to take a more active role in a surgical procedure, such as by controlling actuation of a surgical instrument based on the instrument’s tracked pose relative to the patient tissue of interest. For instance, when tracking data indicates a surgical instrument has contacted or is near contacting patient tissue to be avoided, these surgical navigation systems can stop actuation of the surgical instrument so as to prevent or reduce manipulation of such tissue.

[0003] Certain objects, such as relatively small anatomical structures, non-rigid anatomical structures, and hard-to-reach anatomical structures, are difficult to track with sufficient accuracy using conventional surgical navigation technology. A need exists for improved systems that overcome such deficiencies and provide other benefits as described in more detail below.SUMMARY

[0004] This Summary introduces a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to limit the scope of the claimed subject matter nor identify key features or essential features of the claimed subject matter.

[0005] A first general aspect includes a navigation system having a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgicalobjects including a first anatomical object, a surgical instrument for treating patient tissue at a surgical site, and a video device for capturing realtime video of the surgical site. The system also includes at least one controller coupled to the localizer and the video device and configured to: track a pose of each of the surgical objects in the known coordinate system based on the localizer data; receive the realtime video captured by the video device and that corresponds to the tracked pose of the video device in the known coordinate system; generate and / or manipulate a virtual boundary associated with a second anatomical object in the known coordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object, where the second anatomical object is a soft tissue object; and control operation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system.

[0006] Implementations of this aspect may optionally include one or more of the following features.

[0007] The at least one controller may be configured to manipulate the virtual boundary over time based on the received realtime video and the tracked pose of the video device to provide a dynamic virtual boundary associated with the soft tissue object.

[0008] The at least one controller may be configured to generate the virtual boundary in the known coordinate system based on a first frame of the received realtime video and subsequently manipulate the virtual boundary in the known coordinate system based on a second frame of the received realtime video following the first frame.

[0009] The at least one controller may be configured to generate and / or manipulate the virtual boundary associated with the second anatomical object in the known coordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object by being configured to: identify a region in a frame of the received realtime video that likely corresponds to the second anatomical object based on the tracked pose of the video device and the tracked pose of the first anatomical object in the known coordinate system; and generate and / or manipulate the virtual boundary in the known coordinate system based on the identified region and the tracked pose of the video device. Optionally, the at least one controller may be configured to identify the region in the frame of the received realtime video that likely corresponds to the second anatomical object based on the tracked pose of the video device, the tracked pose of the first anatomical object, andanatomical data indicating a positional relationship between the first anatomical object and the second anatomical object. Optionally, the anatomical data includes may be a patient image of the first and second anatomical objects. Optionally, the patient image may be a preoperative patient image.

[0010] The at least one controller may be configured to generate and / or manipulate the virtual boundary associated with the second anatomical object in the known coordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object by being configured to: determine a positional relationship between the first anatomical object and the received realtime video based on the tracked pose of the first anatomical object and of the video device in the known coordinate system; and input the received realtime video and the positional relationship into a machine learning algorithm configured to output segmentation data indicative of a location of the second anatomical object within a frame of the received realtime video. Optionally, the output segmentation data may indicate an identity of the second anatomical object, and the at least one controller is configured to control operation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system and the identity of the second anatomical object indicated by the output segmentation data.

[0011] The first anatomical object may be a bone. Optionally, the at least one controller may be configured to: receive a patient image illustrating the bone impinging on the second anatomical object; display at least a portion the patient image with an annotation indicative of a portion of the bone to resect to relieve the impingement of the second anatomical object; and adjust the annotation of the patient image as the portion of the bone is resected to indicate a remaining portion of the bone to be resected based on the received realtime video and the tracked pose of the video device the known coordinate system. Optionally, the adjustment of the annotation may also be based on the tracked pose of the surgical instrument in the known coordinate system. Optionally, the at least one controller may be configured to display the realtime video superimposed with an implant to be received in a space of a portion of the bone to be resected as the portion of the bone is resected based on the tracked pose of the bone and of the video device in the known coordinate system, and optionally based on the tracked pose of the surgical instrument in the known coordinate system.

[0012] The second anatomical object may be an object to be avoided. Optionally, the second anatomical object is a nerve or artery. Optionally, the at least one controller is configured to control the operation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system by being configured to, responsive to determining that the surgical instrument has reached or crossed the virtual boundary, cease or reduce operation of an actuator of the surgical instrument.

[0013] The second anatomical object may be a spinal disc with disc tissue to be resected. Optionally, the at least one controller may be configured to: receive a patient image of the spinal disc; adjust the virtual boundary associated with the spinal disc as the disc tissue is resected based on the received realtime video and the tracked pose of the video device in the known coordinate system; and display the patient image with an annotation indicative of an amount of remaining disc material to be resected from the spinal disc based on the virtual boundary. Optionally, the adjustment of the virtual boundary may be further based on the tracked pose of the surgical instrument. Optionally, the at least one controller may be configured to display the realtime video superimposed with an implant to be received in a space of the resected disc tissue as the disc tissue is resected based on the virtual boundary and the tracked pose of the video device in the known coordinate system.

[0014] The at least one controller may be configured to: track a number of contact instances between the surgical instrument and the second anatomical object based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system and / or based on the realtime video; and control operation of the surgical instrument based on the tracked number of contact instances.

[0015] A second general aspect includes a method for treating a skull base tumor of a patient. The method includes tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument in a known coordinate system. The method also includes positioning the video device adjacent a sphenoid bone of the patient trans-nasally. The method also includes receiving, by the navigation system, realtime video of an environment of the tumor captured by the video device and that corresponds to the tracked pose of the video device. The method also includes forming a hole in the sphenoid bone. The method also includes defining, by the navigation system, a first virtual boundary associated with the formed hole in the known coordinate system and a second virtual boundary associated with the tumor in the knowncoordinate system based on the received realtime video and the tracked pose of the video device. The method also includes inserting the surgical instrument through the formed hole trans-nasally while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the first virtual boundary in the known coordinate system. The method also includes after inserting the surgical instrument through the formed hole, resecting the tumor with the surgical instrument while receiving feedback from the navigation system indicative of the tracked pose of the surgical instrument relative to the second virtual boundary in the known coordinate system.

[0016] Implementations of this aspect may optionally include one or more of the following features.

[0017] The method may include the surgical navigation system tracking the pose of the video device and of the surgical instrument in the known coordinate system based on localizer data generated by a localizer and indicative of the pose of the video device and of the surgical instrument in the known coordinate system.

[0018] The method may include: determining, by the surgical navigation system and based on a received patient image, anatomical data indicative of a positional relationship between the sphenoid bone and a carotid artery of the patient; defining, by the surgical navigation system, a third virtual boundary associated with the carotid artery in the known coordinate system based on the positional relationship, the received realtime video, and the tracked pose of the video device; and inserting the surgical instrument through the formed hole trans-nasally while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the third virtual boundary in the known coordinate system. Optionally, the method may include the navigation system: tracking a pose of the sphenoid bone in the known coordinate system; and defining the third virtual boundary associated with the carotid artery in the known coordinate system based on the tracked pose of the sphenoid bone, the positional relationship, the received realtime video, and the tracked pose of the video device.

[0019] The method may include: adjusting, by the navigation system, the second virtual boundary as a portion of the tumor is resected based on the received realtime video and the tracked pose of the video device; and continuing resection of the tumor with the surgical instrument while receiving feedback from the navigation system that is indicative of thetracked pose of the surgical instrument relative to the second virtual boundary as adjusted in the known coordinate system.

[0020] A third general aspect includes a navigation system for tracking a surgical instrument for treatment of a skull-base tumor of a patient trans-nasally. The navigation system includes a video device configured to capture realtime video of an environment of the tumor, the video device including a light source for exciting a fluorescing agent in patient tissue. The system also includes a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgical objects including the surgical instrument and the video device. The system also includes at least one controller coupled to the localizer and the video device and configured to: define first, second, and third virtual boundaries in the known coordinate system, the first virtual boundary associated with a position proximal to a hole formed in a sphenoid bone of the patient for accessing the tumor, the second virtual boundary associated with a position of the hole formed in the sphenoid bone, and the third virtual boundary associated with a position between the second virtual boundary and the tumor; responsive to the tracked pose of the surgical instrument indicating a distal end of the surgical instrument has reached or crossed the first virtual boundary, trigger display of a first view including the realtime video overlayed with a graphic indicating a target trajectory for inserting the surgical instrument through the hole formed in the sphenoid bone based on the tracked pose of the video device; responsive to the tracked pose of the surgical instrument indicating the distal end of the surgical instrument has passed through the first virtual boundary and reached or crossed the second virtual boundary, trigger display of a second view including the realtime video overlayed with annotations of anatomical features of the realtime video; and responsive to the tracked pose of the surgical instrument indicating the distal end of the surgical instrument has passed through the first and second virtual boundaries and reached or crossed the third virtual boundary, trigger display of a third view including the realtime video filtered to highlight fluorescence emitted from the tumor in the realtime video.

[0021] A fourth general aspect includes a method for treating a spine of a patient. The method includes tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument relative to a vertebra in a known coordinate system. The method also includes receiving, by the navigation system, realtime video of the spine of the patient captured by the video device and that corresponds to the tracked pose of the video device. The method alsoincludes tracking a pose of an exiting nerve root adjacent the vertebra in the known coordinate system based on the tracked pose of the video device, the received realtime video, and the tracked pose of the vertebra. The method also includes inserting the surgical instrument through a triangle defined by the exiting nerve root and the vertebra while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the exiting nerve root and the vertebra in the known coordinate system. The method also includes treating tissue of the spine with the surgical instrument while extended through the triangle.

[0022] Implementations of this aspect may optionally include one or more of the following features.

[0023] The triangle may be defined by the exiting nerve root, a superior endplate of the vertebra, and an articular process of the vertebra.

[0024] The triangle may be defined by the exiting nerve root, a superior endplate of the vertebra, and the spinal cord.

[0025] A fifth general aspect includes a navigation system having a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgical objects including a bone of a patient in a surgical site, a surgical instrument for treating the bone, and a video device for capturing realtime video of the surgical site. The system also includes at least one controller coupled to the localizer and the video device and configured to: track a pose of the video device relative to the bone in the known coordinate system based on the localizer data, determine a tracking error based on the realtime video of the surgical site and the tracked pose of the video device relative to the bone, and trigger an action responsive to determining the tracking error.

[0026] Implementations of this aspect may optionally include one or more of the following features.

[0027] The triggered action may include triggering an alert device.

[0028] The triggered action may include adjusting a virtual boundary associated with the bone in the known coordinate system.

[0029] The triggered action may include adjusting a registration between the bone and a tracker that is attached to the bone and is detectable by the localizer to generate the data indicative of the pose of the bone in the known coordinate system.

[0030] A sixth general aspect includes a method for treating nerve compression between first and second vertebra of a patient. The method includes tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument relative to the first and second vertebra in a known coordinate system. The method also includes receiving, by the navigation system, realtime video of the first and second vertebra captured by the video device and that corresponds to the tracked pose of the video device. The method also includes resecting a portion of at least one of the first and second vertebra with the surgical instrument while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the first and second vertebra. The method also includes tracking, by the navigation system, a pose of the nerve between the first and second vertebra in the known coordinate system based on the realtime video and the tracked poses of the video device, the first and second vertebra, and the surgical instrument during the resection of a portion of the at least one of the first and second vertebra. The method also includes continuing resection of at least one the first and second vertebra while receiving feedback from the navigation system that is indicative of the pose of the nerve in the known coordinate system relative to the surgical instrument and the first and second vertebra.

[0031] A seventh general aspect includes a navigation system having a video device for capturing realtime video of a surgical site. The system also includes a localizer configured to generate data indicative of a pose of a surgical instrument in a known coordinate system, the surgical instrument for treating patient tissue at the surgical site. The system also includes at least one controller coupled to the localizer and the video device and configured to: define a virtual boundary associated with the patient tissue in the known coordinate system; track a pose of the surgical instrument relative to the virtual boundary in the known coordinate system based on the localizer data; and based on the tracked pose of the surgical instrument relative to the virtual boundary, trigger a display to switch between a first view illustrating a pose of the surgical instrument relative to a virtual model of the patient tissue and the realtime video of the surgical site.

[0032] An eighth general aspect includes a system that combines patient navigation, surgical tool navigation, and machine-learning based computer vision algorithms to automatically identify, localize, and protect critical anatomy (e.g., Carotid (ICA), Cranial Nerves, Optic Nerves, Spine Nerves, etc. identified in the realtime video data generated by the video device(e.g., Endoscope, Microscope, Exoscope), by defining a fixed or dynamic moving virtual boundary around the 2D or 3D segmented bone or soft tissue in a known coordinate system, which prevents a surgical instrument coming into contact or close proximity to the tissue of such anatomy, and thus prevents thermal damage or direct physical damage to such anatomy.

[0033] Additional general aspects and implementations include methods including steps corresponding to the configured functions of the above-described systems; systems including one or more processors and one or more memories that upon execution of the one or more processors are configured to cause the one or more processes to implement the above-described methods; and computer program products comprising computer executable instructions stored on a non-transitory computer readable medium and configured, upon execution by one or more processors, to cause the one or more processors to implement the steps of the above-described methods.

[0034] Two or more of the above aspects and / or implementations may also be combined in whole or in part. For instance, features of the first aspect and its implementations related to defining virtual boundaries may be used to define virtual boundaries of the other aspects and implementations. As a further example, display and navigation guidance functions of the first aspect and its implementations may be utilized with the other aspects and implementations.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Advantages of the present disclosure will be readily appreciated as the same becomes better understood by reference to the following Detailed Description when considered in connection with the accompanying drawings.

[0036] FIG. 1 illustrates a surgical system including a plurality of surgical instrument assemblies and a surgical navigation system for tracking a surgical instrument associated with each of the various surgical instrument assemblies.

[0037] FIG. 2 illustrates an exemplary layout of an operating room including at least one of the surgical instrument assemblies and the surgical navigation system of FIG. 1 for performing a surgical procedure on a patient.

[0038] FIG. 3 illustrates a portion of a patient’s spine including disc tissue targeted for treatment during a surgical procedure.

[0039] FIG. 4 illustrates a model view that may be displayed by the surgical navigation system of FIG. 1 during the surgical procedure on the spine of FIG. 3 prior to treatment of the target disc tissue.

[0040] FIG. 5 illustrates an intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during the surgical procedure on the spine of FIG. 3 prior to treatment of the target disc tissue.

[0041] FIG. 6 illustrates an intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during the surgical procedure on the spine of FIG. 3 after partial treatment of the target disc tissue.

[0042] FIG. 7 illustrates a model view that may be displayed by the surgical navigation system of FIG. 1 during the surgical procedure on the spine of FIG. 3 after partial treatment of the target disc tissue.

[0043] FIG. 8 illustrates an intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during a spinal fusion procedure.

[0044] FIG. 9 illustrates a model view that may be displayed by the surgical navigation system of FIG. 1 during a skull base surgical procedure to resect tumorous tissue.

[0045] FIG. 10 illustrates a combined model and intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during the skull base surgical procedure.

[0046] FIG. 11 illustrates an annotated intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during the skull base surgical procedure.

[0047] FIG. 12 illustrates a fluorescence-based intraoperative video view that may be displayed by the surgical navigation system of FIG. 1 during the skull base surgical procedure.DETAILED DESCRIPTION

[0048] Referring to the Figures, wherein like numerals indicate like or corresponding pails through the several views, FIG. 1 illustrates an exemplary surgical system 10 that may include a surgical navigation system 100 for tracking one or more surgical objects during a surgical procedure, such as patient tissue and one or more surgical instrument assemblies 200, 300, 400. Each surgical instrument assembly 200, 300, 400 may include a surgical instrument 220, 320, 420 to assist the medical professional, such as a surgeon, in executing the surgical procedure.

[0049] The surgical navigation system 100 may include a navigation controller 140 and a user interface coupled to the navigation controller 140. The user interface may include one or more display units 120 and one or more user input devices 130. The display unit(s) 120 of the surgical navigation system 100 may be driven by the navigation controller 140, and to this end may be configured to display graphical user interfaces (GUIs) including information related to a current surgical procedure, and / or various elements, prompts or data entry boxes for interaction by the user, such as via the user input device(s) 130. In some implementations, the user input device(s) 130 may include a touch screen interface integrated with the display unit(s) 120.

[0050] For example, the display unit(s) 120 may be configured to display a text box or prompt that allows the surgeon to manually enter or select the type of surgical procedure to be performed. The display unit(s) 120 may also be configured to display patient data, such as preoperative patient images, intraoperative patient images, and / or intraoperative video of patient tissue of interest, and / or display virtual models of the patient tissue of interest, such as developed from patient images and / or video. The preoperative and / or intraoperative patient images may be based on MRI scans, radiological scans or computed tomography (CT) scans of the patient's anatomy. The intraoperative patient video, which may be analog, digital, or a digitized optical image (e.g., generated by a hybrid optical digital scope) may be generated by endoscopes, microscopes, exoscopes, or other types of video devices disposed in the operating room relative to the surgical site.

[0051] The display unit(s) 120 may be further configured to display a surgical plan for a surgical procedure, such as overlaid on the patient data. The surgical plan may include information related to the type of surgical procedure being performed, the patient tissue of interest (e.g., target sites defining patient tissue to be treated, patient tissue to be avoided, anatomical structures or objects including target sites and / or patient tissue to be avoided), the types of surgical devices (e.g., instruments and / or implants) to be used during the surgical procedure, and / or operating settings for the surgical navigation system 100. The surgical plan may also include planned trajectories for surgical devices relative to the patient tissue of interest, and / or may include target poses for implants or other medical devices to be implanted during the surgical procedure relative to the patient tissue of interest. The display unit(s) 120 may be configured to display at least a portion of the surgical plan, such as the plannedtrajectories and / or target poses, on the patient data, such as overlaid on patient images and / or video of the patient tissue of interest, and / or relative to virtual models representative of the patient tissue of interest, so as to assist the surgeon to visualize and / or implement the surgical plan.

[0052] It is contemplated that the surgical navigation system 100 may be configured to display and / or project a holographic image of at least a portion of the surgical plan, such as the planned trajectories and / or target poses, on surfaces in the operating room, such as surfaces of the patient tissue of interest. It is also contemplated that the surgical navigation system 100 may be configured to display and / or project a holographic image of at least a portion of the surgical plan on a head unit worn by the surgeon, such as a lens, shield, or glasses of the head unit. An exemplary configuration of a surgical navigation system 100 including a display unit worn by the surgeon to display planned trajectories and / or target poses is disclosed in International Patent Appl. No. PCT / IB2018 / 053130, the entirety of which is hereby incorporated by reference herein.

[0053] The user input device(s) 130 may allow the user to interact with the GUIs displayed on the display unit(s) 120 to input or modify the surgical plan. As an example, the user may interact with the user input device(s) 130 to segment and / or annotate patient tissues of interest to a surgical procedure in patient images and / or video of the patient, such as patient tissue to be treated and thus defining a target site for the surgical procedure, and patient tissue to be avoided. The patient tissue to be avoided may include critical anatomical structures such as cortical walls, nerves, blood vessels including veins and arteries, and / or similar critical anatomical structures that the surgeon wishes to avoid. As described in more detail below, a user may further interact with the user input device(s) 130 to establish virtual objects defining zones surrounding the patient tissue of interest. A zone for patient tissue to be treated may be referred to as a fly zone for a surgical device, and a zone for patient tissue to be avoided may be referred to as a no-fly zone for the surgical device. The virtual objects may be used by the surgical navigation system 100 to provide guidance for the surgical procedure. The user may also interact with the user input device(s) 130 to select surgical devices to be used in the surgical procedure, and / or select trajectories and / or target poses for the surgical devices, such as a target depth relative to the patient tissue of interest for a surgical instrument or implant.

[0054] The user input device(s) 130 may also allow the user to interact with the GUIs displayed on the display unit(s) 120 to input or modify the patient data, which may include the patient images and / or video of the patient tissue of interest, boundaries and / or annotations of the patient tissue of interest relative to the patient images and / or video, and / or virtual models representative of the patient tissue interest, such as developed from the patient images and / or video based on the boundaries and / or annotations. As an example, the user may interact with the user input device(s) 130 to identify and / or adjust boundaries of the patient tissue of interest within patient images and / or video, such as boundaries generated using a segmentation routine or algorithm. The user may also annotate the patient tissue of interest within the patient images and / or video, input and / or adjust dimensions of the patient tissue of interest, and / or create and / or adjust the virtual models representative of such patient tissue, as described in more detail below.

[0055] The surgical navigation system 100, or more particularly the navigation controller 140, may be configured to utilize segmentation of patient images and / or video to identify boundaries of patient tissue of interest within the patient images and / or video, recognize an identity of and annotate varying patient tissue of interest within the patient images and / or video, and facilitate the generation of virtual objects associated with the patient tissue of interest for guiding movement of a surgical instrument as described herein. At least for the intraoperative patient video, the surgical navigation system 100, or more particularly the navigation controller 140, may also be configured to identify boundaries of, recognize an identity of and annotate, and generate virtual models of surgical devices illustrated in the video utilizing segmentation. Segmentation as utilized herein may refer to automatic segmentation, semi-automatic segmentation, or manual segmentation. Exemplary methods of segmentation that may be performed by the surgical navigation system 100 are disclosed in International Patent Application No. PCT / US2021 / 073160, which is hereby incorporated by reference in its entirety.

[0056] In one example of segmentation which may be implemented by the navigation system 100, patient video data or image data may be received, such as by the navigation controller 140, and such as from an imaging device or video device described in more detail below. In the case of received video data, one or more image frames may then be extracted from the video data. In some implementations, the received video data may be a stream of interoperativevideo data captured by a video device and received in real time during a procedure, in which case the image frames may be extracted from the video data in real-time, i.c., as the surgical procedure is being performed. In some implementations, the navigation controller 140 may be configured to automatically extract image frames from the received video data at a predetermined interval or frequency. Alternatively or additionally, one or more image frames can be extracted from the received video data in response to user input, such as the surgeon pushing a button or other user input device to indicate that they want to capture an image from the video data at or around any particular moment in time.

[0057] The image frame(s) or patient image data may then be processed using an artificial intelligence (Al) based segmentation model. For instance, the model may include one or more machine learning classifiers that are configured to process each image and determine whether the image includes features of interest to a given surgical procedure (e.g., anatomical features and / or surgical device features). More particularly, the one or more machine learning classifiers may be configured to automate the process of identifying which images of the one or more images include the features of interest. In some implementations, the one or more machine classifiers may be created using a supervised training process in which one or more training images (e.g., images that are known to contain specific features) are used to create a classifier that can determine if an image input into the machine classifier contains the features.

[0058] For instance, the one or more machine classifiers may be configured to identify one or more anatomical features shown in an image by being configured to identify one or more types of tissue shown in the image, such as bone, nerve, blood vessel, disc, skin, brain, or any other type of tissue that can be viewed using an imaging tool such as described herein. The one or more machine classifiers may also be configured to determine an anatomical identity of and / or label one or more anatomical objects shown in the image, such as a particular vertebrae, disc, nerve, blood vessel, bone, and so on. Additionally or alternatively, the one or more machine learning classifiers may be configured to determine whether an image shows a particular surgical step being or having been performed. For instance, the one or more machine classifiers may be configured to determine if a particular image shows a damaged disc or a disc post-repair.

[0059] In some implementations, multiple machine classifiers may be configured to work collectively with one another to determine what features are present in a given image. As anexample, a first machine learning classifier may be used to determine if a particular anatomical feature is present in a given image. If the machine classifier finds that it is more likely than not that the image contains a particular anatomical feature, then the image can be sent to a machine learning classifier corresponding to the anatomical feature to determine whether a procedure step associated with the feature is shown or represented by the image. For instance, if a machine classifier determines that a particular image likely shows disc tissue, then the image may be sent to a machine classifier configured to determine if the image shows a damaged disc or a disc post-repair. However, if the former machine classifier determines the image does not likely show disc tissue, then the image may not be sent to the latter machine classifier for further processing.

[0060] In some implementations, the one or more machine classifiers may include one or more image clarity classifiers that are configured to determine how clear or obscured a particular image is. During a surgical procedure, certain conditions can obfuscate or make an image unclear. For instance blood, turbidity, bubbles, smoke, or other debris can make the quality of an image poor, in which case the image may not be suitable to facilitate navigation or other features as described herein. These one or more machine classifiers may thus be configured to determine if an image is clear enough to be analyzed, classified, and / or annotated. If a predetermined number of images are determined unclear, then the navigation controller 140 may be configured to trigger an alert, such as via the display unit(s) 120, to indicate that the imaging device and / or video device is producing unclear images and should be cleaned and / or adjusted.

[0061] In some implementations, the one or more machine classifiers may be configured to generate a classification metric that is indicative of whether or not a particular feature (that the machine classifier is configured to determine) exists within a particular image. Thus, rather than making a binary determination (yes or no) as to whether a particular image includes a particular feature, the classification metric may be configured to inform the process as to how likely it is that a particular image includes a particular feature. As an example, a machine classifier that is configured to classify whether an image contains disc tissue can output a classification metric in the range of 0 to 1 with 0 indicating that it is extremely unlikely that a particular image shows disc tissue and 1 indicating that it is extremely likely that a particular image shows disc tissue. Intermediate values between 0 and 1 can indicate the likelihood thatan image contains a particular feature. For instance, if a machine learning classifier outputs a 0.8, it can mean that it is more likely than not that the image shows disc tissue, while a classification metric of .1 means that it is not likely that the image contains disc tissue.

[0062] In some implementations, the one or more machine classifiers may be implemented using one or more convolutional neural networks (CNNs). CNNs are a class of deep neural networks that can be especially useful for analyzing visual imagery to determine whether certain features exist in an image. Each CNN used to generate a machine classifier may include one or more layers, with each layer of the CNN configured to aide in the process of determining whether a particular image includes a feature that the overall CNN is configured to determine. Alternatively or additionally, the CNNs may be configured as Region-Based Convolutional Networks (R-CNNs) that can not only determine if a particular image contains a feature, but can identify the specific location in the image where the feature is shown.

[0063] Once the one or more images have been processed by the one or more machine learning classifiers, a determination may be made as to what features are present within a particular image. This determination may be based on the classification metric(s) output from each of the machine learning classifiers. As an example, each of the classification metrics generated by each of the machine learning classifiers can be compared to a pre-determined threshold, and if the classification metric exceeds the pre-determined threshold, then a determination may be made that the image contains the feature corresponding to that machine learning classifier. As an example, if a machine learning classifier processing an image for disc tissue outputs a classification metric of .7, and the pre-determined threshold is set at .5, then a determination may be made that the image shows disc tissue. In some examples, a determination can be made for each and every machine learning classifier that a given image is processed through. Different predetermined thresholds may be used for different machine learning classifiers.

[0064] Once the determinations are made as to what features a particular image contains, those determinations can be used to generate one or more annotations to be applied to the image. Annotating an image in this context can refer to applying text next to or overlaid on the image. Alternatively or additionally, annotating an image can refer to appending metadata to an image that indicates what features the image was found to contain. In some implementations, annotating an image can also or alternatively include placing one or more graphics on the image, such as graphics outlining and / or highlighting each varying feature.

[0065] Once the annotations have been generated, such annotations may be displayed. In some implementations, displaying the annotations can refer to displaying an image and its corresponding annotations on a display of a computing device such as a computer or a tablet, such as on the display unit(s) 120.

[0066] In another example of segmentation which may additionally or alternatively be implemented by the navigation system 100, the surgeon or other user may utilize the user input device(s) 130 to define a geometric primitive corresponding to a region of interest within an image. A method of defining geometric primitives for the purpose of segmentation may comprise the steps of: manual pre-segmentation by defining enclosing geometric primitives in a 3D or 2D patient image for generating initial envelopes; analyzing the anatomy within the pre-segmented geometric primitives; using the result of the analysis for adjustment of the envelopes; and visualizing the envelopes. Analysis of the anatomy within a pre-segmented geometric primitive may include computing cell affiliations of pixels or voxels within the presegmented geometric primitive, and the adjustment of a visualized envelope may be achieved by computing a boundary around the pixels or surface mesh of the voxels which are affiliated completely and / or partially to a given cell. Further, the adjustment of a visualized envelope may be achieved by optimizing the type, orientation, position and / or size of the enclosing geometric primitive. Exemplary methods and systems for defining a geometric primitive and guiding a surgical instrument based thereon arc disclosed in U.S. Patent Publ. No. 2017 / 0148173 and U.S. Patent Publ. No. 2017 / 0231714, both of which are hereby incorporated by reference herein in their entirety.

[0067] In some examples, following an automated or semi-automated segmentation process, a manual intervention process may be provided by the surgical navigation system 100, such as via the display unit(s) 120 and user input device(s) 130, to allow the surgeon to identify and / or adjust feature boundaries within the image, and / or to identify and / or adjust locations within the image that is appropriate to place an annotation or associate with an annotation. For example, in cases where one or more machine classifiers are not able to automatically identify a precise location within an image to identify with an annotation (e.g. a precise location of a tissue condition or type of tissue), then a manual intervention process may be provided in a user interface to allow the surgeon to identify or confirm in the image the precise location, and to either position the annotation at that location or to otherwise associate the annotation withthat location, such as hy connecting it with an arrow pointing to the location. Responsive to a user manually indicating an annotation of an image, such user input may be used to further train the Al-based segmentation model (e.g., machine learning classifiers) so as to be able to automatically recognize and annotate such feature in the future.

[0068] The navigation controller 140 may be configured to implement the functions, features and processes of the surgical navigation system 100 described herein. In one example, the navigation controller 140 may include a processor, memory, and non-volatile storage. The processor may include one or more devices selected from microprocessors, micro-controllers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, and / or any other devices that manipulate signals (analog or digital) based on operational instructions read into the memory from the non-volatile storage. The memory may include a single memory device or a plurality of memory devices including, but not limited to, read-only memory (ROM), random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, and / or any other device capable of storing information. The non-volatile storage may include one or more persistent data storage devices such as a hard drive, optical drive, tape drive, non-volatile solid state device, and / or any other device capable of persistently storing information.

[0069] The navigation controller 140 may operate under control of surgical navigation software, such as embodied by computer-executable instructions residing in the non-volatile storage, including one or more modules and / or operating instructions. The surgical navigation software, upon execution by the navigation controller 140, may be configured to cause the navigation controller 140 to implement the various functions, features, and processes of the surgical navigation system 100 disclosed herein. As one example, the surgical navigation software may include software configured to control operation of the surgical instruments 220, 320, 420. Such software may include a boundary generator. An exemplary system for and method of boundary generation may be found in U.S. Patent Publ. No. 2004 / 0034283, which is hereby incorporated by reference herein in its entirety.

[0070] The boundary generator may be a software program or module configured to generate one or more virtual boundaries for constraining movement and / or operation of the surgicalinstruments 220, 320, 420 relative to the patient tissue of interest. In some examples, the boundary generator may provide one or more virtual boundaries that define a virtual drill and / or driver guide (e.g., a virtual implant planning guide). Virtual boundaries may also be provided to control operation of the surgical instruments 220, 320, 420 relative to patient tissue to be avoided, and / or to set target depths and / or target poses relative to the patient tissue of interest. Virtual boundaries may also be used to trigger display of different types of views to help guide the surgeon based on the pose of surgical instruments 220, 320, 420 relative to the patient tissue of interest. Exemplary views, as described in more detail below, may include model views, intraoperative video views, combination views, and fluorescence views.

[0071] The virtual boundaries may be one-dimensional (ID), two-dimensional (2D), or three- dimensional (3D), and may comprise a point, line, axis, trajectory, plane (an infinite plane or plane segment bounded by the anatomy or other boundary), volume or other shapes, including complex geometric shapes. The virtual boundaries may be represented by pixels, point clouds, voxels, triangulated meshes, other 2D or 3D models, combinations thereof, and the like. Boundaries to ensure that surgical devices are positioned at a desired depth may be defined by a virtual planar' boundary, a virtual volumetric boundary, or other forms of virtual boundary. U.S. Patent Publ. No. 2018 / 0333207 and U.S. Patent No. 8,898,043 are hereby incorporated by reference herein, and any of their features, including those relating to the use of virtual boundaries and / or implants, may be used to facilitate planning or execution of the surgical procedure.

[0072] Virtual boundaries may also be referred to as virtual objects. The virtual boundaries may be defined with respect to a virtual model of patient tissue of interest, such as an anatomical structure or object defining a target site (which may also be referred to as a target volume) of patient tissue to be treated and / or an anatomical structure or object including tissue to be avoided. In other words, the points, lines, axes, trajectories, planes, volumes, and the like that are associated with the virtual boundaries may each be defined in a coordinate system that is fixed relative to a coordinate system of such a virtual model such that tracking of the virtual model (e.g., via tracking the associated patient tissue to which it is registered) in a known coordinate system enables tracking of the virtual boundary in the known coordinate system.

[0073] The virtual models of patient tissue of interest may each be registered to a patient tracker 150 (FIG. 2) disposed relative to the patient tissue in the surgical site such that thevirtual boundaries associated with the virtual model and fixed relative its coordinate system arc also associated with and / or registered to the patient tracker 150. The virtual boundaries may be implant- specific (e.g., defined based on a size, shape, volume, etc. of a planned implant) and / or patient-specific (e.g., defined based on the patient’s anatomy). The virtual boundaries may be created pre-operatively, intra-operatively, or combinations thereof. In other words, the virtual boundaries may be defined before the surgical procedure begins, during the surgical procedure (including during tissue removal), or combinations thereof. The virtual boundaries may be provided in numerous ways, such as by the navigation controller 140 creating them, receiving them from other sources / systems, or the like. The virtual boundaries may be stored in memory for retrieval and / or updating, such as on the fly during a surgical procedure.

[0074] It is contemplated that in some cases, the virtual boundaries may comprise multiple planar boundaries used to delineate multiple target depths (e.g., three target depths) for separate instruments to be used in a single procedure. For example, the virtual boundaries may include a first virtual boundary representing a target depth for a drill to bore a hole, a second virtual boundary representing a target depth for a tap to thread the hole, and a third virtual boundary representing a target depth for a driver to insert a screw in the threaded hole. These multiple virtual boundaries can be activated, one at a time, by the navigation controller 140 to constrain cutting to one plane at a time. The navigation controller 140 may track the poses of the surgical instruments 220, 320, 420 relative to the virtual boundaries.

[0075] The surgical navigation system 100 may include a tracking unit 110, also referred to as a localizer, having one or more sensors 115 for tracking the poses of the patient tissue of interest, and for tracking the poses of surgical devices such as the surgical instruments 220, 320, 420 relative to the virtual boundaries associated with the patient tissue of interest. The sensors 115 may comprise cameras, such as CCD cameras, CMOS cameras, and / or optical image cameras, magnetic sensors, radio frequency sensors, or any other sensor adapted to detect and / or sense the poses of instrument tracking devices 230, 330, 430, also referred to as instrument trackers, of the surgical instrument assemblies 200, 300, 400, and also to detect and / or sense the poses of the patient tracking devices 150, also referred to as patient trackers, described above. Descriptions of suitable tracking units 110 for the surgical navigation system100 may be found in U.S. Patent Publ. No. 2017 / 0333137, which is hereby incorporated by reference herein in its entirety.

[0076] As illustrated in FIG. 1, the surgical system 10 may include various surgical instrument assemblies 200, 300, 400 in communication with the surgical navigation system 100. The surgical instrument assemblies 200, 300, 400 may be configured to be in wired and / or wireless communication with the surgical navigation system 100, or more particularly the navigation controller 140, such as directly or over one or more networks. Each of the surgical instrument assemblies 200, 300, 400 may have a number of similar- components capable of performing similar functions and / or operations. Similar components between each of the various surgical instrument assemblies 200, 300, 400 will include the same two-digit number with a leading 2, 3, or 4 to reflect the associated surgical instrument assembly 200, 300, 400. For example, each of the surgical instrument assemblies 200, 300, 400 may include a surgical instrument 220, 320, 420.

[0077] The surgical system 10 may comprise a first surgical instrument assembly 200 in communication with the surgical navigation system 100. The first surgical instrument assembly 200 may include a first surgical instrument 220, such as a surgical drill, driver or saw, including a handpiece 225. The handpiece 225 may comprise a housing 210 configured to house the components of the first surgical instrument 220. The handpiece 225 may be shaped to define a handle or grip portion for the surgeon to hold while performing a medical procedure. Suitable handpieces are described in U.S. Patent No. 5,747,953 and U.S. Patent Publ. No. 2024 / 0130736, which are hereby incorporated by reference herein in their entirety.

[0078] The first surgical instrument 220 may further comprise a first instrument controller 215 and an actuator 245, the latter of which may be realized as a motor. Each of the first instrument controller 215 and the actuator 245 may be disposed within the handpiece 225 of the first surgical instrument 220. The first instrument controller 215 and the actuator 245 may be in communication with one another, and the first instrument controller 215 may be configured to control the operation of the actuator 245, and by extension the first surgical instrument 220. For example, the first surgical instrument 220 may comprise an end-effector 240, such as a drill bit for boring a hole or a driver for inserting a screw. The end-effector 240 may be coupled to the handpiece 225 of the first surgical instrument 220 such that the actuator 245 may be operably coupled to the end-effector 240. For example, the actuator 245 may be configured torotate the end-effector 240 when realized as a drill bit to bore a hole and / or remove biological tissue.

[0079] The first instrument controller 215 may thus be configured to control operation of the end-effector 240 via control of the actuator 245. The first instrument controller 215 may also be in communication with the navigation controller 140, and may be configured to receive data related to the pose (e.g., position and orientation) of the first surgical instrument 220 from the navigation controller 140, and / or receive instructions for controlling operation of the first surgical instrument 220 from the navigation controller 140, and / or transmit data related to the operation of the first surgical instrument 220 to the navigation controller 140. For example, the navigation controller 140 may be configured to communicate instructions to the first instrument controller 215 to adjust operation of the first surgical instrument 220 based on the pose of the first surgical instrument 220 as detected by the surgical navigation system 100 relative to virtual boundaries associated with patient tissue of interest. The first instrument controller 215 may be structurally similar to the navigation controller 140, and may thus operate under control of stored software configured upon execution by one or more processors of the first instrument controller 215 to implement the functions, features and processes of the first instrument controller 215 described herein.

[0080] The first surgical instrument assembly 200 may also comprise a power source 260. The power source 260 may be removably coupled to the handpiece 225 of the surgical instrument 220. For example, the power source 260 may comprise a removable battery pack. It is also contemplated that the power source 260 may be formed as part of, or disposed within, the handpiece 225 of the first surgical instrument 220. The power source 260 may be in electrical communication with the first instrument controller 215 and / or the actuator 245 and configured to selectively provide power to the actuator 245 to rotate the end-effector 240. The power source may also be a surgical console providing power to the first surgical instrument 220 with a cord.

[0081] The first surgical instrument assembly 200 may also comprise a first alert device 255. The first alert device 255 may comprise an audible, a tactile, and / or a visually perceptible device. For instance, the first alert device 255 may include a tactile alert device separate from the first surgical instrument 220 and configured to be worn on and / or felt through the surgeon’ s foot, wrist, hand, or fingers. Additionally or alternatively, the first alert device 255 may includea tactile alert device incorporated in the first surgical instrument 220, and thus be generally configured to be felt through the surgeons hand or fingers when holding the first surgical instrument 220.

[0082] The first alert device 255 may be configured to be in communication with the first instrument controller 215 or directly with the navigation controller 140. The first instrument controller 215 or navigation controller 140 may be configured to send a signal to activate the first alert device 255 to provide a warning or notification based on a pre-programmed condition or setting. In some instances, the first alert device 255 may be at least partially implemented by the actuator 245. For instance, the first instrument controller 215 or navigation controller 140 may be configured to send a signal to adjust operation of the actuator 245 in a manner that is perceptible by the surgeon, such as by causing the actuator 245 to exhibit a change in sound or tactile feel, as the provided warning or notification.

[0083] The first surgical instrument assembly 200 may also comprise an instrument tracking device 230. The instrument tracking device 230 may be coupled to the handpiece 225 of the first surgical instrument 220. In some implementations, the instrument tracking device 230 may comprise a plurality of markers 235 that are identifiable by the tracking unit 110 of the surgical navigation system 100. The markers 235 may comprise passive tracking elements (e.g., reflectors) for transmitting light signals (e.g., reflecting light emitted from the tracking unit 110) to the sensor(s) 115. In other configurations, the markers 235 may be configured as active tracking markers (e.g., LED’s). It is also contemplated that the markers 235 may comprise a combination of active and passive arrangements.

[0084] The markers 235 may be arranged in a defined or known position and orientation relative to the first surgical instrument 220 to allow the surgical navigation system 100 to determine the pose of the first surgical instrument 220 relative to a known coordinate system, such as a coordinate system specific to the tracking unit 110, based on an identified pose of the instrument tracking device 230 in the known coordinate system, which in turn may be determined based on positions of the markers 235 in the known coordinate system that a e detected by the tracking unit 110. For example, the markers 235 may be registered to a coordinate system specific to the first surgical instrument 220 to allow the surgical navigation system 100 to determine the pose of the end-effector 240 or cutting portion of the first surgicalinstrument 220 relative to the virtual boundaries described herein, which may also be generated and tracked in the known coordinate system.

[0085] The surgical system 10 may also comprise a second surgical instrument assembly 300 to be used with the surgical navigation system 100. The second surgical instrument assembly 300 may comprise a second surgical instrument 320, such as a high-speed surgical bur, shaver, or microdebrider, including a handpiece 325. The handpiece 325 may be coupled to a console 310 that is configured to control the operation of various components of the second surgical instrument 320. The handpiece 325 may be shaped to define a handle or grip portion for the surgeon to hold while performing a medical procedure. Exemplary second surgical instruments that connect to consoles may be found in U.S. Patent Publ. No. 2024 / 0099738 and U.S. Patent Publ. No. 2019 / 0117322, which are hereby incorporated by reference herein in their entirety.

[0086] The second surgical instrument assembly 300 may further comprise a second instrument controller 315 and an actuator 345, the latter of which may be realized as a motor. The second instrument controller 315 may be disposed within the console 310 of the second surgical instrument assembly 300. The actuator 345 may be disposed within the handpiece 325 of the second surgical instrument 320. The second instrument controller 315 and the actuator 345 may be in communication with one another, and the second instrument controller 315 may be configured to control the operation of the actuator 345, and by extension the second surgical instrument 320. For example, the second surgical instrument 320 may be coupled to the console 310 by a cord connecting the second instrument controller 315 to the actuator 345 to allow communication between the second instrument controller 315 and the actuator 345 to control operation of the actuator 345.

[0087] The second instrument controller 315 may also comprise an end-effector 340, such as a high-speed cutting bur. The end-effector 340 may be coupled to the handpiece 325 of the second surgical instrument 320 such that the actuator 345 may be operably coupled to the endeffector 340. For example, the actuator 345 may be configured to actuate the end-effector 340 when realized as a high-speed cutting bur to grind and / or remove biological tissue from the surgical site. The second instrument controller 315 may thus be configured to control the operation of the end-effector 340 via control of the actuator 345.

[0088] The second instrument controller 315 may also be in communication with the navigation controller 140, and may be configured to receive data related to the pose (c.g., position and orientation) of the second surgical instrument 320 from the navigation controller 140, and / or receive instructions for controlling operation of the second surgical instrument 320 from the navigation controller 140, and / or transmit data related to the operation of the second surgical instrument 320 to the navigation controller 140. For example, the navigation controller 140 may be configured to communicate instructions to the second instrument controller 315 to adjust operation of the second surgical instrument 320 based on the pose of the second surgical instrument 320 as detected by the surgical navigation system 100 relative to virtual boundaries associated with patient tissue of interest. It is also contemplated that additional surgical instruments may be coupled the console 310 and / or in communication with the second instrument controller 315 disposed within the console 310. The second instrument controller 315 may be structurally similar to the navigation controller 140, and may thus operate under control of stored software configured upon execution by one or more processors of the second instrument controller 315 to implement the functions, features and processes of the second instrument controller 315 described herein.

[0089] The second surgical instrument assembly 300 may also comprise a power source. The power source may be coupled to the console 310 of the second surgical instrument assembly 300 and configured to provide energy to the actuator 345 of the second surgical instrument 320 to actuate the end-effector 340. It is also contemplated that the console 310 may comprise a cord configured to be plugged into an outlet that is connected to an electrical grid for supplying energy to the second surgical instrument assembly 300. The power source may be in electrical communication with the second instrument controller 315 and / or the actuator 345 and configured to selectively provide power to the actuator 345 to actuate the end-effector 340.

[0090] The second surgical instrument assembly 300 may also comprise a second alert device 355. The second alert device 355 may comprise an audible, a tactile, and / or a visually perceptible device. For instance, the second alert device 355 may include a tactile alert device separate from the second surgical instrument 320 and configured to be worn on and / or felt through the surgeon’s foot, wrist, hand, or fingers. In some implementations, the second alert device 355 may be at least partially disposed in a footswitch 350 of the second surgical instrument assembly 300, which may be for user control of the second surgical instrument 320.In this case, the second alert device 355 may include a tactile device that outputs vibrations to be felt in the user’s foot. Additionally or alternatively, the second alert device 355 may include a tactile alert device incorporated in the second surgical instrument 320, and thus be generally configured to be felt through the surgeons hand or fingers when holding the second surgical instrument 320.

[0091] The second alert device 355 may be configured to be in communication with the second instrument controller 315 or directly with the navigation controller 140. The second instrument controller 315 or navigation controller 140 may be configured to send a signal to activate the second alert device 355 to provide a warning or notification based on a pre-programmed condition or setting. In some instances, the second alert device 355 may be at least partially implemented by the actuator 345. For instance, the second instrument controller 315 or navigation controller 140 may be configured to send a signal to adjust operation of the actuator 345 in a manner that is perceptible by the surgeon, such as by causing the actuator 345 to exhibit a changed sound or tactile feel, as the provided warning or notification.

[0092] The second surgical instrument assembly 300 may also comprise an instrument tracking device 330. The instrument tracking device 330 may be coupled to the handpiece 325 of the second surgical instrument 320. The instrument tracking device 330 may be similar to the instrument tracking device 230 described above for the first surgical instrument assembly 200. For instance, in some implementations, the instrument tracking device 330 may include markers 335 arranged in a defined or known position and orientation relative to the second surgical instrument 320 to allow the surgical navigation system 100 to determine the pose of the second surgical instrument 320 relative to a known coordinate system, such as a coordinate system specific to the tracking unit 110, based on an identified pose of the instrument tracking device 330 in the known coordinate system, which in turn may be determined based on positions of the markers 335 in the known coordinate system that are detected by the tracking unit 110. As an example, the markers 335 may be registered to a coordinate system specific the second surgical instrument 320 to allow the surgical navigation system 100 to determine the pose of the end-effector 340 of the second surgical instrument 320 relative to the virtual boundaries described herein, which like the instrument tracking device 330 may also be generated and tracked in the known coordinate system.

[0093] The surgical system 10 may also comprise a third surgical instrument assembly 400 in communication with the surgical navigation system 100. The third surgical instrument assembly 400 may comprise a third surgical instrument 420, such as an ultrasonic or RF surgical instrument, including a handpiece 425. The handpiece 425 may be coupled to a console 410 that is configured to control the operation of various components of the third surgical instrument 420. The handpiece 425 may be shaped to comprise a handle or grip portion for the surgeon to hold while performing a medical procedure. Exemplary third surgical instruments that connect to consoles may be found in U.S. Patent No. 10,016,209 and U.S. Patent Publ. No. 2020 / 0179033, which are hereby incorporated by reference herein in their entirety.

[0094] The third surgical instrument assembly 400 may further comprise a third instrument controller 415 and an actuator 445, the latter of which may be realized as an ultrasonic transducer. The third instrument controller 415 may be disposed within the console 410 of the third surgical instrument assembly 400. The actuator 445 may be disposed within the handpiece 425 of the third surgical instrument 420. The third instrument controller 415 and the actuator 445 may be in communication with one another.

[0095] The actuator 445 when realized as an ultrasonic transducer may comprise a piezoelectric element configured to expand and contract upon the application of an electric current to the piezoelectric element. The piezoelectric element may comprise a plurality of disc-shaped piezoelectric elements arranged end to end in a stack. The third instrument controller 415 may be configured to control the operation of the actuator 445, and by extension the third surgical instrument 420. For example, the third surgical instrument 420 may comprise an end-effector 440, such as an ultrasonic tip assembly. The ultrasonic tip assembly may include a horn of which an ultrasonic tip portion vibrates at an ultrasonic wave velocity as the piezoelectric element(s) expand and contract. The ultrasonic tip assembly may also include an external sheath at least partially disposed over the horn except for the ultrasonic tip portion.

[0096] The end-effector 440 may be coupled to the handpiece 425 of the third surgical instrument 420 such that the actuator 445 may be operably coupled to the end-effector 440. For example, the actuator 445 may be configured to actuate the end-effector 440 when realized as the ultrasonic tip assembly to grind and / or remove biological tissue from the surgical site. The third instrument controller 415 may be in communication with the actuator 445 andconfigured to control the flow of electric current to the piezoelectric element(s), controlling operation of the actuator 445, and by extension the end-effector 440.

[0097] The third instrument controller 415 may also be in communication with the navigation controller 140 and may be configured to receive data related to the pose (e.g., position and orientation) of the third surgical instrument 420 from the navigation controller 140, and / or receive instructions for controlling operation of the third surgical instrument 420 from the navigation controller 140, and / or transmit data related to the operation of the third surgical instrument 420 to the navigation controller 140. For example, the navigation controller 140 may be configured to communicate instructions to the third instrument controller 415 to adjust operation of the third surgical instrument 420 based on the pose of the third surgical instrument 420 as detected by the surgical navigation system 100 relative to virtual boundaries associated with patient tissue of interest. The third instrument controller 415 may be structurally similar to the navigation controller 140, and may thus operate under control of stored software configured upon execution by one or more processors of the third instrument controller 415 to implement the functions, features and processes of the third instrument controller 415 described herein.

[0098] The third surgical instrument assembly 400 may also comprise a power source. The power source may be coupled to the console 410 of the third surgical instrument assembly 400 and configured to provide energy to the actuator 445 of the third surgical instrument 420 to actuate the end-effector 440. For example, the power source may comprise a removable battery pack. It is also contemplated that the console 410 may comprise a cord configured to be plugged into an outlet that is connected to an electrical grid for supplying energy to the third surgical instrument assembly 400. The power source may be in electrical communication with the third instrument controller 415 and / or the actuator 445 and configured to selectively provide power to the actuator 445 to actuate the end-effector 440.

[0099] The third surgical instrument assembly 400 may also comprise a third alert device 455. The third alert device 455 may comprise an audible, a tactile, and / or a visually perceptible device. For instance, the third alert device 455 may include a tactile alert device separate from the third surgical instrument 420 and configured to be worn on and / or felt through the surgeon’s foot, wrist, hand, or fingers. In some implementations, the third alert device 455 may be at least partially disposed in a footswitch 450 of the third surgical instrument assembly400, which may be for user control of the third surgical instrument 420. In this case, the third alert device 455 may include a tactile device that outputs vibrations to be felt in the user’s foot. Additionally or alternatively, the third alert device 455 may include a tactile alert device incorporated in the third surgical instrument 420, and thus be generally configured to be felt through the surgeons hand or fingers when holding the third surgical instrument 420.

[0100] The third alert device 455 may be configured to be in communication with the third instrument controller 415 or directly with the navigation controller 140. The third instrument controller 415 or navigation controller 140 may be configured to send a signal to activate the third alert device 455 to provide a warning or notification based on a pre-programmed condition or setting. In some instances, the third alert device 455 may be at least partially implemented by the actuator 445. For instance, the third instrument controller 415 or navigation controller 140 may be configured to send a signal to adjust operation of the actuator 445 in a manner that is perceptible by the surgeon, such as by causing the actuator 445 to exhibit a changed sound or tactile feel, as the provided warning or notification.

[0101] The third surgical instrument assembly 400 may also comprise an instrument tracking device 430. The instrument tracking device 430 may be coupled to the handpiece 425 of the third surgical instrument 420. The instrument tracking device 430 may be similar to those described above for the other surgical instrument assemblies 200, 300. For instance, in some implementations, the instrument tracking device 430 may include markers 435 arranged in a defined or known position and orientation relative to the third surgical instrument 420 to allow the surgical navigation system 100 to determine the pose of the third surgical instrument 420 relative to a known coordinate system, such as a coordinate system specific to the tracking unit 110, based on an identified pose of the instrument tracking device 430 in the known coordinate system, which in turn may be determined based on positions of the markers 435 in the known coordinate system that are detected by the tracking unit 110. As an example, the markers 435 may be registered to a coordinate system specific the third surgical instrument 420 to allow the surgical navigation system 100 to determine the pose of the end-effector 440 of the third surgical instrument 420 relative to the virtual boundaries described herein, which like the instrument tracking device 430 may also be generated and tracked in the known coordinate system.

[0102] The surgical instrument assemblies 200, 300, 400 described above are intended to be exemplary instruments and / or configurations within the surgical system 10 but arc not intended to be limiting. Other types and forms of surgical instrument assemblies are contemplated. While a plurality of exemplary surgical instrument assemblies 200, 300, 400 are described as being a part of the surgical system 10 and in communication with the surgical navigation system 100, it is contemplated that the surgical system 10 may only comprise a single surgical instrument assembly 200, 300, 400 and the surgical navigation system 100. Furthermore, while the surgical system 10 illustrated in FIG. 1 includes three surgical instrument assemblies 200, 300, 400 and a single surgical navigation system 100, it is contemplated that the surgical system 10 may be configured to include any combination of surgical instrument assemblies 200, 300, 400, and / or surgical navigation systems 100. For example, the surgical system 10 may include a single surgical instrument assembly 200, 300, 400 and a plurality of surgical navigation systems 100. Further exemplary features of the surgical system 10 may include those of the system described in U.S. Patent Publ. No. 2022 / 0338938, which is hereby incorporated by reference herein in its entirety.

[0103] The surgical navigation system 100 may further include a video device 470. The video device 470 may be configured to capture realtime video data of the surgical site during the procedure, including the patient tissue of interest and surgical devices disposed relative to the surgical site. As non-limiting examples, the video device 470 may be realized as an endoscope, exoscope, microscope, fluoroscope, or combination thereof, each offering unique advantages depending on the specific surgical procedure. For example and without limitation, the video device 470 may include an endoscope such as described in US Patent No. 8,248,413 and / or International Patent Appl. No. PCT / US2021 / 073160. Additionally or alternatively, the video device 470 may include a microscope such as described in U.S. Patent Publ. No. 2021 / 0186627, and / or U.S. Patent Publ. No. 2024 / 0148252, and / or U.S. Patent Appl. No. 18 / 603,982 filed March 13, 2024 and titled “Neurosurgical Methods And Systems For Detecting And Removing Tumorous Tissue.” Additionally or alternatively, the video device 470 may include a fluoroscopic device such as described in U.S. Patent Publ. No. 2024 / 0148252, and / or U.S. Patent Appl. No. 18 / 603,982 filed March 13, 2024 and titled “Neurosurgical Methods And Systems For Detecting And Removing Tumorous Tissue”, and / or International Patent Appl. No. PCT / US2021 / 073160. Additionally or alternatively, thevideo device 470 may include vision devices such as described in U.S. Patent Publ. No. 2017 / 0333137, and / or U.S. Patent Publ. No. 2022 / 0117682, and / or U.S. Patent Publ. No. 2024 / 0156542. The disclosure of each of the above publications and applications is hereby incorporated by reference herein in its entirety.

[0104] The navigation controller 140 may be coupled to the video device 470, and may be configured to receive and display the received realtime video data on the display unit(s) 120. Responsive to receiving the realtime video data, the navigation controller 140 may also be configured to apply a segmentation model or algorithm to the frames of the video data that identifies the boundaries of (e.g., locates) and / or determines an identify of and / or annotates features present in the video (e.g., anatomical features and / or surgical device features), as described above. Additionally or alternatively, the surgeon may interact with the video to identify and / or adjust the boundaries and / or annotations. The navigation controller 140 may also be configured to generate and store virtual models representative of the patient tissue of interest and / or surgical devices from the video data based on the identified boundaries and / or annotations.

[0105] The navigation controller 140 may also be configured to generate tracking data indicative of a tracked pose the patient tissue of interest and / or surgical devices in the known coordinate system based on the identified boundaries and / or annotations of the video data. More specifically, the navigation controller 140 may be configured to track a pose of a coordinate system of the video data captured by the video device 470 relative to the known coordinate system, such as by tracking a pose of the video device 470 relative to the known coordinate system and applying a known registration between the coordinate system of the video device 470 and the captured video data to the tracked pose. Based on the tracked pose of the coordinate system of the captured video data relative to the known coordinate system, the navigation controller 104 may be configured to determine a position of components (e.g., pixels) of the captured video in the known coordinate system, thereby enabling the navigation controller 140 to track the features identified in the captured video data in the known coordinate system.

[0106] In some implementations, the video data generated by the video device 470 may have a known relationship with the known coordinate system such that components of the video data, such as pixels, may be transformed to the known coordinate system based on the knownrelationship. In some implementations, the video device 470 may be positioned relative to the tracking unit 110 according to a predefined relationship that corresponds to the known relationship. Alternatively, the video device 470 may include a video device tracker 472, which may function similarly to the instrument tracking devices 230, 330, 430 described above. The pose of the video device tracker 472 may be registered to a coordinate system of the video data. Thus, responsive to determining a pose of the video device tracker 472 relative to the known coordinate system, such as using the tracking unit 110, the navigation controller 140 may be configured to determine the known relationship between the coordinate system of the video data and the known coordinate system based on a pose of the video device tracker 472 relative to the known coordinate system and the registration between the video device tracker 472 and the video data.

[0107] As described in more detail below, the integration of the video device 470 in the surgical navigation system 100 enables tracking of certain difficult to track objects, such as smaller anatomical objects difficult to be attached to a tracker and soft tissue that tends to change position and / or shape during manipulation of adjacent tissue during the procedure. The navigation controller 140 may thus be configured to utilize the video data generated by the video device 170 to update registrations between anatomy and trackers, generate new virtual objects based on features identified in the video data that may not be associated with a tracker, and / or modify pre-operatively planned virtual objects, such as those defining fly and no fly zones, based on the video data, such as to provide a dynamic virtual boundary that is adjusted with manipulation of surgical objects during a surgical procedure. Incorporation of the video device 470 in the surgical navigation system 100 also enables the navigation controller 140 to dynamically adjust the viewpoint presented to the surgeon so as to present increasingly beneficial and relevant information as a function of the current state of the surgical procedure.

[0108] As described above, the navigation controller 140 may be configured to utilize segmentation routines to locate and / or identify features of interest within the realtime video data generated by the vision device 470. In some implementations, the navigation controller 140 may be configured to leverage the tracked pose of the vision device 470 relative to one or more other surgical objects of interest in the known coordinate system to facilitate this process.

[0109] More specifically, the navigation controller 140 may be configured to determine anatomical data indicating a positional relationship between multiple anatomical objects ofinterest of a patient, such as based on a patient image depicting the anatomical objects. For example, the positional relationship data for a given set of anatomical objects may indicate an angular position of each of the objects relative to the other object(s) (e.g., above, below, etc.), and / or a distance range of each of the objects relative to the other object(s) (e.g., a closest distance and a furthest distance between the surfaces the objects, etc.). Relative to a given set of anatomical objects for which such positional relationship data is determined, at least one of the anatomical objects, such as a bone, may be tracked in the known coordinate system by the localizer 110 as described above. At least one other of these anatomical objects, such as soft tissue and / or other objects which may not be suitable for tracking via a patient tracker 150, may not be tracked in the known coordinate system via the localizer 110.

[0110] Thereafter, when evaluating a frame of the realtime video data to locate and / or identify features of interest, the navigation controller 140 may be configured to identify a region of the frame expected to correspond to one anatomical object (e.g., a soft tissue object) based on the positional relationship data for the one anatomical object, the tracked pose relative to the known coordinate system of another anatomical object (e.g., bone) indicated in the positional relationship data, such as determined based on localizer data generated by the localizer 110, and the tracked pose of the video device 407 in the known coordinate system. As previously mentioned, the one anatomical object expected to be located in the identified region may not be tracked by the localizer 110. Specifically, based on the tracked pose of the video device 407 and the another anatomical object in the known coordinate system, the navigation controller 140 may be configured to determine a pose of the another anatomical object relative to the realtime video data generated by the video device 407 as described above. Then, based on the pose of the another anatomical object relative to the realtime video data generated by the video device 407, the navigation controller 140 may be configured to reference the positional relationship data to predict where the one anatomical object is located within the realtime video data (e.g., above or below the another anatomical object, within a range of distances from the another anatomical object, etc.). The navigation controller 140 may then apply segmentation algorithms as described above to the identified region to locate and / or identify the one anatomical object from the realtime data with improved efficiency relative to searching the entire image frame.

[0111] In some implementations, the above region-based process may be implemented by a machine learning algorithm implemented by the navigation controller 140 as described above. Specifically, the navigation controller 140 may be configured to determine a positional relationship between an anatomical object tracked in the known coordinate system by the localizer 110 and the received realtime video based on the tracked pose of the video device 470 in the known coordinate system as described above, and input the received realtime video and this positional relationship into the machine learning algorithm. The machine learning algorithm may then be configured to output segmentation data indicative of a location and / or identity of another anatomical object within the received realtime video as described above.

[0112] The above methodology may be used to locate and / or identify anatomical objects such as nerves that are being impinged by another anatomical object such as bone, and to assist with surgical planning to help relieve such impingement. For instance, during a surgical procedure, the navigation controller 140 may be configured to define virtual boundaries in the known coordinate system that correspond to the impinged anatomical object, the impinging anatomical object, and a portion of the impinging anatomical object to be resected to relieve the impinged anatomical object using the methodologies described herein. The navigation controller 140 may also be configured to display at least a portion of a received patient image or the realtime video data with an annotation indicative of the portion of the impinging anatomical object to resect to relieve the impingement of impinged anatomical object, such as based on the tracked poses of the defined virtual boundaries associated with the impinging anatomical object relative to the patient image or realtime video in the known coordinate system.

[0113] As the portion of the impinging anatomical object is resected, the navigation controller 140 may be configured to utilize the tracked pose of the impinging anatomical object relative to the realtime video data in the known coordinate system to locate and / or identify the anatomical object being impinged and an updated shape of the impinging anatomical object from the resection as described above (e.g., using predetermined positional relationship data between the anatomical objects), and then configure (e.g., generate and / or update) the aforementioned virtual boundaries based on the located and / or identified objects in the realtime video data and the tracked pose of the realtime video data in the known coordinate system as described herein. In some implementations, the navigation controller 140 may also beconfigured to utilize the tracked pose of the surgical instrument 220, 320, 420 relative to the impinging anatomical object and / or the realtime video data in the known coordinate system to predict a location in the realtime video of the resected portion of the impinging anatomical object in the realtime video data, and process such location to locate the updated boundary of the impinging anatomical object.

[0114] Contemporaneously with configuring the aforementioned virtual boundaries, the navigation controller 140 may be configured to adjust the annotation of the patient image and / or realtime video as the portion of the bone is resected to indicate a remaining portion of the bone to be resected based on the configured virtual boundaries. The navigation controller 140 may also be configured to contemporaneously update the surgical plan, such as by dynamically revising the virtual boundary cor esponding to the portion of the impinging anatomical object to be resected based on distances between the impinging anatomical object and the anatomical object being impinged as indicated by the virtual boundaries. For instance, responsive to the navigation controller 140 determining from the configured virtual boundaries that a shortest distance between a point of the portion of the impinging anatomical object to be resected and the impinged anatomical object is greater than a target threshold, the navigation controller 140 may be configured to revise the virtual boundary associated with the portion of the impinging anatomical object to be resected to omit the point and / or an area of the impinging anatomical object surrounding the point.

[0115] In some implementations, the navigation controller 140 may also be configured to determine a tracking error based on the realtime video data captured by the video device 470 and the tracked pose of a surgical object of interest determined with the localizer 110, and trigger an action accordingly. More specifically, the navigation controller 140 may be configured to compare the pose of the surgical object in the known coordinate system indicated by the localizer data generated by the localizer 110 to the pose of the surgical object in the known system indicated by segmentation of the real time video data and the tracked pose of the video device 470 in the known coordinate system as described above. If the comparison indicates the poses differ by more than a threshold (e.g., 3 mm), the navigation controller 140 may be configured to trigger an action. For instance, the triggered action may include triggering activation of one the alert devices 255, 355, 455, and / or adjusting a virtual boundary associated with the surgical object within the known coordinate system, and / or adjusting aregistration between the surgical object and a tracker attached thereto based on the pose indicated by the realtime video data.

[0116] In a similar- manner, the navigation controller 140 may be configured to compare the shape of a surgical object indicated by segmentation of the realtime video data with the virtual model of the surgical object associated with a tracker by determining a pose of each in the known coordinate system and running a comparison, and adjust the virtual model to match the shape indicated by the realtime video data if the comparison indicates a mismatch.

[0117] As previously described, in some implementations the navigation controller 140 may be configured to determine an identity of objects in the realtime video data for which the virtual boundaries are configured in the known coordinate system. In some implementations, control of a surgical instrument 220, 320, 420 based on the tracked pose of the surgical instrument 220, 320, 420 relative to a virtual boundary in the known coordinate system may depend on the determined identity of the object within the realtime video data that corresponds to the virtual boundary. Specifically, depending the identity of the object, the navigation controller 140 may be configured to configure the virtual boundary as defining a no-fly zone (e.g., the identified object includes tissue to be avoided) or a fly zone (e.g., the identified object includes tissue to be resected).

[0118] Referring to FIG. 2, an exemplary configuration of an operating room or surgical suite for performing a medical procedure on a patient 20 using the surgical system 10 described above is shown. The surgical system 10 including the surgical navigation system 100 and at least one of the surgical instrument assemblies 200, 300, 400 described above may be placed in the operating room surrounding the patient 20 and / or the surgical site where the medical procedure is to be performed.

[0119] While only the second surgical instrument assembly 300 is illustrated in FIG. 2, it will be understood that this is only an exemplary configuration of the surgical system 10, and that it is contemplated that any number of surgical instrument assemblies 200, 300, 400 may be positioned within the operating room. As described above, the second surgical instrument assembly 300 comprises the second surgical instrument 320 including the end-effector 340 and the instrument tracking device 330.

[0120] As shown in the example illustrated in FIG. 2, it is contemplated that the surgical system 10 may also comprise an imaging system 500, such as CT or MRI imaging device. Theimaging system 500 may comprise a scanner 510, a display unit 520, and an imaging controller 540. The scanner 510 may be utilized to take an image of the surgical site 30 on the patient 20, which may include the patient tissue of interest, and display it on the display unit 520. For example, the scanner may comprise a C-arm configured to be rotated about the patient 20 to produce a plurality of images of the surgical site 30. The imaging controller 540 may be structurally similar- to the navigation controller 140. For instance, the imaging controller 540 may operate under control of imaging software stored therein. The imaging software, upon execution by the imaging controller 540, may be configured to cause the imaging controller 540 to operate the scanner 510 to capture a plurality images and produce a 2-D and / or a 3-D image of the surgical site 30 based on the images. The imaging controller 540 may be configured to display the resulting image on the display unit 520.

[0121] The imaging controller 540 may also be in wired or wireless communication with the navigation controller 140 of the surgical navigation system 100, such as directly or via one or more networks. For example, the imaging controller 540 may be configured to provide preoperative and / or intra-operative image data, such as the resulting 2-D and / or 3-D images of the surgical site, to the navigation controller 140. The navigation controller 140 may then be configured to display the resulting 2D and / or 3D image on the display unit(s) 120. Responsive to receiving the image data, the navigation controller 140 may also be configured to apply segmentation to the images that identifies the boundaries of and / or annotates the features present in the images, as described above. Additionally or alternatively, the surgeon may interact with the patient images using the user input device(s) 130 to select and / or modify such boundaries and / or annotated features. The navigation controller 140 may then be configured to generate and store virtual models representative of the patient tissue of interest from the patient images based on the identified boundaries and / or annotations.

[0122] As described above, the surgical navigation system 100 may be configured to generate data indicating the poses of patient tissue of interest in a known coordinate system by tracking patient trackers 150 disposed relative to the patient tissue in the known coordinate system. Each patient tracker 150 may have been previously registered to a coordinate system of a virtual model of the patient tissue to which the tracker is attached so that, by determining a pose of a patient tracker in the known coordinate system, the surgical navigation system 100 may apply the registration to determine the pose of the virtual model in the known coordinatesystem. Correspondingly, the surgical navigation system 100 may determine the pose of any virtual objects associated with the patient tissue for guiding a surgical procedure in the known coordinate system. The surgical navigation system 100 may generate tracking data indicative of the poses of the surgical instruments 220, 320, 420 in the known coordinate system in a similar manner.

[0123] More particularly, during operation of the surgical navigation system 100, the tracking unit 110, such as at the direction of the navigation controller 140, may cooperate with one or more trackers in the surgical field, such as one or more patient trackers 150 and / or instrument trackers 230, 330, 430, to generate localizer data indicating the detected poses of each tracker, or more particularly the detected position of each marker of each tracker, in the known coordinate system, such as a coordinate system specific to the tracking unit 110. The tracking unit 110, which may also be referred to as a localizer, may communicate the localizer data to the navigation controller 140, which in turn may be configured to apply previously stored registration data to the localizer data to determine tracking data indicative of the poses of the surgical objects, or more particularly of the virtual models associated with the surgical objects (e.g., patient tissue of interest and / or surgical devices), in the known coordinate system. The navigation controller 140 may then communicate the tracking data to other components of the surgical system 10 and / or control other components of the surgical system 10, such as one of the instrument controllers 215, 315, 415, based on the tracking data.

[0124] Additionally or alternatively, the navigation controller 140 may be configured to determine the pose of a virtual model for patient tissue of interest in the known coordinate system based on the received patient images and / or realtime video data. For instance, like the video data, patient images generated by the imaging system 500 may have a known relationship with the known coordinate system such that components of the patient images, such as pixels or voxels, may be transformed to the known coordinate system based on the known relationship. In some implementations, the imaging system 500 may be positioned relative to the tracking unit 110 according to a predefined relationship that corresponds to the known relationship. Alternatively, the imaging system 500 may include an imaging tracker 542, similar to the video device tracker 472, the pose of which may be registered to a coordinate system of the patient images generated by the imaging system 500. Thus, responsive to determining a pose of the imaging tracker 542 relative to the known coordinate system such asusing the tracking unit 1 10 as described above, the navigation controller 140 may be configured to determine the known relationship between the coordinate system of the patient images and the known coordinate system based on a pose of the imaging tracker 542 relative to the known coordinate system and the registration between the imaging tracker 542 and the patient images.

[0125] The surgical navigation system 100 may have various functions and features as described in U.S. Patent No. 7,725,162 and U.S. Patent Publ. No. 2020 / 0100849, the contents of each which are hereby incorporated by reference herein in their entirety. For instance, while examples have been provided in which the surgical navigation system 100 is realized as an optical tracking system for detecting markers on the various trackers, other types of tracking systems may also or alternatively be employed. For instance, in some implementations, the surgical navigation system 100 may be realized as or additionally include an electromagnetic (EM) tracking system. The EM tracking system may be configured to cooperate with EM tracker(s) associated with one or more surgical objects of interest, such as one or more of the surgical instruments 220, 320, 420. An EM tracker associated with an object of interest may include a position sensor located at and / or embedded in the object of interest, such as in the handpiece 225, 325, 425 or end-effector 240, 340, 440 of a surgical instrument 220, 320, 420. The position sensor may include a coil that, when moved within an electromagnetic field, generates electrical current in the coil, which may then be communicated to the surgical navigation system 100. This phenomenon may enable the surgical navigation system 100 to determine the location of the coil within a three-dimensional space, such as that associated with the tracking unit 110, and such as relative to the patient tissue of interest.

[0126] By way of example only, the position sensor may be constructed and operable in accordance with at least some of the teachings of U.S. Patent No. 8,702,626, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent No. 8,320,711, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent No. 8,190,389, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent No. 8, 123,722, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent No. 7,720,521, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent Publ. No. 2014 / 0364725, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent Publ. No. 2014 / 0200444, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent Publ. No. 2012 / 0245456, theentire disclosure of which is hereby incorporated by reference herein; U.S. Patent Publ. No. 2011 / 0060214, the entire disclosure of which is hereby incorporated by reference herein; U.S. Patent Publ. No. 2008 / 0281156, the entire disclosure of which is hereby incorporated by reference herein; and / or U.S. Patent Publ. No. 2007 / 0208252, the entire disclosure of which is hereby incorporated by reference herein.

[0127] It is further contemplated that one or more of the surgical instruments 220, 320, 420 may form part of an end effector of a robotic manipulator. The robotic manipulator may include a base, several links extending from the base, and several active joints for moving the ultrasonic instrument with respect to the base. The links may form a serial arm structure, a parallel arm structure, or other suitable structure.

[0128] The robotic manipulator may be selectively operable in various modes, such as an autonomous mode, a hands-on mode, and a telemanipulation mode. In the autonomous mode, the surgical navigation system 100 may cause the robotic manipulator to maneuver the surgical instrument(s) 220, 320, 420 to treat the target site while avoiding other objects adjacent the target site, such as other medical tools and adjacent anatomical structures desired to be avoided, with little or no surgeon interaction, such as by tracking the pose of the surgical instrument(s) 220, 320, 420 relative to the virtual objects as described above.

[0129] In the hands-on mode, the surgeon may manually control the robotic manipulator by holding and maneuvering an aim of the robotic manipulator to maneuver the surgical instrument(s) 220, 320, 420 to treat the target site, or alternatively by holding and maneuvering the surgical instrument(s) 220, 320, 420 to treat the target site, while also receiving guidance from the robotic manipulator. In this case, at least one of the virtual objects generated by the navigation controller 140 may be a haptic object that generates a force (e.g., positive or negative) on the robotic manipulator, such as based on a tracked pose of the surgical instrument 220, 320, 420 relative to the haptic object, so as to guide the user in positioning the surgical instrument 220, 320, 420 to treat the target tissue. For instance, the robotic manipulator may be configured, such as at the direction of the surgical navigation system 100, to provide haptic feedback and / or constrain movement of the surgical instrument(s) 220, 320, 420 as the surgical instrument(s) 220, 320, 420 are moved by the surgeon relative to the target site, such as to promote full treatment of the target site and avoid adjacent objects. Non-limiting examples of haptic objects and corresponding operation of the robotic manipulator based on such objectsare described in U.S. Patent Nos. 9,775,681 and 10,398,449, the contents of each of which are hereby incorporated by reference herein in their entirety.

[0130] In the telemanipulation mode, the surgeon may control the robotic manipulator to maneuver the surgical instrument(s) 220, 320, 420 using a remote control device, such as a force feedback controller, while also receiving guidance from the remote control device, such as in the form of haptic feedback provided by the remote control device and / or constraints on movement of the remote control device based on haptic objects as described above.

[0131] FIG. 3 illustrates a portion of a patient’s spine 600 that may be operated on using the surgical system 10. In the illustrated example, the spine portion 600 includes adjacent vertebra 602 A, 602B and a spinal disc 604 disposed between the vertebra 602 A, 602B. The disc 604 includes target disc tissue 606 to be treated, and thus defining a target site of patient tissue for the surgical procedure. The spine portion 600 further includes patient tissue to be avoided while treating the target disc tissue 606, including exiting nerves 608A, 608B and the spinal cord 610. Prior to the surgical procedure, each of the vertebra 602A, 602B, disc 604, target site defined by the target disc tissue 606, nerves 608 A, 608B, and spinal cord 610 may be represented by a virtual model and associated with one or more virtual objects as described above.

[0132] FIG. 4 illustrates a screen 700 that may be displayed by the navigation controller 140 during the surgical procedure on the spine portion 600. The surgical plan for this procedure may indicate the use of the third surgical instrument 420 for treating the target disc tissue 606. As previously described, in addition to illustrating the pose of the third surgical instrument 420 relative to the patient tissue of interest, the screen 700 may display at least a portion of the surgical plan for treating the patient tissue of interest. In this case, the screen 700 illustrates a trajectory 702 for the third surgical instrument 420 during the procedure, which indicates a lateral approach that passes through the triangle, also referred to as Kambin’s triangle, formed by the exiting nerve 608A, a superior endplate 612 of the lower vertebra 602B, and an articular process 614 of the lower vertebra 602B (FIG. 3).

[0133] During movement of the third surgical instrument 420 to treat the target disc tissue 606, it may be generally desired to avoid contact by the third surgical instrument 420 with surgical objects adjacent the target disc tissue 606 and mentioned above. The screen 700 may thus illustrate one or more virtual boundaries and / or zones defined in association with these objectsto be avoided. More specifically, each virtual boundary may define a virtual zone associated with the object. For instance, as shown in the illustrated example, the spinal cord 610 has been associated with a virtual Boundary 1 and a virtual boundary 2, defining a virtual zone 1 and virtual zone 2 respectively. The virtual boundary 2 may correspond to the boundary of the spinal cord 610, and the virtual boundary 1 may correspond to a margin surrounding the spinal cord 610. These virtual boundaries and zones, which may represent no-fly zones for the third surgical instrument 420, may be configured such that, responsive to the tracking data generated by the navigation controller 140 indicating that the third surgical instrument 420 reaches the virtual boundary 1 or enters the virtual zone 1, the navigation controller 140 or third instrument controller 415 may be configured to trigger the third alert device 455 to issue an alert indicative of the same. Further, responsive to determining that the surgical instrument 420 reaches the virtual boundary 2 or enters the virtual zone 2, the navigation controller 140 or third instrument controller 415 may be configured to trigger the third surgical instrument 420 to cease actuation, such as by communicating a corresponding signal to the actuator 445.

[0134] Similarly, the nerve 608A may be associated with a virtual boundary 3 defining a virtual zone 3 representing a no-fly zone for the third surgical instrument 420. Responsive to the tracking data generated by the navigation controller 140 indicating that the third surgical instrument 420 reaches the virtual Boundary 3 or enters the virtual zone 3, the navigation controller 140 and / or third instrument controller 415 may be configured to trigger the third surgical instrument 420 to cease actuation and / or trigger the third alert device 455 to provide an alert indicative of the same. The vertebrae 602A, 602B may similar be associated with virtual boundaries and / or zones, designated as no fly zones by navigation controller 140.

[0135] The disc 604 may also be associated with multiple virtual boundaries and / or zones, such as boundaries and / or zones delineating between the disc tissue to be treated and tissue to be avoided. For instance, in the illustrated example, the disc 604 is associated with a virtual boundary 4 defining a virtual zone 4 associated with disc tissue to be avoided and thus defining a no-fly zone of the third surgical instrument 420, and with a virtual boundary 5 defining a virtual zone 5 associated with the target site and thus defining a fly zone for the third surgical instrument 420. Responsive to the tracking data generated by the navigation controller 140 indicating that the third surgical instrument 420, or more particularly the end-effector 440 of the third surgical instrument 420, reaches the virtual boundary 5 or enters the virtual zone 5,the navigation controller 1 0 and / or third instrument controller 415 may be configured to trigger actuation of the of the surgical instrument 420 for resecting tissue if not already actuated. Responsive to the tracking data generated by the navigation controller 140 indicating that the third surgical instrument 420 reaches the virtual boundary 4 or enters the virtual zone 4, the navigation controller 140 and / or third instrument controller 415 may be configured to trigger the third surgical instrument 420 to cease actuation and / or trigger the third alert device 455 to provide an alert indicative of the same.

[0136] The screen 700 may further include a completion percentage 704 indicative of the extent of the target site that has been resected, which may be updated based on the tracking data and / or video data as described in more detail below.

[0137] So as to provide the surgeon with improved views of the patient tissue of interest and enable tracking of patient tissue of interest that is difficult to track with an affixed tracker, the surgical plan may also indicate disposing a video device 470, in this case realized as an endoscope, adjacent the patient tissue of interest. As described above, the video device 470 may be configured to capture video data of the patient tissue of interest, which may be utilized by the navigation controller 140 to track surgical objects displayed therein and provide beneficial views to the surgeon during discrete stages of the procedure.

[0138] To this end, the screen 700 may also illustrate a virtual boundary 6 defining a virtual zone 6 at least partially surrounding the patient tissue of interest. During the procedure, the tracking unit 110 may cooperate with the patient trackers 150, instrument tracker 430, imaging tracker 542, and / or video device tracker 472 so as to track the pose of the third surgical instrument 420 and / or video device 470 relative to the patient tissue of interest as described above. Responsive to the tracking data indicating that the end-effector 440 and / or video device 470 has reached or crossed the virtual boundary 6 or entered the virtual zone 6, the navigation controller 140 may be configured to switch from displaying a model view, such as shown in FIG. 4, to displaying an intraoperative video view, such as shown in FIG. 5.

[0139] The navigation controller 140 may also be configured to guide placement of the video device 470, such as based on the segmentation of the video data described above and / or tracking data generated from the above-described trackers, prior to switching the display. For instance, as the video device 470 is moved relative to the patient tissue of interest according to the surgical plan, the navigation controller 140 may be configured to update the poses of patienttissue of interest in the known coordinate system based on the video data generated by the video device 470, and provide updated guidance for placing the video device 470 in a target pose (e.g., within the virtual zone 6) relative to the patient tissue of interest based on the updated poses of the patient tissue and the tracking data indicative of the pose of the video device 470 in the known coordinate system. For instance, the navigation controller 140 may be configured to update the screen 700 to reflect the updated poses of the patient tissue and video device 470, and / or display one or more graphics (e.g., arrows) indicating a trajectory for the video device 470.

[0140] As another example, the navigation controller 140 may be configured to illustrate guidance for placing the video device 470 in a target pose relative to the patient tissue of interest based on the tracked pose of the video device 470 in the known coordinate system, and responsive to the tracking data indicating the pose of the video device 470 relative to the patient anatomy corresponds to a target pose for the video device 470, the navigation controller 140 may be configured to check whether the intraoperative patient video generated by the video device 470 that corresponds to when the tracking data indicates the target pose includes a target view of at least a portion of the patient tissue of interest, such as the target site and certain adjacent tissues to be avoided, such as using the segmentation routines described above. If not, then the navigation controller 140 may be configured trigger an alert, such as via the display unit(s) 120 and / or the third alert device 455, that instructs the surgeon to cease movement of the third surgical instrument 420 further towards the surgical site, and / or to recalibrate the registration between the video device tracker 472 and the coordinate system of the video device 470.

[0141] FIG. 5 illustrates a screen 800 that may be displayed by the navigation controller 140 and illustrates the intraoperative video view mentioned above. In addition to the video data captured by the video device 470 and the completion percentage 704, the screen 800 may also include boundaries and / or annotations of the various surgical objects in the view of the video device 470, such as identified using the segmentation routines described above. In the illustrated example, the screen 800 shows the boundaries and annotations of the nerve 608A, vertebra 602A, and vertebra 602B to be avoided, the target disc tissue 606, and the third surgical instrument 420. Because one or more of these objects, such as the nerve 608A and the disc 604, may not be suitable for connection to a tracker and may move and / or changeshape during the procedure, the intraoperative video view may be a more helpful view of the patient tissue of interest as compared to the model view illustrated in FIG. 4.

[0142] FIGS. 6 and 7 illustrate further screens 900, 1000 that may be displayed by the navigation controller 140 during the surgical procedure following treatment of a portion of the target disc tissue 606. Screen 900 includes an updated video view that shows a portion of the target disc tissue 606 having been removed. Correspondingly, a portion of the video data previously corresponding to the resected target disc tissue 606 has been changed from being within the identified boundary of the “target tissue disc L4-L5” object to being within the boundary of the “vertebra L5” object. The completion percentage 704 has also been updated to 50% in accordance with about half the target disc tissue 606 having been removed from the target site, such as indicated in the video data and / or by the tracking data generated by the navigation controller 140.

[0143] The navigation controller 140 may be configured to update the virtual boundaries and / or zones, such as no-fly zones and fly zones, based on the tracking data and / or video data. For instance, as the tracking data indicates movement of the end-effector 440 through various locations in the virtual zone 5 corresponding to the target disc tissue 606, and / or as segmentation of the video data shows locations associated with the target disc tissue 606 being changed to association with an object to be avoided such that the vertebra L5, the navigation controller 140 may be configured to adjust the virtual boundary 5 and / or the virtual zone 5 so as to remove such locations from the virtual zone 5 defined by the virtual boundary 5. In some implementations, the navigation controller 140 and / or third instrument controller 415 may also be configured to generate and / or update another virtual object, such as the no-fly zone represented by zone 4 (FIG. 4), to include such locations and thus represent areas where tissue has been resected. In some implementations, the

[0144] The screen 1000 includes an updated model view following resection of a portion of the target disc tissue 606, which may generally correspond to the intraoperative video view illustrated in FIG. 6. In some implementations, the navigation controller 104 may be configured to automatically switch from an intraoperative video view, such as that shown in FIG. 6, to the model view illustrated in FIG. 6 in response to the tracking data and / or video data indicating that the third surgical instrument 420 has left the virtual zone 6 over the virtual boundary 6. As shown in the illustrated example, the virtual boundary 5 and zone 5 has beenupdated to represent the amount of tissue resected from the target disc tissue 606, and the completion percentage 704 has been updated accordingly. Furthermore, the shape and position of the virtual boundary 3 and zone 3 have been updated relative to that illustrated in FIG. 4 based on video data captured by the video device 470 during the surgical procedure indicating movement of the 6O8A.

[0145] Although the above example is described in the context of a lateral approach discectomy procedure, it will be appreciated that the above features may be utilized with other spinal procedures. For instance, in preparation of a nerve decompression or facetectomy procedure, the navigation controller 140 may be configured to automatically segment and annotate bony surfaces of interest, such as from a patient image (e.g., CT scan). Such data may be used to generate virtual objects associated with a target volume of boney tissue to be removed. For instance, the navigation controller 140 may be configured to locate bony surfaces surrounding or adjacent a known nerve position that are within a threshold distance of each other, and develop one or more virtual objects representative of tissue to be removed so as to increase the distance between the boney surfaces. Thereafter, during the procedure, intraoperative video of the patient tissue of interest such as the boney surfaces and nerve may be automatically segmented and annotated. Such data may then be displayed and / or used to track the patient tissue of interest as described above, which in turn may be used to adjust the virtual objects and / or surgical plan intraoperatively as discussed above.

[0146] In any of the above-described views illustrating patient tissue to be resected, the navigation controller 140 may be configured to display a dynamic annotation over the tissue to be resected or a remaining portion of the tissue to be resected, thereby providing the surgeon with a visual cue of whether further tissue should be resected in addition to the completion percentage 704. For instance, the navigation controller 140 may be configured to determine a position of the above-described virtual boundaries and / or objects relative to the view in the known coordinate system using the above methodologies, and highlight areas adjacent to the boundaries that correspond to the tissue to be resected. In this way, as the virtual boundaries and / or objects are adjusted with resection of tissue such as disc material, the annotation may be adjusted correspondingly. The navigation controller 140 may also be configured to highlight a patient image registered in the known coordinate system in a similar manner.

[0147] As previously described, some surgical procedures may involve the use of an implant planned to be inserted or received in a space left by tissue being resected with one of the surgical instruments 220, 320, 420. During such a surgical procedure, the navigation controller 140, such as responsive to determining that the surgical instrument 220, 320, 420 reaches a given virtual boundary, may be configured to display the realtime video data captured by the video device 470 superimposed with a graphic representative of the implant and located at the planned pose of the of the implant with respect to the video, such as to guide the surgeon in resecting the tissue to make room for the implant. Specifically, the navigation controller 140 may be configured to track such tissue being resected in the known coordinate system as described above, and correspondingly determine a planned pose of the implant in the known coordinate system. Thereafter, based on the tracked pose of the video device 470 in the known coordinate system and a registration between the video device 470 and the realtime video, the navigation controller 140 may be configured to determine a planned posed of the implant relative to the realtime video, and place the graphical representation of the implant in the realtime video accordingly.

[0148] For instance, FIG. 8 illustrates a screen 1050 that may be displayed by the navigation controller 140 during a spinal fusion procedure. The screen 1050 illustrates a video view including the real time video captured by the video device 470 of target disc tissue to be resected from between the L4 and L5 vertebrae, and also illustrates a graphic corresponding to an implant to be disposed in place of the resected disc tissue. In this way, the surgeon may thus leverage the graphic to determine where to place the ultrasonic tool to resect the disc tissue.

[0149] In the above example, the navigation controller 140 may be configured to provide feedback to the surgeon as the surgeon navigated the triangle formed by the exiting nerve root 608A, a superior endplate 612 of the lower vertebra 602B, and an articular process 614 of the lower vertebra 602B. In alternative implementations, such as when the procedure involves removal of the articular process 614, the navigation controller 104 may be configured to provide similar guidance and / or control as described herein to assist the surgeon is navigating through a triangle defined by the exiting nerve root 608 A, the superior endplate 612, and the spinal cord 610, which may be may be associated with virtual boundaries designated as defining no-fly zones.

[0150] FIG. 9 illustrates a screen 1 100 that may be displayed by the navigation controller 140 during an alternative surgical procedure, namely, a procedure for removing a skull-base tumor 1102 from the patient. In some implementations, the procedure may be carried out on a patient who was administered a fluorescing agent, such a 5-ALA. In this case, the video device 470 may be configured to excite the patient tissue of interest with light of particular’ wavelengths that causes certain tissues, such as tumorous tissue or blood vessels, to emit fluorescence by virtue of the fluorescing agent. The video device 470 may then be configured to capture and / or filter video data containing such fluorescence. Although the illustrated example is shown as being focused on a pituitary tumor behind a sphenoid bone 1104 of the patient, it will be appreciated that the details provided below may be earned out on other types of skull-base tumors as well.

[0151] The screen 1100 illustrates model view of the patient tissue of interest. According to the surgical plan, the skull-base tumor 1102 may be accessed through the nose and sinus of the patient (i.e., trans-nasally) using one of the surgical instruments 220, 320, 420. To this end, a hole may be formed through the sphenoid bone 1104 of the patient to allow the surgical instrument to reach the skull-base tumor 1102. Subsequently, the surgical instrument 220, 320, 420 may be inserted trans-nasally through the sphenoid bone 1104 to resect the skull-base tumor 1102.

[0152] As shown in the illustrated example, the surgical plan for the provided surgical procedure may include inserting a video device 470, such as an endoscope or fluoroscope, trans-nasally, so that a field of view of the video device 470 includes at least a portion of the skull-base tumor 1102 of the patient and patient tissues of interest adjacent the skull-base tumor 1102. For instance, the video device 170 may be positioned adjacent, or more particularly proximal adjacent, the sphenoid bone 1104.

[0153] Similar to the previous example, the screen 1100 also illustrates virtual boundaries and virtual zones associated with the patient tissue of interest, which may be used to prompt triggering of different view modes. To this end, the navigation controller 140 may be configured to track the pose of the sphenoid bone 1104 and the skull-base tumor 1102 in the known coordinate system, such as based on a received patient image that is registered to the known coordinate system as described above. Various virtual boundaries may then beconfigured in the known coordinate system relative to the sphenoid bone 1104 and the skullbase tumor 1102 based on the tracked poses.

[0154] For instance, a virtual boundary 1 may be positioned proximal to the hole formed in the sphenoid bone 1104, which may likewise be represented by a virtual object in the known coordinate system as described above. A virtual boundary 2 may represent or be distal to the opening of the hole formed in the sphenoid bone 1104. A virtual boundary 3 may be positioned distal to the virtual boundary 2 so as to be disposed between the virtual boundary 2 and the skull-base tumor 1102, which may likewise be represented by a virtual object or boundary in the known coordinate system as described above. A virtual zone 1 may be defined between the virtual boundary 1 and the virtual boundary 2, and represent an area immediately proximate the sphenoid bone 1104. A virtual zone 2 may be defined between the virtual boundary 2 and the virtual boundary 3, and may represent an area immediately distal the opening of the hole formed in sphenoid bone 1104. A virtual zone 3 may be defined between the virtual boundary 3 and the skull-base tumor 1102, which may represent area immediately adjacent the skullbase tumor 1102.

[0155] Following the video device 470 being guided to a target pose relative to the patient tissue of interest as described above, responsive to the tracking data and / or video data indicating that the surgical instrument 220, 320, 420 has reached or crossed the virtual boundary 1 or entered the virtual zone 1, the navigation controller 140 may be configured to stop displaying the model view of FIG. 9, and instead trigger display of a combined intraoperative video and virtual model, such as combined view 1200 illustrated in FIG. 10. In this view mode, one or more virtual models of a target trajectory for the surgical instrument 220, 320, 420 and / or of one or more other surgical objects, such as patient tissue to be avoided and / or target tissue, may be overlaid on top of the video data so as to guide the surgeon to follow the trajectory with limited visibility in the video data. Specifically, the navigation controller 140 may be configured to display the virtual model(s) of the trajectory and / or surgical objects relative to video data based on the tracking data, which may indicate the poses of the surgical objects and / or surgical instrument 220, 320, 420 relative to the video data in a known coordinate system as described above.

[0156] In some implementations, at least one of the virtual objects superimposed on the realtime video data may represent a surgical object that is not visible in the realtime video data,such as because the surgical object is covered by other patient tissue or objects from the viewpoint of the video device 470. In this case, the superimposed virtual object may indicate to the surgeon what is behind the visible features of the realtime video so as to determine a desired trajectory of the surgical instrument 220, 320, 420.

[0157] For instance, in one example, when the vision device 470 is initially placed adjacent the hole formed in the sphenoid bone 1104, the vision device 470 may capture realtime video depicting the carotid artery and the sphenoid bone 1104. Based on the tracked pose of the sphenoid bone 1104 relative to the realtime video in the known coordinate system, such as determined with the localizer 110, the navigation controller 140 may locate and / or identify in the realtime video the hole formed in the sphenoid bone. Additionally or alternatively, the navigation controller 140 may locate and / or identify in the realtime video the carotid artery, such based on a previously determined positional relationship between the sphenoid bone 1104 and the carotid artery, which may have been determined from a preoperative image, and based on a tracked pose of the sphenoid bone 1104 in the known coordinate system relative to the realtime video. The navigation system 140 may then be configured to configure virtual boundaries associated with the carotid artery and / or the hole formed the sphenoid bone 1104 in the known coordinate system. The navigation system 140 may also be configured to generate and / or determine the pose(s) of virtual model(s) of the carotid artery and / or sphenoid hole relative to the realtime video in the known coordinate system, and superimpose such model(s) and / or virtual boundary(s) over the realtime video based on the determined pose(s).

[0158] For instance, in the example combined view 1200 illustrated in FIG. 10, the realtime video data 1202 is overlaid with a virtual model 1204 corresponding to the carotid artery and a virtual model 1206 corresponding to the hole formed in the sphenoid bone 1104, each of which may been previously tracked in the known coordinate system using patient images and / or the realtime video data as described above, but not be currently viewable in the realtime video data 1202. The combined view 1200 may also include a virtual model 1208 corresponding to a target trajectory for the surgical instrument 220, 320, 420 through the hole formed in the sphenoid bone 1104, which may be used as a visual guide when passing the surgical instrument 220, 320, 420 therethrough.

[0159] Referring again to FIG. 9, responsive to the tracking data indicating that the surgical instrument 220, 320, 420 has reached or crossed the virtual boundary 2 or entered the virtualzone 2, and possibly following the video device 470 being guided to a further target pose relative to the patient tissue of interest such as described above, the navigation controller 140 may be configured to trigger display of an annotated intraoperative video view, such as generally illustrated in FIG. 11. In this view mode, the video data captured by the video device 470 may be shown with the various surgical objects of interest within the video data being delineated and labeled.

[0160] Referring again to FIG. 9, responsive to the tracking data indicating that the surgical instrument 220, 320, 420 has reached or crossed the virtual boundary 3 or entered the virtual zone 3, and possibly following the video device 470 being guided to yet another target pose relative to the patient tissue of interest such as described above, the navigation controller 140 may be configured to trigger display of fluorescent video view, such as generally illustrated in FIG. 12. In this view mode, the video data captured by the video device 470 may be filtered to show tissues, such as a tumorous tissue or blood vessels, that emit fluorescence responsive to being incited with certain types of light, as described above. Similar to previous examples, the navigation controller 140 may be configured to segment the filtered video, such as based on the emitted fluorescence and using segmentation as described above, to identify boundaries and / or annotate the types of tissue shown in the view.

[0161] As described in the above examples, the navigation controller 140 may be configured to leverage intraoperative video data from the video device 470 to update virtual objects associated with patient tissue in real time, and provide improved guidance and operation of surgical devices as a result, in turn leading to improved surgical outcomes. In some implementations, the navigation controller 140 may be configured to leverage additional surgical data to determine a state of the surgical system 10, and control operation of the surgical system 10 based on the same. For instance and without limitation, in addition to intraoperative video and tracking data described above, the navigation controller 140 may be configured to receive instrument data relating to operation of the surgical instruments 220, 320, 420, such as motor speed, motor torque, motor temperature, motor current, and motor power consumption. Additionally or alternatively, the navigation controller 140 may be configured to receive physiological data relating to the patient and / or surgeon. Additionally or alternatively, the navigation controller 140 may be configured to receive historical data relating to the patient and / or surgeon.

[0162] The navigation controller 140 may be configured to correlate one or more of the received data types in time with the intraoperative video data and / or tracking data, and correspondingly the tracked poses of objects of interest indicated by the tracking data. The navigation controller 140 may then be configured to manage the virtual objects and / or regulate operation of the surgical instruments 220, 320, 420 based on the correlated data. As one nonlimiting example, in addition to updating the pose of virtual objects defining no fly zones for patient tissue to be avoided based on the tracked pose of such patient tissue indicated by the intraoperative video and tracking data, responsive to the correlated data indicating that the surgical instruments 220, 320, 320 are being operated relatively aggressively in terms of movement through the surgical site or actuation, the navigation controller 140 may also be configured to space the virtual objects defining the no fly zones further out from the patient tissue to be avoided. Various surgical data that may be received and / or correlated by the navigation controller 140, and exemplary sensors for obtaining such data, are disclosed in PCT Appl. No. PCT / US24 / 14416 filed on February 5, 2024 and titled “Systems and Methods for Evaluating and Aiding Surgical Performance”, which is hereby incorporated by reference herein its entirety.

[0163] In some instances, the navigation controller 140 and / or instrument controller 215, 315, 415 may be configured to adjust operation of the surgical instrument 220, 320, 420 based on surgical performance tracked using the methodologies described herein. For instance, based on the tracked pose of the surgical instrument 220, 320, 420 relative to a virtual boundary in the known coordinate system, the navigation controller 140 may be configured to track a number of contact instances between the surgical instrument 220, 320, 420 and an anatomical object to be avoided, such as a nerve or artery. Responsive to the tracked number reaching or exceeding a threshold, the navigation controller 140 may be configured to trigger an action such as deactivating the surgical instrument 220, 320, 420 and / or triggering an alert device 255, 355, 455.

[0164] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims.For instance, it will be appreciated that the above concepts may be utilized to provide improved guidance and surgical instrument control during intraventricular endoscopy procedures, open cranial microscopy procedures, and lateral skull base endoscopy procedures, supra orbital / orbital endoscopy procedures, endoscopic ear procedures, and general neuro procedures.

[0165] It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the examples is described above as having certain features, any one or more of those features described with respect to any example of the disclosure can be implemented in and / or combined with features of any of the other examples, even if that combination is not explicitly described. In other words, the described examples are not mutually exclusive, and permutations of one or more examples with one another remain within the scope of this disclosure.

[0166] In the examples and implementations described herein, the term “pose” may be replaced with “position and / or orientation”, and the term “poses” may be replaced with “positions and / or orientations”, without departing from the scope of this disclosure. Similarly, the terms “position” and “orientation” may be replaced with “pose” without departing from the scope of this disclosure.

[0167] Although the surgical system 10 is described as having multiple controllers for implementing various functions, features, and processes, it is within the scope of this disclosure for two or more of such controllers to be integrated in a single controller configured to provide the described functions, features, and processes of the separate controllers. For instance, at least one of the instrument controllers 215, 315, 415 and the navigation controller 140 may be integrated into a single controller configured to perform the functions, features, and processes of each controller. It will also be appreciated that one or more of the functions, features, and processes described above as being provided by a given controller of the surgical system 10, such as one of the instrument controllers 215, 315, 415 may, in alternative implementations, be provided by another controller of the surgical system 10, such as the navigation controller 140.

[0168] The functions, features, and processes of a given controller described herein may also be distributed across a plurality of controllers. In this application, it will be understood thatone or more controllers being configured to implement a plurality of functions, features, and / or processes may include a plurality of controllers each configured to implement a different one of the functions, features, and / or processes, and that one or more controllers being configured to implement a given function, feature, or process may include a plurality of controllers that are configured to cooperate, such as via communications therebetween, to implement the given function, feature, or process.

[0169] Spatial and functional relationships between elements (for example, between controllers, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.

[0170] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The term subset does not necessarily require a proper subset. In other words, a first subset of a first set may be coextensive with (equal to) the first set.

[0171] In the FIGS., the direction of an arrow, as indicated by the arrowhead, generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information, but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgements of, the information to element A.

[0172] In this application, including the definitions below, the term “controller” or “module” may be replaced with the term “circuit.” The term “controller” may refer to, be part of, or include: at least one Application Specific Integrated Circuit (ASIC); at least one programmable system on a chip (PSoC); at least one digital, analog, or mixed analog / digital discrete circuit;at least one digital, analog, or mixed analog / digital integrated circuit; at least one combinational logic circuit; at least one field programmable gate array (FPGA); at least one processor (shared, dedicated, or group) that executes code; at least one memory circuit (shared, dedicated, or group) that stores code executed by the at least one processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

[0173] The controller may include one or more interface circuits with one or more transceivers, such as radio frequency (RF) or optical based transceivers (e.g., infrared (IR)). In some examples, the interface circuit(s) may implement wired or wireless interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2016 (also known as the WIFI wireless networking standard) and IEEE Standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of a WPAN are the BLUETOOTH wireless networking standard from the Bluetooth Special Interest Group and IEEE Standard 802.15.4.

[0174] The controller may communicate with other controllers using the interface circuit(s). Although the controller may be depicted in the present disclosure as logically communicating directly with other controllers, in various implementations the controller may actually communicate via a communications system. The communications system may include physical and / or virtual networking equipment such as hubs, switches, routers, gateways and transceivers. In some implementations, the communications system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communications system may include multiple LANs connected to each other over the Internet or point-to-point leased lines using technologies including Multiprotocol Label Switching (MPLS) and virtual private networks (VPNs).

[0175] In various implementations, the functionality of the controller may be distributed among multiple controllers that are connected via the communications system. For example, multiple controllers may implement the same functionality distributed by a load balancing system. In a further example, the functionality of the controller may be split between a server (also known as remote, or cloud) controller and a client (or, user) controller.

[0176] Some or all hardware features of a controller may be defined using a language for hardware description, such as IEEE Standard 1364-2005 (commonly called “Verilog”) and IEEE Standard 1076-2008 (commonly called “VHDL”). The hardware description language may be used to manufacture and / or program a hardware circuit. In some implementations, some or all features of a controller may be defined by a language, such as IEEE 1666-2005 (commonly called “SystemC”), that encompasses both code, as described below, and hardware description.

[0177] The term code, as used above, may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple controllers. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more controllers. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple controllers. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more controllers.

[0178] The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Nonlimiting examples of a non-transitory computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0179] The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purposecomputer to execute one or more particular functions embodied in computer programs. The functional blocks and flowchart elements described above may serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

[0180] The computer programs may include processor-executable instructions that are stored on at least one non-transitory computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input / output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0181] The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C#, Objective C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

What is claimed is:

1. A navigation system comprising: a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgical objects including a first anatomical object, a surgical instrument for treating patient tissue at a surgical site, and a video device for capturing realtime video of the surgical site; and at least one controller coupled to the localizer and the video device and configured to: track a pose of each of the surgical objects in the known coordinate system based on the localizer data; receive the realtime video captured by the video device and that corresponds to the tracked pose of the video device in the known coordinate system; define a virtual boundary associated with a second anatomical object in the known coordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object, wherein the second anatomical object is a soft tissue object; and control operation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system.

2. The navigation system of claim 1, wherein the at least one controller is configured to manipulate the virtual boundary over time based on the received realtime video and the tracked pose of the video device to provide a dynamic virtual boundary associated with the soft tissue object.

3. The navigation system of claim 1, wherein the at least one controller is configured to generate the virtual boundary in the known coordinate system based on a first frame of the received realtime video and subsequently manipulate the virtual boundary in the known coordinate system based on a second frame of the received realtime video following the first frame.

4. The navigation system of claim 1, wherein the at least one controller is configured to define the virtual boundary associated with the second anatomical object in the knowncoordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object by being configured to: identify a region in a frame of the received realtime video that likely corresponds to the second anatomical object based on the tracked pose of the video device and the tracked pose of the first anatomical object in the known coordinate system; and define the virtual boundary in the known coordinate system based on the identified region and the tracked pose of the video device.

5. The navigation system of claim 4, wherein the at least one controller is configured to identify the region in the frame of the received realtime video that likely corresponds to the second anatomical object based on the tracked pose of the video device, the tracked pose of the first anatomical object, and anatomical data indicating a positional relationship between the first anatomical object and the second anatomical object.

6. The navigation system of claim 5, wherein the anatomical data comprises a patient image of the first and second anatomical objects, wherein optionally, the patient image is a preoperative patient image.

7. The navigation system of claim 1, wherein the at least one controller is configured to define the virtual boundary associated with the second anatomical object in the known coordinate system based on the received realtime video, the tracked pose of the video device, and the tracked pose of the first anatomical object by being configured to: determine a positional relationship between the first anatomical object and the received realtime video based on the tracked pose of the first anatomical object and of the video device in the known coordinate system; and input the received realtime video and the positional relationship into a machine learning algorithm configured to output segmentation data indicative of a location of the second anatomical object within a frame of the received realtime video.

8. The navigation system of claim 7, wherein the output segmentation data indicates an identity of the second anatomical object, and the at least one controller is configured to controloperation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system and the identity of the second anatomical object indicated by the output segmentation data.

9. The navigation system of claim 1, wherein the first anatomical object is a bone.

10. The navigation system of claim 9, wherein the at least one controller is configured to: receive a patient image illustrating the bone impinging on the second anatomical object; display at least a portion the patient image with an annotation indicative of a portion of the bone to resect to relieve the impingement of the second anatomical object; and adjust the annotation of the patient image as the portion of the bone is resected to indicate a remaining portion of the bone to be resected based on the received realtime video and the tracked pose of the video device the known coordinate system, and optionally based on the tracked pose of the surgical instrument in the known coordinate system.

11. The navigation system of claim 9, wherein the at least one controller is configured to display the realtime video superimposed with an implant to be received in a space of a portion of the bone to be resected as the portion of the bone is resected based on the tracked pose of the bone and of the video device in the known coordinate system, and optionally based on the tracked pose of the surgical instrument in the known coordinate system.

12. The navigation system of claim 1 , wherein the second anatomical object is an object to be avoided.

13. The navigation system of claim 12, wherein the second anatomical object is a nerve or artery.

14. The navigation system of claim 12, wherein the at least one controller is configured to control the operation of the surgical instrument based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system by being configuredto, responsive to determining that the surgical instrument has reached or crossed the virtual boundary, cease or reduce operation of an actuator of the surgical instrument.

15. The navigation system of claim 1, wherein the second anatomical object is a spinal disc with disc tissue to be resected.

16. The navigation system of claim 15, wherein the at least one controller is configured to: receive a patient image of the spinal disc; adjust the virtual boundary associated with the spinal disc as the disc tissue is resected based on the received realtime video and the tracked pose of the video device in the known coordinate system, and optionally based on the tracked pose of the surgical instrument; and display the patient image with an annotation indicative of an amount of remaining disc material to be resected from the spinal disc based on the virtual boundary.

17. The navigation system of claim 15, wherein the at least one controller is configured to display the realtime video superimposed with an implant to be received in a space of the resected disc tissue as the disc tissue is resected based on the virtual boundary and the tracked pose of the video device in the known coordinate system.

18. The navigation system of claim 1, wherein the at least one controller is configured to: track a number of contact instances between the surgical instrument and the second anatomical object based on the tracked pose of the surgical instrument relative to the virtual boundary in the known coordinate system and / or based on the realtime video; and control operation of the surgical instrument based on the tracked number of contact instances.

19. A method for treating a skull base tumor of a patient, the method comprising: tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument in a known coordinate system;positioning the video device adjacent a sphenoid bone of the patient trans-nasally; receiving, by the navigation system, realtime video of an environment of the tumor captured by the video device and that corresponds to the tracked pose of the video device; forming a hole in the sphenoid bone; defining, by the navigation system, a first virtual boundary associated with the formed hole in the known coordinate system and a second virtual boundary associated with the tumor in the known coordinate system based on the received realtime video and the tracked pose of the video device; inserting the surgical instrument through the formed hole trans-nasally while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the first virtual boundary in the known coordinate system; and after inserting the surgical instrument through the formed hole, resecting the tumor with the surgical instrument while receiving feedback from the navigation system indicative of the tracked pose of the surgical instrument relative to the second virtual boundary in the known coordinate system.

20. The method of claim 19, comprising the surgical navigation system tracking the pose of the video device and of the surgical instrument in the known coordinate system based on localizer data generated by a localizer and indicative of the pose of the video device and of the surgical instrument in the known coordinate system.

21. The method of claim 19, comprising: determining, by the surgical navigation system and based on a received patient image, anatomical data indicative of a positional relationship between the sphenoid bone and a carotid artery of the patient; defining, by the surgical navigation system, a third virtual boundary associated with the carotid artery in the known coordinate system based on the positional relationship, the received realtime video, and the tracked pose of the video device; and inserting the surgical instrument through the formed hole trans-nasally while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the third virtual boundary in the known coordinate system.

22. The method of claim 21, comprising the navigation system: tracking a pose of the sphenoid bone in the known coordinate system; and defining the third virtual boundary associated with the carotid artery in the known coordinate system based on the tracked pose of the sphenoid bone, the positional relationship, the received realtime video, and the tracked pose of the video device.

23. The method of claim 19, comprising: adjusting, by the navigation system, the second virtual boundary as a portion of the tumor is resected based on the received realtime video and the tracked pose of the video device; and continuing resection of the tumor with the surgical instrument while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the second virtual boundary as adjusted in the known coordinate system.

24. A navigation system for tracking a surgical instrument for treatment of a skull-base tumor of a patient trans-nasally, the navigation system comprising: a video device configured to capture realtime video of an environment of the tumor, the video device including a light source for exciting a fluorescing agent in patient tissue; a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgical objects including the surgical instrument and the video device; and at least one controller coupled to the localizer and the video device and configured to: define first, second, and third virtual boundaries in the known coordinate system, the first virtual boundary associated with a position proximal to a hole formed in a sphenoid bone of the patient for accessing the tumor, the second virtual boundary associated with a position of the hole formed in the sphenoid bone, and the third virtual boundary associated with a position between the second virtual boundary and the tumor; responsive to the tracked pose of the surgical instrument indicating a distal end of the surgical instrument has reached or crossed the first virtual boundary, trigger display of a first view including the realtime video overlayed with a graphic indicating a target trajectory for inserting the surgical instrument through the hole formed in the sphenoid bone based on the tracked pose of the video device;responsive to the tracked pose of the surgical instrument indicating the distal end of the surgical instrument has passed through the first virtual boundary and reached or crossed the second virtual boundary, trigger display of a second view including the realtime video overlay ed with annotations of anatomical features of the realtime video; and responsive to the tracked pose of the surgical instrument indicating the distal end of the surgical instrument has passed through the first and second virtual boundaries and reached or crossed the third virtual boundary, trigger display of a third view including the realtime video filtered to highlight fluorescence emitted from the tumor in the realtime video.

25. A method for treating a spine of a patient, the method comprising: tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument relative to a vertebra in a known coordinate system; receiving, by the navigation system, realtime video of the spine of the patient captured by the video device and that corresponds to the tracked pose of the video device; tracking a pose of an exiting nerve root adjacent the vertebra in the known coordinate system based on the tracked pose of the video device, the received realtime video, and the tracked pose of the vertebra; inserting the surgical instrument through a triangle defined by the exiting nerve root and the vertebra while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the exiting nerve root and the vertebra in the known coordinate system; and treating tissue of the spine with the surgical instrument while extended through the triangle.

26. The method of claim 25, wherein the triangle is defined by the exiting nerve root, a superior endplate of the vertebra, and an articular process of the vertebra.

27. The method of claim 25, comprising the navigation system tracking a pose of a spinal cord of the patient in the known coordinate system based on the tracked pose of the video device, the received realtime video, and the tracked pose of the vertebra, wherein the triangle is defined by the exiting nerve root, a superior endplate of the vertebra, and the spinal cord.

28. A method for treating nerve compression between first and second vertebra of a patient, the method comprising: tracking, by a surgical navigation system, a pose of a video device and of a surgical instrument relative to the first and second vertebra in a known coordinate system; receiving, by the navigation system, realtime video of the first and second vertebra captured by the video device and that corresponds to the tracked pose of the video device; resecting a portion of at least one of the first and second vertebra with the surgical instrument while receiving feedback from the navigation system that is indicative of the tracked pose of the surgical instrument relative to the first and second vertebra; tracking, by the navigation system, a pose of the nerve between the first and second vertebra in the known coordinate system based on the realtime video and the tracked poses of the video device, the first and second vertebra, and the surgical instrument during the resection of a portion of the at least one of the first and second vertebra; and continuing resection of at least one the first and second vertebra while receiving feedback from the navigation system that is indicative of the pose of the nerve in the known coordinate system relative to the surgical instrument and the first and second vertebra.

29. A navigation system comprising: a video device for capturing realtime video of a surgical site; a localizer configured to generate data indicative of a pose of a surgical instrument in a known coordinate system, the surgical instrument for treating patient tissue at the surgical site; and at least one controller coupled to the localizer and the video device and configured to: define a virtual boundary associated with the patient tissue in the known coordinate system; track a pose of the surgical instrument relative to the virtual boundary in the known coordinate system based on the localizer data; and based on the tracked pose of the surgical instrument relative to the virtual boundary, trigger a display to switch between a first view illustrating a pose of the surgical instrument relative to a virtual model of the patient tissue and the realtime video of the surgical site.

30. A navigation system comprising: a localizer configured to generate data indicative of poses of surgical objects in a known coordinate system, the surgical objects including a bone of a patient in a surgical site, a surgical instrument for treating the bone, and a video device for capturing realtime video of the surgical site; and at least one controller coupled to the localizer and the video device and configured to: track a pose of the video device relative to the bone in the known coordinate system based on the localizer data; determine a tracking error based on the realtime video of the surgical site and the tracked pose of the video device relative to the bone; and trigger an action responsive to determining the tracking error.

31. The navigation system of claim 30, wherein the triggered action includes triggering an alert device.

32. The navigation system of claim 30, wherein the triggered action includes adjusting a virtual boundary associated with the bone in the known coordinate system.

33. The navigation system of claim 30, wherein the triggered action includes adjusting a registration between the bone and a tracker that is attached to the bone and is detectable by the localizer to generate the data indicative of the pose of the bone in the known coordinate system.

Images