Surgical system for computer-assisted navigation during surgical procedures

Abstract

A surgical system for computer-assisted navigation during a surgical procedure includes at least one processor that obtains a 3D radiological representation of a target anatomy of a patient and a fiducial set of a registration fixture. An attempt is made to register a position of the fiducial set in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system. Based on a determination that one fiducial of the fiducial set has an unsuccessful registration to a position of the 3D imaging space, the following operations are performed: displaying at least one view of the 3D radiological representation having a graphical overlay indicating that the fiducial has an unsuccessful registration to the 3D imaging space; receiving user-provided position information identifying where the fiducial is located in the 3D radiological representation; and registering the position of the fiducial to the 3D imaging space based on the user-provided position information.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 176,424, filed April 19, 2021, the disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to computer-aided navigation of devices and operations during surgical procedures. BACKGROUND

[0004] Surgical operating rooms can house a wide variety of medical devices, which can include computer-aided surgical navigation systems, medical imaging devices (e.g., computed tomography (CT) scanners, fluoroscopy imaging, etc.), surgical robots, etc.

[0005] Computer-aided surgical navigation systems can provide surgeons with computerized visualizations of the current pose of a surgical tool relative to medical images of a patient’s anatomy. Camera tracking systems for computer-aided surgical navigation typically use a set of cameras to track the pose of a reference array on a surgical tool relative to a patient reference array (also referred to as a “dynamic reference base” (DRB)) attached to a patient, which is positioned by a surgeon during a surgical procedure. The reference array enables the camera tracking system to determine the pose of the surgical tool relative to the anatomy imaged by the patient’s medical images and relative to the patient. Thus, the surgeon can use real-time visual feedback of the pose to navigate the surgical tool during a surgical procedure on the patient.

[0006] Many surgical workflows that use computer-aided surgical navigation systems require image scans, such as CT scans or magnetic resonance imaging scans, during a surgical procedure. Vertical scan slices (axial, sagittal, and coronal) can be used to enable the operator to visually observe the patient’s anatomy alongside the relative pose of the surgical instrument. Under the time constraints and other pressures of a surgical operating environment, surgeons and other surgical team members can have challenges in recalling, interpreting, and following surgical workflows. Improved surgical workflows and computer-implemented operations are needed to reduce the workload of the surgical team and ensure that the workflows and best practices are followed. SUMMARY

[0007] Some embodiments of the present disclosure relate to a surgical system for computer-aided navigation during a surgical procedure. The surgical system includes at least one processor operative to obtain a three-dimensional (3D) radiological representation of a target anatomy of a patient and a fiducial set of a registration fixture. The operations attempt to register a position of the fiducial set in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system. Based on determining that one of the fiducials has an unsuccessful registration to a position of the 3D imaging space, the operations display at least one view of the 3D radiological representation with a graphical overlay indicating that the fiducial has an unsuccessful registration to the 3D imaging space, receive user-provided position information identifying where the fiducial is located in the 3D radiological representation, and register a position of the fiducial to the 3D imaging space based on the user-provided position information.

[0008] In some further embodiments, the operations of attempting to register a position of the fiducial set in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system include obtaining an optical image of a reference array affixed to the patient from at least one camera of the camera tracking system. The reference array includes a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations attempt to register a position of a pattern of the set of optical markers to a position of a pattern of the set of fiducials in the 3D radiological representation and identify any of the optical markers of the reference array that have an unsuccessful registration to any of the fiducials of the registration fixture.

[0009] In some further embodiments, the operations include displaying a virtual implant device as an overlay on a view of the 3D radiological representation of the target anatomy. The operations display a graphical indication of a trajectory of the virtual implant device, the graphical indication representing an implant trajectory of the virtual implant device into the target anatomy. The operations update a pose of the graphical indication of the virtual implant device trajectory displayed in the view of the 3D radiological representation to track manipulation inputs received through a user interface of the surgical system. The operations store a user-specified pose in the pose of the graphical indication of the trajectory as a planned trajectory of the virtual implant device.

[0010] In some further embodiments, prior to obtaining the 3D radiological representation of the target anatomical structure of the patient and the fiducial set of the registration fixture, the operations include obtaining, from at least one camera of the camera tracking system, optical images of a reference array fixed to the patient and of the registration fixture attached to the radiological imaging device. The reference array includes a first set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space, and the registration fixture includes a second set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations obtain fluoroscopic images of the target anatomical structure of the patient and of the fiducial set of the registration fixture. The operations determine whether a first condition is satisfied based on a defined number of the optical markers of the first set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine whether a second condition is satisfied based on a defined number of the optical markers of the second set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine whether a third condition is satisfied based on a defined number of the fiducial set of the registration fixture being visible in the fluoroscopic images. When one of the first condition, the second condition, and the third condition is not satisfied, the operations display an indication of the one of the first condition, the second condition, and the third condition not being satisfied, and prevent the radiological imaging process from capturing the 3D radiological representation of the target anatomical structure of the patient and of the fiducial set of the registration fixture. Conversely, when each of the first condition, the second condition, and the third condition is satisfied, the operations enable the radiological imaging process to capture the 3D radiological representation of the target anatomical structure of the patient and of the fiducial set of the registration fixture.

[0011] Other surgical systems, methods, and computer program products according to embodiments of the inventive subject matter will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional surgical systems, methods, and computer program products be included within this description, be within the scope of the inventive subject matter, and be protected by the accompanying claims. Moreover, all embodiments disclosed herein are intended to be combinable with each other in any manner and / or combination. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of the inventive concepts. In the drawings:

[0013] FIG. 1 ExcelsiusHub components are shown in accordance with some embodiments;

[0014] FIG. 2 An exemplary user interface of the CONFIGURE tab during procedure setup is shown in accordance with some embodiments;

[0015] FIG. 3 An example user interface of the WORKFLOW tab during procedure setup is shown, in accordance with some embodiments;

[0016] FIG. 4 An example user interface of the VERIFY tab during procedure setup is shown, in accordance with some embodiments;

[0017] FIG. 5 Instrument verification using the verification recess is shown, in accordance with some embodiments;

[0018] FIG. 6 An example user interface during instrument verification is shown, in accordance with some embodiments;

[0019] FIG. 7 A process for fixing a dynamic reference base (DRB) to a patient attachment instrument is shown, in accordance with some embodiments;

[0020] FIG. 8 A process for tightening a DRB knob using a clamp driver is shown, in accordance with some embodiments;

[0021] FIG. 9 DRB and monitoring marker placement is shown, in accordance with some embodiments;

[0022] FIG. 10 A process for removing a Quattro spike with a removal tool is shown, in accordance with some embodiments;

[0023] FIG. 11 A process for fixing a pivot arm starburst on a registration jig is shown, in accordance with some embodiments;

[0024] FIG. 12 A process for fixing a registration jig to a patient attachment instrument post is shown, in accordance with some embodiments;

[0025] FIG. 13 A complete intraoperative CT registration assembly and release button is shown, in accordance with some embodiments;

[0026] FIG. 14 An example user interface of the IMAGE tab during an intraoperative CT imaging workflow is shown, in accordance with some embodiments;

[0027] FIG. 15 An example user interface of a manual registration screen during an intraoperative CT imaging workflow is shown, in accordance with some embodiments;

[0028] FIG. 16An exemplary user interface of the PLAN tab during an intraoperative CT imaging workflow is shown, in accordance with some embodiments;

[0029] FIG. 17 An exemplary user interface of the NAVIGATE tab during an intraoperative CT imaging workflow is shown, in accordance with some embodiments;

[0030] FIG. 18 An exemplary user interface of the IMAGE tab during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0031] FIG. 19 An exemplary user interface of the PLAN tab during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0032] FIG. 20 An exemplary user interface of the NAVIGATE tab during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0033] FIG. 21 A fluoroscopy registration jig attached to an image intensifier is shown, in accordance with some embodiments;

[0034] FIG. 22 An exemplary user interface of imaging acquisition during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0035] FIG. 23 An exemplary user interface after segment selection during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0036] FIG. 24 An exemplary user interface after successful registration during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0037] FIG. 25 An exemplary user interface of real-time instrument / implant trajectory and navigated instruments during a preoperative CT imaging workflow is shown, in accordance with some embodiments;

[0038] FIG. 26 An exemplary user interface of the IMAGE tab during a fluoroscopy imaging workflow is shown, in accordance with some embodiments;

[0039] FIG. 27 An exemplary user interface of image acquisition during a fluoroscopy imaging workflow is shown, in accordance with some embodiments;

[0040] FIG. 28An example user interface of the PLAN tab during a fluoroscopy imaging workflow is shown, in accordance with some embodiments;

[0041] FIG. 29 An example user interface of the NAVIGATE tab during a fluoroscopy imaging workflow is shown, in accordance with some embodiments;

[0042] FIG. 30 is a top view of personnel in an operating room optionally wearing an extended reality (XR) headset during a surgical procedure, the operating room including a camera tracking system for the navigated surgical procedure and a surgical robot system for robotically assisted surgery and configured in accordance with some embodiments;

[0043] FIG. 31 is a block diagram of a surgical system including a camera tracking system and a navigation system, and also optionally including a surgical robot and an XR headset each operable in accordance with some embodiments; and

[0044] FIGS. 32-34 is an overview of operations that can be performed by a surgical system in accordance with some embodiments. DETAILED DESCRIPTION

[0045] The following discussion is presented to enable a person skilled in the art to make and use embodiments of the disclosure. Various modifications to the illustrated embodiments will be readily apparent to those skilled in the art, and the principles described herein can be applied to other embodiments and applications without departing from the scope of the present disclosure. Thus, the embodiments are not intended to be limited to the

[0046] System Overview

[0047] The ExcelsiusHub by Globus Medical, Inc. (hereinafter “Globus Medical” or “Globus”) enables real-time surgical navigation and visualization using radiological patient images and guides compatible surgical instruments to precise locations and trajectories or provides visualization to aid in freehand navigation based on implant plans. The software reformats patient-specific CT images acquired prior to or during a surgical procedure or fluoroscopic images acquired during a surgical procedure and displays them on a screen based on a preferred viewing angle. Prior to surgery, a surgeon can create, store, access, and simulate instruments and other trajectories relative to patient anatomy captured in CT images. During a surgical procedure, the system identifies instruments in use and helps the user place implants with consistent accuracy through freehand navigation. The ExcelsiusHub tracks the location of surgical instruments in or on patient anatomy and continuously updates the instrument location on these images. The surgical procedure is performed by the surgeon using various Globus Medical specialty surgical instruments.

[0048] While various embodiments are described in the context of operating extensions to the Excelsius system product and other Globus Medical products, these and other embodiments are not limited thereto and can be used with any surgical navigation system.

[0049] Device Description

[0050] The ExcelsiusHub is a surgical navigation system that enables real-time surgical visualization using radiological patient images (e.g., pre-operative CT, intra-operative CT, and / or fluoroscopy), a patient dynamic reference base, and an advanced camera tracking system. The system maps based on a registration between a virtual patient (points on the patient image) and a physical patient (corresponding points on the patient anatomy). Once this registration is created, the software displays the relative position of tracked instruments on the patient image. This visualization can help guide the surgeon in planning and approach for implant placement and other surgical procedures. The patient scan in conjunction with the registration provides guidance assistance to the surgeon when using the system independently for freehand navigation or can provide robotic guidance and align the end effector when used with the ExcelsiusGPS robotic system. During a surgical procedure, the system tracks the location of compatible instruments (including end effectors on robotic arms) in or on patient anatomy using a camera tracking system and continuously updates the instrument location on the patient image with optical tracking. The system software can be responsible for navigation functions, data storage, network connectivity, user management, case management, and security functions. The ExcelsiusHub surgical instruments are typically manually operable, non-sterile, reusable instruments.

[0051] The ExcelsiusHub freehand instruments include registration instruments, patient reference instruments, and implant-specific surgical instruments. The system can also be used with the ExcelsiusGPS robotic system, which includes an active tracking end effector. The registration instruments incorporate an array of reflective markers for tracking patient anatomy as well as surgical instruments and implants. Components include a verification probe, a monitoring marker, a surgical instrument array, an intraoperative CT registration jig, a fluoroscopic registration jig, and a dynamic reference base (DRB). The patient reference instruments are clamped or driven into any appropriate rigid anatomical structure that is deemed safe and provides a rigid fixed point for the DRB. The surgical instruments are used to prepare an implant site or implant a device and include a awl, a drill, a driver, a tap, and a probe.

[0052] Indications for Use

[0053] The ExcelsiusHub can be used as an aid in precisely locating anatomical structures in open or percutaneous surgery and precisely positioning compatible surgical instruments or implants during surgical procedures. The ExcelsiusHub is indicated for any medical condition where stereotactic surgical procedures can be appropriate and where reference to rigid anatomical structures such as the spine or pelvis can be identified relative to CT, X-ray, and / or MRI based anatomical models. The ExcelsiusHub supports preoperative CT, intraoperative CT, and / or intraoperative fluoroscopic procedures.

[0054] Further System Overview

[0055] The ExcelsiusHub visualization system provides independent navigation and guidance for previously approved posterior approach fixation and interbody implant placement. The ExcelsiusHub can work with the ExcelsiusGPS robotic system to provide camera and tracking system functionality. The ExcelsiusHub can also provide a universal viewing station and registration operations for an intraoperative mobile CT system. Excelsius enabling technology can be supported and communicated through the ExcelsiusHub, which can provide the operating room with the required integrated components.

[0056] Hardware

[0057] According to some embodiments, the camera tracking system 200 of the ExcelsiusHub components can be as follows FIG. 1The illustrated is divided into three subassemblies: a camera arm assembly 100, a display assembly 120, and a housing base assembly 130. The camera arm assembly 100 includes a spaced apart stereo camera 102 and an articulated arm 104. The housing base assembly 130 includes a processing platform with at least one processor and memory, input and / or output user interfaces, and communication circuitry configured to communicate with other system components through wired (e.g., connector panel 107) and / or wireless connections. The camera 102, working with the processing platform, is configured to detect the pose of markers (e.g., reflective markers) on instruments, patients (e.g., dynamic reference base (DRB)), surgical robots, etc.

[0058] Instrumentation

[0059] Navigation Instrumentation

[0060] The Excelsius Hub works with all previously existing array instruments of Globus navigation. This includes drills, awls, probes, taps, and drivers for placing Globus screws, as well as dilators, disc preparation instruments (curettes, Cobb elevators, rasps, spatulas, etc.), trial molds, and inserters for navigating placement of Globus interbodies. Each instrument is identified by a unique array pattern recognized by the camera.

[0061] Patient Attachment and Registration Instrumentation

[0062] The Excelsius Hub can be used with existing patient fixation instruments and current DRBs.

[0063] Registration Fixture

[0064] The Excelsius Hub can be used with existing intraoperative registration jigs and fluoroscopic registration.

[0065] System Software

[0066] Software working with the Excelsius Hub can include existing spinal software available on the Excelsius GPS robotic system. This system software can be responsible for navigation functions, data storage, network connectivity, user management, case management, and security functions.

[0067] Applications

[0068] The Excelsius Hub system supports spinal surgery. The CONFIGURE tab displays the type of surgery. Spinal surgery steps are the same as in the Excelsius GPS robotic system, as the software on both hardware is the same.

[0069] Spinal Surgery

[0070] Various spinal surgical procedures supported by the ExcelsiusHub system are listed in the table below.

[0071] Supported Spinal Procedures

[0072] Surgery Patient Positioning Posterior Approach Cervical Spine Prone Position Posterior Approach Thoracic Spine Prone Position Anterolateral Approach Thoracic Spine Lateral Position Posterior Approach Lumbar Spine Prone Position Lateral Approach Lumbar Spine Lateral Position Lateral Approach Lumbar Interbody Fusion Lateral Position Transforaminal Approach Lumbar Interbody Fusion Prone Position Posterior Approach Lumbar Interbody Fusion Prone Position

[0073] Globus spinal implant systems compatible with the ExcelsiusHub system include those listed in the table below.

[0074] Compatible Spinal Implant Systems

[0075] CREO Stabilization System REVERE Stabilization System REVOLVE Stabilization System ELLIPSE Cervico-Thoraco-Lumbar System QUARTEX Cervico-Thoraco-Lumbar System SUSTAIN Spacer (Sloped, Small) (Posterior Interbody) ALTERA Expandable Spacer (Posterior Interbody) RISE Expandable Spacer (Posterior Interbody) CALIBER Expandable Spacer (Posterior Interbody) RISE-L Spacer (Lateral Interbody) CALIBER-L Expandable Spacer (Lateral Interbody) ELSA Integrated Expandable Spacer (Lateral Interbody)

[0076] Surgical Setup

[0077] CONFIGURE Tab

[0078] After a case is selected, the CONFIGURE tab is displayed on the monitor. FIG. 2 An example user interface is shown according to some embodiments. Using the CONFIGURE tab, a surgeon is selected from a list of registered surgeons, and an imaging modality and a procedure type are selected. The imaging modalities can include intraoperative CT, fluoroscopy, and preoperative CT. Clicking the right arrow advances to the next tab.

[0079] WORKFLOW Tab

[0080] Using the WORKFLOW tab, the desired surgical stages (e.g., intervertebral or screw) are selected in the desired order of operation (e.g., intervertebral first). For each stage, the imaging modality, intervertebral implant system, and desired intervertebral level on the anatomical model are selected, which can include cervical, thoracic, lumbar, and sacroiliac joints. Stages are added to the workflow by clicking the “Add Stage” button. Clicking “Verify Instruments” advances to the next tab. FIG. 3 An example user interface is shown according to some embodiments.

[0081] VERIFY Tab

[0082] The VERIFY tab displays navigation details, including the visibility, position, and verification status of the instruments selected on the WORKFLOW tab. Verification is used to indicate whether one or more instruments can be damaged, for example, during grasping. Verification should be performed on all instruments with arrays prior to use, as appropriate, using a verification adapter, instrument, implant, or dilator.

[0083] FIG. 4 An exemplary user interface is shown in accordance with some embodiments. The VERIFY tab displays a CAMERA VIEW and an INSTRUMENT STATUS listing a set of identified instruments.

[0084] The CAMERA VIEW can be updated in real-time from the perspective of the camera, for example, displaying colored circles indicating the position of the instruments. A solid colored circle can be displayed to indicate that the corresponding instrument is visible to the camera 102, while a hollow circle indicates that it is not visible to the camera 102. The size of the colored circle can be dynamically updated to change size to indicate the distance from the physical camera 102. In one embodiment, the size of the circle is adapted to grow larger as the instrument moves closer to the camera 102 and smaller as the instrument moves away from the camera 102. According to some embodiments, the ideal distance from the camera 102 can be about 2 meters or 6 feet.

[0085] The INSTRUMENT STATUS lists each instrument and its verification status, with each instrument identified with a corresponding colored circle. According to some embodiments, FIG. 4 Verification status symbols are shown.

[0086] Instrument Verification

[0087] Referring to FIG. 5 According to some embodiments, each instrument can be verified by placing the tip of the instrument into a verification point provided at a known location on another piece of equipment, for example, on a registered array of instruments or on a registered robotic arm of a surgical robotic system, or by placing a verification adapter to be verified into a verification dimple provided at a known location on the instrument. The dimple for verifying the instrument can be formed at a defined location on the navigated array of instruments. If the ExcelsiusGPS robotic system is used, the dimple can also be located on the top surface of the end effector or other known location that can be tracked by the camera 102. Next, ensure that both instruments are visible to the camera 102 and remain stable. A pop-up screen is then displayed on the VERIFY tab to indicate the operation of the verification process.

[0088] As FIG. 6As shown, according to some implementations, once verification is complete, the verification status is indicated on the screen, with the top error displayed in mm. If verification fails, a corresponding indicator 600 is displayed, for example, a circle with a red cross, and the verification operation can be repeated until successful verification is completed. Conversely, successful verification is indicated by another displayed indicator 601 (e.g., a green circle). When all instruments have been successfully verified, the user can click the right arrow to proceed to the next tab.

[0089] Patient Attachment Instrumentation

[0090] The patient attachment device is secured to a rigid bone anatomical structure near the surgical site. The user selects the desired device. According to some implementations, the patient attachment device should be placed no more than 185 mm from the center of the surgical site to maintain accuracy. Bone clips are clamped onto anatomical structures such as spinous processes, iliac crests, long bones, or any rigid bone structure that can be securely clamped. Quattro pins are inserted into the iliac crest or long bone. A rod attachment is secured to an existing spinal rod, for example, with a diameter of 4.5 mm to 6.35 mm. Exemplary recommended anatomical locations for various patient attachment devices are described in the table below, according to some implementations.

[0091] Patient Attachment Instrumentation - Suggested Anatomical Position

[0092]

[0093] Dynamic Reference Base Insertion

[0094] The user positions the compression clamp on the Dynamic Reference Base (DRB) above the patient attachment device and tightens the knob, as per some implementation schemes. FIG. 7 As shown in the diagram. If necessary, a clamp driver can be used to further tighten the DRB knob, as shown in some implementations. FIG. 8 As shown in the diagram. The user positions the reflective marker on the DRB in the direction of camera 102. Care should be taken with the initial placement of the patient reference device to avoid interfering with the surgical procedure. After navigation, remove the patient attachment device.

[0095] Monitoring Markers

[0096] Monitoring markers are inserted into rigid bone anatomy to enable the camera tracking system to use camera 102 to track the relative distance to the DRB, for example, to identify unwanted displacement in the DRB during surgery. FIG. 9 The placement of the DRB900, spaced apart from the monitoring marker 902, is shown according to some embodiments.

[0097] The monitoring marker can be inserted into the iliac crest or long bone or can be attached to the spinous process using a bone clamp. The user verifies that the clamp is rigidly fixed. In some embodiments, the monitoring marker should be placed no more than 185 mm from the dynamic reference base. Exemplary recommended anatomic locations for various monitoring markers are described in the following table, according to some embodiments.

[0098] Monitoring Markers - Suggested Anatomical Position

[0099]

[0100] The user attaches the disposable reflective marker to the marker post of the monitoring marker 902. The user attaches the crush cap designed to fit over the reflective marker ball 904 to the monitoring marker 902. The user inserts the monitoring marker 902 into a rigid bony anatomy near the surgical site and can tap gently with a mallet. The user removes the crush cap. The user removes the reflective marker 904 before using the removal tool. To use the bone clamp with the marker, the user attaches the disposable marker to the top end of the bone clamp. The user can use the clamp driver to secure the bone clamp. The user then verifies that the clamp is rigidly fixed.

[0101] Removal

[0102] The Quattro spike and monitoring marker are removed from the bony anatomy, either manually or using the removal tool. The bone clamp is removed by loosening the clamp with the clamp driver, attaching the removal tool, and lifting the bone clamp, as shown in FIG. 10 , according to some embodiments.

[0103] Intraoperative CT Imaging Workflow

[0104] Intraoperative CT Registration Fixture Setup

[0105] The pivot arm starburst 1100 is placed on the starburst post of the registration clamp post 1102 of the registration clamp 1110 and rotated 90° to secure, as shown in FIG. 11 , according to some embodiments. The registration clamp 1110 is positioned on the patient attachment instrument post 1202 and the compression clamp 1200 knob is tightened, as shown in FIG. 12 , according to some embodiments. If needed, the clamp driver can be used to further tighten the knob on the compression clamp 1200. FIG. 13A completed assembly is shown in accordance with some embodiments. To release the pivot arm, push the release button 1300 on the clamp, rotate the pivot arm 90° and pull up. The intraoperative CT registration clamp has six degrees of freedom and can be moved by adjusting one of the three joints to stabilize and hover over the surgical site. Only the metal fiducials embedded in the clamp need to be in the 3D scan (non-reflective markers). It can be operationally helpful or necessary for the intraoperative CT registration clamp not to move between image acquisition and performing anatomic landmark checks.

[0106] Load Images

[0107] FIG. 14 Selection of the IMAGE tab shown in FIG. 15A displays the steps required to load CT scan images in accordance with some embodiments. Images can be loaded from, for example, a USB drive, a hard drive, and / or a networked device. If the images are transmitted via Ethernet, the images can automatically appear on the hard drive when the transmission is complete.

[0108] To view the images on the USB drive, insert the USB drive into the USB port on the connector panel 107. To trigger the loading of the images, select the hard drive or USB drive icon and then select the desired patient images. The right arrow can be selected to load the patient images and advance to the next tab.

[0109] Manual Registration

[0110] When the images are loaded, an automatic registration can be performed. However, if the automatic registration operation fails or is otherwise unusable, a manual registration screen can be displayed to allow manual registration as shown in FIG. 15B in accordance with some embodiments. The registration screen left panel shows the image with the depiction of the full scan of the intraoperative CT. FIG. 15

[0111] In some embodiments of the registration clamp, the clamp and seven fiducials should be visible under the image. The operator needs to adjust the unregistered fiducials. On the screen, select the fiducial that has not been registered; this image then appears on the right. The user moves the colored circle 1500 on the screen until the user determines that it encircles the displayed fiducial marker. FIG. 15 The three small boxes 1510 shown in the right panel bottom of FIG. 15C show the x, y, and z directions of the fiducial, and all of these boxes should be adjusted until the blue circle is centered. In the exemplary embodiment of FIG. 15C, the registered fiducial 1500 is shown with a solidly filled center, another fiducial 1504 that is not visible to the system is shown with an unfilled center, and yet another fiducial 1502 that is partially visible to the system is shown with a partially filled center. FIG. 15 ​​

[0112] Ensure that all seven fiducials are correctly identified by reviewing the 3D model of the intraoperative registration jig. Fiducials can be deleted by selecting the delete icon on the right panel. Click the right arrow to confirm that the fiducials have been correctly identified before proceeding to the next step.

[0113] Now in the context of FIG. 32 corresponding operations are described that can be performed. FIG. 32 Operations that can be performed by a surgical system in accordance with some embodiments are shown. The surgical system includes at least one processor that can reside in components of a computer platform 3600, such as camera tracking system 10 and / or navigation system 3604. See FIG. 32 , the operations include obtaining 3200 a set of fiducials of a registration jig (e.g. FIG. 9 900 in FIG. 21 2100) and a three-dimensional (3D) radiological representation of a target anatomy of a patient (e.g. CT scan and / or fluoroscopy scan). The operations attempt 3202 to register the location of the set of fiducials in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system (e.g. FIG. 31 system 10 in ). The registration can include attempting to associate a set of poses (e.g. locations and orientations) of the fiducials in the 3D radiological representation to poses of optical markers detected in the 3D imaging space. Based on determining 3204 that one of the set of fiducials has an unsuccessfully registered location to the 3D imaging space, the following operations are performed: displaying 3206 at least one view of the 3D radiological representation with a graphical overlay indicating that the fiducial has an unsuccessfully registered location to the 3D imaging space; receiving 3208 user-provided location information identifying where the fiducial is located in the 3D radiological representation; and registering 3210 the location of the fiducial to the 3D imaging space based on the user-provided location information.

[0114] The operation of receiving 3208 user-provided location information identifying where the fiducial is located in the 3D radiological representation can include displaying three orthogonal views of the fiducial in the 3D radiological representation and / or the 3D imaging space, and displaying a graphical object superimposed on an initial location in the three orthogonal views. The operations move the location of the graphical object display in the three orthogonal views in response to user input through a user interface, and determine the location of the fiducial in the 3D radiological representation and / or the 3D imaging space based on the location of the graphical object display in the three orthogonal views. The operation of registering the location of the fiducial to the 3D imaging space is based on the determined location of the fiducial in the 3D radiological representation.

[0115] The operation of receiving 3208 that can display a graphical object superimposed on an initial location in the three orthogonal views can include determining the initial location to correspond to a predicted location of the fiducial based on the relative location of the fiducial defined by the registration jig template.

[0116] The operation of receiving 3208 may also include moving the position of the graphical object displayed in three orthogonal views to track directional input received through the user interface of the surgical system.

[0117] The operation 3202, attempting to register the position of a reference set in a 3D radiographic representation to a 3D imaging space tracked by a camera tracking system, may include acquiring an optical image of a reference array fixed to the patient from at least one camera of the camera tracking system. The reference array includes a set of optical markers detectable by at least one camera of the camera tracking system in the 3D imaging space. The reference array may also be connected to a registration fixture. The operation may then attempt to register the position of a pattern of optical markers to the position of a pattern of references in the 3D radiographic representation, and identify any optical marker in the reference array that was not successfully registered to any of the references in the registration fixture.

[0118] Landmark Check

[0119] After registration is complete, a landmark check can be performed to ensure successful registration calculation. Using a verification probe, touch the anatomical landmarks or reference points on the registration fixture to trigger verification displayed at the corresponding locations on the system monitor. This process can be repeated using, for example, two or three other landmarks.

[0120] The corresponding operations that can be performed by the surgical system may include: simultaneously moving the tool toward a reference that has been successfully registered to a 3D image space while the user moves the tool, tracking its position in video captured from a camera of a camera tracking system, and displaying an updated representation of the tool based on the tracked position in the 3D imaging space. The operation confirms the registration accuracy of the reference that has been successfully registered to the 3D image space by comparing a specified position of the tracked position of the tool with the position of the reference that has been successfully registered to the 3D image space.

[0121] Remove Registration Fixture

[0122] The intraoperative CT registration clamp can then be removed, while ensuring that the patient's attached instruments do not move.

[0123] PLAN Tab

[0124] The PLAN tab allows users to plan all screw insertion trajectories (e.g., 1600) and intervertebral placements (e.g., 1610) on the patient image, such as according to some implementation scheme. FIG. 16 As shown in the exemplary user interface. Based on the selection made in the PREPLAN tab, the implant is pre-loaded (e.g., with defined characteristics) in the system and displayed on the right side of the screen.

[0125] To add implants to the plan page, the user can drag the appropriate implant label onto the image at the desired slice. The active plan is shown in a defined color. The details of the active screw plan are shown in the lower right of the screen, including screw series, diameter, and length. Once the plan for all screws is complete, the right arrow can be selected to advance to the next tab.

[0126] Once the implant is placed on the image, the implant plan function is used to adjust the implant position by, for example, dragging the implant image on the touch screen. The user selects or otherwise defines the specific implant dimensions (width, length, height, lordosis) on the right side panel of the screen.

[0127] Now in the context of FIG. 33 corresponding operations are described that can be performed. FIG. 33 Operations that can be performed by a surgical system in accordance with some embodiments are shown. See FIG. 33 , the operations include displaying 3300 a virtual implant device as an overlay on a view of a 3D radiological representation of a target anatomical structure, and displaying 3302 a graphical indication of a virtual implant device trajectory representing an implant trajectory of the virtual implant device into the target anatomical structure. The operations update 3304 a pose of the graphical indication of the virtual implant device trajectory displayed in the view of the 3D radiological representation to track manipulation inputs received through a user interface of the surgical system. The operations store 3306 a user-specified pose in the pose of the graphical indication of the trajectory as a planned trajectory of the virtual implant device.

[0128] The operations can also include displaying a set of user-selectable implant devices for an implant plan, and generating a graphical representation of a virtual implant device based on a template of one of the set of user-selectable implant devices selected by a user through a user interface.

[0129] NAVIGATE Tab

[0130] The NAVIGATE tab allows the user to visualize the trajectory of the navigated instrument and the planned trajectory relative to the patient anatomy. FIG. 17 An example user interface is shown in accordance with some embodiments. The desired implant label can be selected on the right side of the screen.

[0131] The real-time instrument / implant trajectory 1700a, 1700b (actual plan) is displayed on the patient image, e.g., in the orthogonal image slice view, along with the planned screw, allowing the user to confirm the desired trajectory. If the real-time trajectory is not acceptable, the user can return to the PLAN tab to select another trajectory. If the real-time trajectory is acceptable, the user inserts the screw to the desired depth according to the current trajectory of the instrument.

[0132] The instrument being navigated is displayed as it advances to the planned location. While navigating the instrument, the user repeatedly looks at the monitor and the surgical site to ensure consistency between the haptic feedback and the navigational feedback.

[0133] Corresponding operations of the surgical system can include displaying the planned trajectory of the virtual implant device as an overlay on a view of the 3D radiological representation of the target anatomical structure. The operations obtain, from at least one camera of the camera tracking system, an optical image of a reference array fixed to a real implant device corresponding to the virtual implant device, the reference array including a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. While a user positions the real implant device relative to the target anatomical structure of the patient, the operations track a pose of the real implant device in the 3D imaging space based on a pose of the reference array in the optical image. In accordance with the tracked pose in the 3D imaging space, the operations display an updated graphical representation of the real implant device relative to the planned trajectory of the virtual implant device.

[0134] Preoperative CT Imaging Workflow

[0135] IMAGE Tab

[0136] Load Images

[0137] The IMAGE tab displays the steps required to load CT scan images. FIG. 18 An exemplary user interface is shown in accordance with some embodiments. Images can be loaded from, for example, a USB drive, a hard drive, or a networked device. If the images are transferred over Ethernet, the images can automatically appear on the hard drive when the transfer is complete.

[0138] To view images on a USB drive, plug the USB drive into the USB port on the connector panel. To load images, the user selects the hard drive or USB drive icon and selects the desired patient image. The right arrow can be selected to load the patient image and advance to the next tab.

[0139] PLAN Tab

[0140] The PLAN tab allows the user to plan all screw trajectories and interbody placement on the patient image. FIG. 19 An exemplary user interface is shown in accordance with some embodiments. Based on the selections made in the PREPLAN tab, implants are pre-loaded (e.g., characteristics are pre-defined in the system) on the right side of the screen.

[0141] To add implants to the plan page, the user can drag the appropriate implant tag onto the image at the desired slice. The active plan is shown in a defined color. The details of the active screw plan are shown in the lower right of the screen, including screw series, diameter, and length. Once the plan for all screws is complete, the right arrow can be selected to advance to the next tab.

[0142] Once the implant is placed on the image, the implant planning function is executed to adjust the implant position by, for example, dragging the implant image on the touchscreen. The specific implant dimensions (width, length, height, lordosis) are selected or defined on the right side panel of the screen.

[0143] NAVIGATE Tab

[0144] The NAVIGATE tab allows the user to visualize the navigated instruments and trajectory alignment relative to the patient anatomy based on the implant plan.

[0145] Registration Setup

[0146] According to some embodiments, another display screen, such as shown in FIG. 20 highlights the three steps to be completed before a fluoroscopic image can be taken to register the preoperative CT image. These steps can be inserting the DRB, positioning the C-arm of the C-arm imaging device, and registering the monitoring marker with the camera navigation system. Animations can be used to visually depict these steps.

[0147] As shown in FIG. 21 , the fluoroscopic registration clamp 2100 is attached to the image intensifier 2110 on the C-arm by rotating the clamp clockwise until it is tightened. Before orienting the clamp 2100, a new optical marker is installed on the clamp 2100 so that the optical marker faces the camera 102. The video capture cable is connected to the C-arm viewing station. The video capture USB cable is plugged into one of the USB ports on the ExcelsiusHub connector panel 107.

[0148] The user can ensure that the dynamic fiducial base is visible to the camera 102 after the C-arm is in the proper position.

[0149] The monitoring marker is registered with the camera tracking system by, for example, placing the instrument close to the reflective ball 904 on the monitoring marker 902 but not touching it. When the box is activated, the box is then shown in a defined color.

[0150] The right arrow can be selected to advance to the next tab.

[0151] Registration

[0152] Operate to acquire fluoroscopy images, one anterior-posterior (AP) and one lateral for each planned segment. The same image can be used for multiple segments.

[0153] In some embodiments, the operation verifies that the following three conditions are met before enabling image acquisition: 1) the DRB is visible to the camera 102; 2) the fluoroscopy registration jig is visible to the camera 102; and 3) a valid fluoroscopy image is taken.

[0154] FIG. 22 An example user interface is shown, according to some embodiments. When the image capture is ready, each of the three images on the left side of the screen changes to a defined color. When all three conditions are met, acquire the fluoroscopy image intra-operatively, and then select the “CAPTURE” button to transmit the image to the system. Once both images are successfully captured, the spinal segment on the right side of the screen displays a checkmark. The right arrow can be selected to advance to the next tab.

[0155] FIG. 23 An example user interface for selecting the desired segment is shown, according to some embodiments. The planned screw can be dragged and dropped onto the fluoroscopy image. The displayed graphical object (e.g., circles 2300a and / or 2300b) can be controlled via user input through the user interface to roughly position the screw within the vertebral body. The user input can correspond to a user touch selection or click on the desired location on the display and / or provide manipulation commands through a keyboard. Confirm that the screw rod is positioned correctly, the head and tail of the screw are in the desired orientation, and are correctly oriented left / right. The registration button can be selected when the confirmation is complete in order to proceed with registration.

[0156] When the registration is successful, a checkmark is displayed next to the active segment. FIG. 24 An example user interface is shown, according to some embodiments. When the registration is complete, the right arrow is selected.

[0157] The corresponding operations that can be performed are now described in the context of FIG. 34 . FIG. 34 Operations that can be performed by a surgical system, according to some embodiments, are shown. See FIG. 34 , before obtaining a 3D radiological representation of a target anatomy of a patient and a fiducial set of a registration jig, the operations include obtaining 3400, from at least one camera of a camera tracking system, a reference array (e.g., a DRB 900 in FIG. 9 ) that is fixed to the patient and a registration jig (e.g., a registration jig 1000 in FIG. 21The reference array includes a first set of optical markers detectable by at least one camera of the camera tracking system in the 3D imaging space, and the registration fixture includes a second set of optical markers detectable by at least one camera of the camera tracking system in the 3D imaging space. An operation obtains 3402 fluoroscopy images of the fiducial set of the registration fixture and the target anatomical structure of the patient. An operation determines 3404 whether a first condition is satisfied based on a defined number of optical markers in the first set being detected by at least one camera of the camera tracking system in the 3D imaging space. An operation determines 3406 whether a second condition is satisfied based on a defined number of optical markers in the second set being detected by at least one camera of the camera tracking system in the 3D imaging space. An operation determines 3408 whether a third condition is satisfied based on a defined number of the fiducial set of the registration fixture being visible in the fluoroscopy images. An operation determines 3410 whether any of the three conditions is not satisfied. When one of the first, second, and third conditions is not satisfied, the operation displays 3412 an indication of the one of the first, second, and third conditions not being satisfied, and prevents the radiological imaging procedure from capturing a 3D radiological representation of the fiducial set of the registration fixture and the target anatomical structure of the patient. Conversely, when each of the first, second, and third conditions is satisfied, the operation enables 3414 the radiological imaging procedure to capture a 3D radiological representation of the fiducial set of the registration fixture and the target anatomical structure of the patient.

[0158] Based on determining that each of the first, second, and third conditions is satisfied, the operation can trigger the capture of anteroposterior and lateral fluoroscopy images at a plurality of defined locations of the target anatomical structure of the patient. The operation can then compute a 3D radiological representation of the fiducial set of the registration fixture and the target anatomical structure of the patient based on the anteroposterior and lateral fluoroscopy images captured at the plurality of defined locations of the target anatomical structure of the patient.

[0159] Landmark Check

[0160] After registration is complete, a landmark check or verification can be performed to operationally ensure that the registration was successfully computed. Using a verification probe, anatomical landmarks are contacted and the corresponding locations are verified to be displayed on the system monitor. This process can be repeated using, for example, 2 to 3 landmarks.

[0161] Navigation

[0162] Reference FIG. 25The user selects the desired implant label on the right side of the screen. The real-time tool / implant trajectory (actual plan) is updated to be displayed on the patient image along with the planned screw, allowing the user to confirm the desired trajectory. If the real-time trajectory is unacceptable, the user can return to the PLAN tab to select an alternative trajectory. If the real-time trajectory is acceptable, the user inserts the screw to the desired depth based on the instrument's current trajectory.

[0163] The guided instrument is displayed as it advances to the planned position. While navigating the instrument, the user can repeatedly observe the monitor and surgical site to ensure consistency between tactile and navigation feedback.

[0164] The corresponding operation of the surgical system may include a stacked view displaying the planned trajectory of the virtual implantation device as a 3D radiographic representation of the target anatomy. The operation acquires optical images from at least one camera of a camera tracking system fixed to a reference array corresponding to the virtual implantation device, the reference array including a set of optical markers detectable by at least one camera of the camera tracking system in 3D imaging space. While the user positions the real implantation device relative to the patient's target anatomy, the operation tracks the pose of the real implantation device in 3D imaging space based on the pose of the reference array in the optical images. Based on the tracked pose in 3D imaging space, the operation displays an updated graphical representation of the planned trajectory of the real implantation device relative to the virtual implantation device.

[0165] Remove Registration Fixture

[0166] Carefully remove the fluoroscopic registration clamp. Ensure the patient-attached instruments do not move.

[0167] Fluoroscopy Imaging Workflow

[0168] IMAGE Tab

[0169] Registration Setup

[0170] FIG. 26 The screen shown can be basically corresponding to FIG. 20 The screen highlights the three steps to be completed before taking fluoroscopic images for patient registration, as described above. FIG. 20 As stated above.

[0171] For example, according to some implementation plans FIG. 21 As shown, the fluorescence fluoroscopy registration clamp can be attached to the image intensifier on the C-arm by rotating the clamp clockwise until it is tightened. Before orienting the clamp, a new optical marker is mounted on the clamp so that the optical marker faces the camera.

[0172] The user ensures that the dynamic fiducial base is still visible to the camera 102 after the C-arm is in the proper position. The scout marker can be registered by placing the instrument close to the reflective ball on the scout marker but not touching it. When the box is activated, the box turns a defined color. The right arrow can be selected to advance to the next tab.

[0173] Image Acquisition

[0174] Intraoperative fluoroscopic images, such as one AP and one lateral, are acquired.

[0175] In some embodiments, the operation verification satisfies the following three conditions before image acquisition is enabled: 1) the DRB is visible to the camera 102; 2) the fluoroscopic registration jig is visible to the camera 102; and 3) a valid fluoroscopic image is taken.

[0176] FIG. 27 An example user interface is shown, according to some embodiments. When the image capture is ready, each of the three images on the left side of the screen turns a defined color. When all three conditions are met, the intraoperative fluoroscopic images are acquired, and then the “CAPTURE” button is selected to transfer the images to the system. Once both images are successfully captured, the segment display on the right side of the screen shows a checkmark. Once the appropriate images are loaded and selected, the right arrow can be selected to continue.

[0177] Landmark Check

[0178] Once registration is complete, a landmark check or verification can be performed to operationally ensure that the registration was successfully computed. Using the navigated verification probe, contact anatomical landmarks and verify that the corresponding locations are displayed on the system monitor. This process can be repeated using, for example, 2 to 3 landmarks.

[0179] Remove Registration Fixture

[0180] The fluoroscopic registration jig is removed while ensuring that the patient does not move the instrument.

[0181] PLAN Tab

[0182] The PLAN tab allows the user to plan all screw trajectories and interbody placement on the patient image. FIG. 28 An example user interface is shown, according to some embodiments. Based on the selections made in the PREPLAN tab, the implants are preloaded on the right side of the screen.

[0183] To add implants to the plan page, the user can drag the appropriate implant label onto the image at the desired slice. The active plan is shown in a defined color. The details of the active screw plan are shown in the lower right of the screen, including screw series, diameter, and length. Once the plan for all screws is complete, the user can select (e.g., click) the right arrow to advance to the next tab.

[0184] Once the implant is placed on the image, the implant plan function is used to adjust the implant position by, for example, dragging the implant image on a touchscreen. The specific implant size (width, length, height, lordosis) is selected on the right side panel of the screen. Alternatively or additionally, the software of the planning system can perform automatic adjustment of the implant position such that the dropped implant satisfies one or more defined rules about the anatomy in the image.

[0185] NAVIGATE Tab

[0186] The NAVIGATE tab allows the user to visualize the trajectory of the navigated instrument and the planned trajectory relative to the patient anatomy. FIG. 29 An example user interface is shown in accordance with some embodiments.

[0187] The user selects the desired implant label on the right side of the screen. The real-time tool / implant trajectory (actual plan) is updated to be displayed on the patient image, for example, as graphical objects 2900a and 2900b along with the planned screws, allowing the user to confirm the desired trajectory. If the real-time trajectory is not acceptable, the user can return to the PLAN tab to select another trajectory. If the real-time trajectory is acceptable, the user inserts the screw to the desired depth according to the current trajectory of the instrument.

[0188] The navigated instrument is displayed as it advances to the planned position. While navigating the instrument, the monitor and the surgical site are repeatedly observed to ensure consistency between the haptic feedback and the navigation feedback.

[0189] Exemplary Operating Room Equipment Layout and Operation

[0190] FIG. 30 is a top view of personnel in an operating room optionally wearing extended reality (XR) headsets 3150a-3150b during a surgical procedure, the operating room including a camera tracking system 10 for the navigated surgical procedure and optionally including a robotic-assisted surgical robot system, and each of which is configured in accordance with some embodiments.

[0191] Referring to FIG. 30, the robotic system 3000 can include, for example, a surgical robot 3102, one or more robotic arms 104, for example, end effectors 3112 configured to attach to the jointed manipulator arms, and can include one or more tracking markers. The robotic system 3000 can also include one or more displays. The DRB 900 includes a plurality of tracking markers and is adapted to be fixed directly to the patient 3130 (e.g., to the bones of the patient 3130). Another reference array 3170 is attached to an instrument or formed on an instrument, etc. The camera tracking system 10 can have any suitable configuration to track the movement, orientation, and support of the cameras 102 in the desired locations, and can include a computer operable to track the pose of the reference arrays.

[0192] The tracking cameras 102 can include any suitable camera or cameras, such as one or more infrared cameras (e.g., dual focus or stereophotographic cameras) capable of identifying active and passive tracking markers, for example, as various reference arrays attached by the patient 3130 (e.g., the DRB 900), the end effectors 3112 (end effector reference arrays), instruments (e.g., the instrument array 3170), extended reality (XR) headsets 3150a-b worn by the surgeon 3120 and / or surgical assistant 3126, etc., in a given measurement volume observable from the perspective of the tracking cameras 102. The tracking cameras 102 can track markers attached on or formed by the robotic arms 140 manipulated by the user (surgeon) and / or the robotic system 3000. The tracking cameras 102 can scan a given measurement volume and detect light emitted or reflected from the reference arrays in order to identify and determine the pose of the reference arrays in three dimensions. For example, active reference arrays can include infrared-emitting markers (e.g., infrared light-emitting diodes (LEDs)) activated by an electrical signal, and passive reference arrays can include retro-reflective markers that reflect infrared light, for example, emitted by an illuminator or other suitable device on the tracking cameras 102 (e.g., they reflect incident IR radiation into the direction of the incident light).

[0193] The XR headsets 3150a and 3150b (also referred to as XR headsets 150) can each include a tracking camera that can track the pose of the reference arrays within their camera fields of view (FOVs) 3152 and 3154, respectively. Thus, as shown, the pose of the reference arrays attached to various objects can be tracked when in the FOVs 3152 and 3154 of the XR headsets 3150a and 3150b and / or the FOV 3600 of the tracking cameras 102. FIG. 30

[0194] ​An XR headset can be configured to augment a real-world scene with computer-generated XR imagery. An XR headset can be configured to provide an augmented reality (AR) viewing environment by displaying computer-generated AR imagery on a see-through display that allows light from the real-world scene to pass through for combined viewing by the user. Alternatively, an XR headset can be configured to provide a virtual reality (VR) viewing environment by preventing or substantially preventing the user from directly viewing light from the real-world scene while the user is viewing computer-generated AR imagery on a display. An XR headset can be configured to provide both AR and VR viewing environments. Thus, the term XR headset can be referred to as an AR headset or a VR headset.

[0195] FIG. 30 Possible configurations for placing the camera tracking system 10 and the surgical robotic system 3000 (when present) in an operating room environment are shown. Computer-assisted navigated surgery can be provided by the camera tracking system 10 controlling the XR headsets 3150a and 3150b, the display 110, and / or the display 3034 to display surgical navigation information. The inclusion of the surgical robotic system 3000 in the operating room is optional for computer-assisted navigated surgery.

[0196] The camera tracking system 10 can use tracking information from the camera tracking system 10 and other information, as well as other tracking information and information from one or more XR headsets 3150a and 3150b, such as inertial tracking information and optical tracking information, and (optionally) microphone information. The XR headsets 3150a and 3150b operate to display visual information and can play audio information to the wearer. This information can come from local sources (e.g., the surgical robot 3102 and / or other operating room equipment), remote sources (e.g., a patient medical image server), and / or other electronic devices. The XR headsets 3150a and 3150b can be used to track the pose of instruments, patient references, and / or robot end effectors in 6 degrees of freedom (6DOF), and can track the wearer’s hands. The XR headsets 3150a and 3150b can also operate to track hand poses and gestures to enable gesture-based interaction with “virtual” buttons and interfaces displayed through the XR headsets 3150a and 3150b, and can interpret pointing or gestures of the hand or fingers as operations that trigger various defined commands. In addition, the XR headsets 3150a and 3150b can have digital color camera sensors with 1 to 10x magnification that are referred to as digital magnifiers.

[0197] The "outside-in" machine vision navigation bar (tracking camera 102) can use a monochrome camera and / or a color camera to track the pose of the articulated manipulator. The machine vision navigation bar generally has a more stable view of the environment because it does not move as frequently or as quickly as the XR headsets 3150a and 3150b tend to move when positioned on the wearer's head. The patient reference array 900 is generally rigidly attached to the patient in a stable pitch and roll relative to gravity. This locally rigid patient reference 900 can be used as a common reference for the frames of reference of other tracking arrays, such as the reference array on the end effector 3112, the instrument reference array 3170, and the reference arrays on the XR headsets 3150a and 3150b.

[0198] In some embodiments, one or more of the XR headsets 3150a and 3150b are minimalist XR headsets that display local or remote information but include fewer sensors and are therefore lighter.

[0199] The robotic system 3000 can be positioned near or beside the patient 3130. The tracking camera 102 can be separate from the robotic system 3000 and positioned at the feet of the patient 3130. This location gives the tracking camera 102 a direct line of sight to the surgical field 208. It is contemplated that the robotic system 3000 and the tracking camera 102 will be located at any suitable location. In the illustrated configuration, the surgeon 3120 can be located across from the robot 3102, but still able to manipulate the end effector 3112 (and articulated manipulator) and the display 110. The surgical assistant 3126 can be located across from the surgeon 3120, still able to access the end effector 3112 and the display 110. If desired, the positions of the surgeon 3120 and the assistant 3126 can be interchanged. The traditional areas occupied by the anesthesiologist 3122 and the nurse or scrub tech 124 remain unobstructed by the positions of the robot 3102 and the camera 102. The anesthesiologist 3122 can operate an anesthesia apparatus including a display 3034.

[0200] The end effector 3112 can be releasably coupled to the robotic arm 104, and movement of the end effector 3112 can be controlled by at least one motor based on input from the camera tracking system 10. In some embodiments, the end effector 3112 can be connected to the articulated manipulator 104 and can include a guide tube 114 that is configured to receive and orient a surgical instrument, tool, or implant used to perform a surgical procedure on the patient 3130.

[0201] As used herein, the term "end-effector" is used interchangeably with the terms "end-effecter" and "effecter element." The term "instrument" is used in a non-limiting manner and is used interchangeably with "tool" and "implant" to generally refer to any type of device that may be used during surgery according to the embodiments disclosed herein. Exemplary instruments, tools, and implants include, but are not limited to, articulated arms, drills, screwdrivers, saws, expanders, retractors, probes, implant inserts, and implantation devices such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the end-effector 3112 may include any structure for achieving movement of the surgical instrument in a desired manner.

[0202] Surgical robot 3102 is operable to control the translation and orientation of end effector 3112. Robot 3102 is operable to move end effector 3112 along, for example, x-axis, y-axis, and z-axis under computer control. End effector 3112 can be configured to selectively rotate, under computer control, about one or more of the x-axis, y-axis, and z-axis, as well as the Z-frame axis (such that one or more of the Euler angles (e.g., roll, pitch, and / or yaw) associated with end effector 3112 can be selectively computer-controlled). In some embodiments, selective control of the translation and orientation of end effector 3112 allows for the execution of medical procedures with significantly improved accuracy compared to conventional robots utilizing, for example, a six-DOF robotic arm that includes only a rotational axis. For example, a surgical robot system 3000 can be used to manipulate a patient 3130, and a robotic arm 104 can be positioned above the body of the patient 3130, wherein the end effector 3112 is angled toward the body of the patient 3130 selectively relative to the z-axis.

[0203] In some exemplary embodiments, the XR headsets 3150a and 3150b can be controlled to dynamically display updated graphical indications of the posture of surgical instruments, so that the user can know the posture of the surgical instruments at all times during the surgical procedure.

[0204] As used herein, the term "attitude" refers to the position and / or rotation angle of an object (e.g., a dynamic reference array, an end effector, a surgical instrument, an anatomical structure, etc.) relative to another object and / or relative to a defined coordinate system. Therefore, attitude can be defined solely based on the multidimensional position of an object relative to another object and / or relative to a defined coordinate system, solely based on the multidimensional rotation angle of that object relative to another object and / or relative to a defined coordinate system, or based on a combination of multidimensional position and multidimensional rotation angle. Thus, the term "attitude" is used to refer to position, rotation angle, or a combination thereof.

[0205] In some further embodiments, the surgical robot 3102 can be configured to correct the path of the jointed manipulator arm moved by the surgeon under guidance of the robotic arm 104. In some example embodiments, the surgical robot 3102 can be configured to perform a stop, modification, and / or manual control of the movement of the end effector 3112. Thus, in use, in example embodiments, the surgeon or other user can operate the system 3000 and can select to stop, modify, or manually control the autonomous movement of the end effector 3112.

[0206] A reference array can be formed on or connected to the robotic arm 104, the end effector 3112, the patient 3130, and / or the surgical instrument. The camera tracking system 10 can track the pose of the reference array in, for example, 6 degrees of freedom (e.g., position along 3 orthogonal axes and rotation about those axes). In some embodiments, a reference array including a plurality of tracking markers can be disposed on (e.g., formed on or connected to) an outer surface of the robot 3102, such as on the robotic arm 104 and / or on the end effector 3112. A patient reference array 900 including one or more tracking markers can also be provided to the patient 3130 (e.g., formed on or connected to the patient). An instrument reference array 3170 including one or more tracking markers can be disposed on a surgical instrument (e.g., a screwdriver, a dilator, an implant inserter, etc.). The reference arrays enable the camera tracking system 10 to track each of the marked objects (e.g., the end effector 3112, the patient 3130, and the surgical instrument), and the tracked poses can be used to provide navigation guidance to the user to perform the surgical procedure and / or can be used to control movement of the surgical robot 3102 to guide the end effector 3112.

[0207] Exemplary Surgical System :

[0208] FIG. 31 A block diagram of a surgical system is shown that includes the camera tracking system 10 and the navigation system 3604, and also optionally includes the surgical robot 3102, the imaging device 3620, and the XR headset 3150 each operable in accordance with some embodiments.

[0209] The imaging device 3620 can include a C-arm imaging device, an O-arm imaging device, and / or a patient image database. The computer platform 3600 includes at least one processor, at least one memory storing program instructions executable by the at least one processor to perform operations. The computer platform 3600 can perform operations of the camera tracking system 10 and / or the navigation system 3604. The XR headset 3150 provides an improved human-machine interface for performing a navigated surgical procedure. The XR headset 3150 can be configured to provide functionality including, but not limited to, any one or more of displaying camera tracking information and surgical navigation information, identifying gesture-based commands, etc., e.g., via the computer platform 3600. The display device 3112 can include a video projector, a flat panel display, etc. The user can view the XR graphical objects as an overlay anchored to a particular real-world object viewed through a see-through display screen. The XR headset 3150 can additionally or alternatively be configured to display video streams from cameras mounted to one or more XR headsets 3150 and other cameras on the display device 3112, and / or medical images obtained from the imaging device 3620.

[0210] The electronic components of the XR headset 3150 can include a plurality of cameras 3122, a microphone 3121, a gesture sensor 3108, a pose sensor (e.g., an inertial measurement unit (IMU)) 3116, a display device 3112, and a wireless / wired communication interface 3124. The cameras 3122 of the XR headset 3150 can be visible light capture cameras, near-infrared capture cameras, or a combination of both.

[0211] The cameras 3122 can be configured to operate as the gesture sensor 3108 by tracking user gestures performed within the field of view of the cameras 3122. Alternatively, the gesture sensor 3108 can be a proximity sensor and / or a touch sensor that senses gestures performed in proximity to the gesture sensor 3108 and / or senses physical contact, e.g., tapping the sensor 3108 or its housing. The pose sensor 3116 (e.g., an IMU) can include a multi-axis accelerometer, a tilt sensor, and / or another sensor that can sense rotation and / or acceleration of the XR headset 3150 along one or more defined coordinate axes. Some or all of these electronic components can be housed in the headset housing, or can be housed in another housing configured to be worn elsewhere, such as on the hip or shoulder.

[0212] As described above, the surgical system includes a camera tracking system 10 that can be part of a computer platform 3600 that can also provide the functionality of a navigation system 3604 and / or an XR headset controller 3110. The surgical system can include imaging devices and / or a surgical robot 3102. The navigation system 3604 can be configured to provide visual navigation guidance to an operator to move and position an instrument and / or an end effector relative to patient anatomy (e.g., relative to the DRB 900) based on a surgical plan (e.g., from a surgical plan function) that defines where to perform a surgical procedure on the anatomy using the instrument and based on a pose of the anatomy determined by the camera tracking system 10. The navigation system 3604 can also be configured to generate navigation information based on a target pose of the instrument, a current pose of the patient anatomy, and a current pose of the end effector of the instrument and / or surgical robot 3102, where the navigation information is used to display information through the XR headset 3150 and / or another display to indicate where the end effector of the instrument and / or surgical robot 3102 should be moved to perform the surgical plan.

[0213] The electronic components of the XR headset 3150 can be operably connected to the electronic components of the computer platform 3600 through a wired / wireless interface 3124. The electronic components of the XR headset 3150 can be operably connected to or directly connected to various imaging devices 3620, such as C-arm imaging devices, I / O-arm imaging devices, patient image databases, and / or other medical devices through the computer platform 3600, for example, through a wired / wireless interface 3124.

[0214] The surgical system also includes at least one XR headset controller 3110 that can reside in the XR headset 3150, the computer platform 3600, and / or another system component connected via a wired cable and / or a wireless communication link. The software executed by the XR headset controller 3110 provides various functions. The XR headset controller 3110 is configured to receive information from the camera tracking system 10 and the navigation controller 3604 and generate XR objects based on the information to display on the display device 3112.

[0215] The XR headset controller 3110 can be configured to operatively process signaling from the camera 3122, microphone 3121, and / or pose sensor 3116, and connected to display XR images on the display device 3112 for viewing by the user. Thus, the XR headset controller 3110, illustrated as a circuit block within the XR headset 3150, should be understood to be operatively connected to other illustrated components of the XR headset 3150, but need not necessarily reside within a common housing or otherwise be capable of being carried by the user. For example, the XR headset controller 3110 can reside within the computer platform 3600, which in turn can reside within a housing of the surgical robot 3102, camera tracking system 10, etc.

[0216] Further Definitions and Embodiments

[0217] In the above description of various embodiments of the present disclosure, aspects of the present disclosure can be illustrated and described in any of a number of patentable classes or contexts including any new and useful processes, machines, manufactures, or compositions of matter, or any new and useful improvements thereof. Accordingly, the aspects of the present disclosure can be embodied in entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or combinations of software and hardware that can all generally be referred to as a "circuit," "module," "component," or "system." Furthermore, aspects of the present disclosure can take the form of a computer program product on one or more computer-readable media having computer readable program code embodied in the medium.

[0218] Any combination of one or more computer readable medium can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an appropriate optical fiber with a repeater, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0219] ​A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program code there within for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable signal medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0220] Computer program code for carrying out operations of aspects of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.; conventional procedural programming languages, such as the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP; dynamic programming languages, such as Python, Ruby and Groovy; or other programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider) or in a cloud computing environment or offered as a service such as software as a service (SaaS).

[0221] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0222] These computer program instructions can also be stored in a computer- readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable instruction execution devices, or other devices to cause a series of operational steps to be performed on the computer, other programmable devices or other devices to produce a computer implemented process such that the instructions executed on the computer or other programmable devices provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0223] It should be understood that the terms used herein are for the purpose of describing particular embodiments and are not intended to be limiting of the present disclosure. Unless defined otherwise, all terms used herein have the same meanings as those which are commonly understood by one of ordinary skill in the art in the field of the present disclosure. It should be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and relevant art and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0224] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0225] The terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the present disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Throughout the description, like reference numbers refer to like elements throughout the description.

[0226] Any devices or steps of the following claims that are incorporated by reference as corresponding structures, materials, acts, and equivalents are intended to include any disclosed structure, material, or acts for performing the function in combination with other claimed The description of the present disclosure is presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the present disclosure to the precise form described. Many modifications and variations are possible in light of the above teaching without departing from the scope and spirit of the present disclosure. The aspects of the present disclosure were chosen and described in order to best explain the principles of the present disclosure and its practical application to thereby enable others skilled in the art to best utilize the present disclosure in various aspects and with various modifications as are suited to the particular use contemplated.

Claims

1. A surgical system for computer-assisted navigation during a surgical procedure, the surgical system comprising at least one processor operative to: obtain a three-dimensional (3D) radiological representation of a target anatomy of a patient and a fiducial set of a registration fixture; attempt to register a position of the fiducial set in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system; and based on a determination that one of the fiducial set has an unsuccessful registration to a position in the 3D imaging space, display at least one view of the 3D radiological representation with a graphical overlay indicating that the fiducial has an unsuccessful registration to the 3D imaging space, receive user-provided position information identifying where in the 3D radiological representation the fiducial is located, and register the position of the fiducial to the 3D imaging space based on the user-provided position information, wherein: the operation of receiving user-provided position information identifying where in the 3D radiological representation the fiducial is located comprises: displaying three orthogonal views of the fiducial in the 3D radiological representation; displaying a graphical object superimposed on an initial position in the three orthogonal views; moving the graphical object display in the three orthogonal views in response to input by the user through a user interface; determining a position of the fiducial in the 3D radiological representation based on the position of the graphical object display in the three orthogonal views; and the operation of registering the position of the fiducial to the 3D imaging space is based on the determined position of the fiducial in the 3D radiological representation, wherein the operation of displaying the graphical object superimposed on the initial position in the three orthogonal views comprises: determining the initial position to correspond to a predicted position of the fiducial based on a relative position of the fiducial defined by a registration fixture template.

2. The surgical system of claim 1, wherein the at least one processor is further operative to move the position of the graphical object display in the three orthogonal views to track directional input received through the user interface of the surgical system.

3. The surgical system of claim 1, wherein the at least one processor is further operative to: track a position of a tool captured in video from a camera of the camera tracking system while a user moves the tool toward one of the fiducials that has a successful registration to the 3D imaging space; display an updated representation of the tool according to the tracked position in the 3D imaging space; and confirm a registration accuracy of the one of the fiducials that has a successful registration to the 3D imaging space based on a comparison of a specified position in the tracked position of the tool to the position of the one of the fiducials that is registered to the 3D imaging space.

4. The surgical system of claim 1, wherein the operation of attempting to register a position of the fiducial set in the 3D radiological representation to the 3D imaging space tracked by the camera tracking system comprises: obtaining, from at least one camera of the camera tracking system, optical images of a reference array affixed to the patient, the reference array comprising a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space; attempting to register a position of a pattern of the set of optical markers to a position of a pattern of the set of fiducials in the 3D radiological representation; and identifying any optical marker of the reference array that is unsuccessfully registered to any of the fiducials of the registration fixture.

5. The surgical system of claim 1, wherein the operations further comprise: displaying a virtual implant device as an overlay on the view of the 3D radiological representation of the target anatomical structure; displaying a graphical indication of a trajectory of the virtual implant device, the graphical indication representing an implant trajectory of the virtual implant device into the target anatomical structure; updating a pose of the graphical indication of the trajectory of the virtual implant device displayed in the view of the 3D radiological representation to track manipulation inputs received through a user interface of the surgical system; and storing a user-specified pose of the pose of the graphical indication of the trajectory as a planned trajectory of the virtual implant device.

6. The surgical system of claim 5, wherein the operations further comprise: displaying a set of user-selectable implant devices for implant planning; and generating a graphical representation of the virtual implant device based on a template of one implant device of the set of user-selectable implant devices selected by a user through the user interface.

7. The surgical system of claim 5, wherein the operations further comprise: displaying the planned trajectory of the virtual implant device as an overlay on the view of the 3D radiological representation of the target anatomical structure; obtaining, from at least one camera of the camera tracking system, optical images of a reference array affixed to a real implant device corresponding to the virtual implant device, the reference array comprising a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space; tracking a pose of the real implant device in the 3D imaging space based on a pose of the reference array in the optical images while a user positions the real implant device relative to the target anatomical structure of the patient; and displaying an updated graphical representation of the real implant device relative to the planned trajectory of the virtual implant device in accordance with the tracked pose in the 3D imaging space.

8. The surgical system of claim 1, wherein prior to obtaining the 3D radiological representation of the target anatomical structure of the patient and the set of fiducials of the registration fixture, the at least one processor is further operative to: obtaining, from at least one camera of the camera tracking system, optical images of a reference array fixed to the patient and of a registration fixture attached to a radiological imaging device, the reference array comprising a first set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space, and the registration fixture comprising a second set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space; obtaining a fluoroscopy image of the target anatomical structure of the patient and of the fiducial set of the registration fixture; determining whether a first condition is satisfied based on a defined number of the optical markers of the first set detected by the at least one camera of the camera tracking system in the 3D imaging space; determining whether a second condition is satisfied based on a defined number of the optical markers of the second set detected by the at least one camera of the camera tracking system in the 3D imaging space; determining whether a third condition is satisfied based on a defined number of the fiducial set of the registration fixture visible in the fluoroscopy image; when one of the first condition, the second condition and the third condition is not satisfied, displaying an indication of the one of the first condition, the second condition and the third condition not satisfied, and preventing a radiological imaging process from capturing the 3D radiological representation of the target anatomical structure of the patient and of the fiducial set of the registration fixture; and when each of the first condition, the second condition and the third condition is satisfied, enabling the radiological imaging process to capture the 3D radiological representation of the target anatomical structure of the patient and of the fiducial set of the registration fixture.

Images