Calibration and registration of pre- and intra-operative images

Abstract

The present disclosure generally relates to systems, devices, and methods for facilitating image-guided medical and / or diagnostic procedures (e.g., surgery or other interventions, among other contemplated medical applications), and the generation of current and / or accurate anatomical images to facilitate image-guided medical and / or diagnostic procedures (e.g., surgery or other interventions) and the calibration and alignment of imaging modalities (e.g., tomography, volumetric imaging, and / or fluoroscopy) used in such medical and / or diagnostic procedures.

Description

[Technical Field]

[0001] The present disclosure generally relates to systems, devices, and methods for facilitating image-guided medical and / or diagnostic procedures (e.g., surgery or other interventions, among other contemplated medical applications), and the generation of current and / or accurate anatomical images to facilitate image-guided medical and / or diagnostic procedures (e.g., surgery or other interventions) and the calibration and alignment of imaging modalities (e.g., tomography, volumetric imaging, and / or fluoroscopy) used in such medical and / or diagnostic procedures. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 438,258, entitled "GENERATION AND DISPLAY OF MEDICAL IMAGE DATA IN IMAGE-GUIDED SURGERY," filed January 11, 2023; U.S. Provisional Application No. 63 / 428,740, entitled "REGISTRATION OF TOMOGRAPHIC AND FLUOROSCOPIC IMAGES," filed November 30, 2022; U.S. Provisional Application No. 63 / 389,958, entitled "REGISTRATION OF TOMOGRAPHIC AND FLUOROSCOPIC IMAGES," filed July 18, 2022; and U.S. Provisional Application No. 63 / 389,955, entitled "FLUOROSCOPE CALIBRATION," filed July 18, 2022, the disclosures of each of which are incorporated herein by reference in their entirety for all purposes.

[0003] Image-guided surgery uses images of a tracked surgical tool or tools and the patient's anatomy to guide the surgery. In such procedures, proper and current imaging or visualization of the region of interest in the patient's anatomy is crucial.

[0004] Near-eye display devices and systems, such as head-mounted displays, including specialized eyewear (e.g., glasses), are used in augmented reality systems.

[0005] See-through displays (e.g., displays including at least a portion that is see-through) are used in augmented reality systems, for example, to perform image-guided and / or computer-assisted surgery. Typically, but not necessarily, such see-through displays are near-eye displays (e.g., integrated into a head-mounted device (HMD)). In this way, computer-generated images can be presented to a healthcare professional performing a procedure so that the images are aligned with the anatomical parts of the patient undergoing the procedure. Systems of this type for image-guided surgery are described, for example, in U.S. Pat. No. 9,928,629, U.S. Pat. No. 10,835,296, U.S. Pat. No. 10,939,977, PCT International Publication No. 2019 / 211741, U.S. Patent Application Publication No. 2020 / 0163723, and PCT International Publication No. 2022 / 053923. The disclosures of all these patents and publications are incorporated herein by reference. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 9,928,629 [Patent Document 2] U.S. Patent No. 10,835,296 [Patent Document 3] U.S. Patent No. 10,939,977 [Patent Document 4] PCT International Publication No. 2019 / 211741 [Patent Document 5] U.S. Patent Application Publication No. 2020 / 0163723 [Patent Document 6] PCT International Publication No. 2022 / 053923 Summary of the Invention [Problem to be solved by the invention]

[0007] According to some embodiments, systems, devices, and methods are described that increase the availability and opportunity for medical professionals to perform image-guided medical procedure navigation (e.g., medical procedures, diagnostics, and / or other interventional procedures). For example, the systems, devices, and methods disclosed herein can advantageously facilitate augmented reality-assisted medical procedure navigation based on preoperative three-dimensional (3D) imaging (e.g., preoperative tomographic imaging such as a computed tomography (CT) scan, or magnetic resonance imaging (MRI)) of at least a portion of a patient's anatomy (e.g., a portion of the spine or other bones, joints, or soft tissues) when certain types of 3D volumetric imaging equipment (e.g., expensive and bulky O-arm or other CT or MR imaging equipment) are not readily available for intraoperative imaging. Some types of intraoperative 3D imaging equipment (e.g., O-arm CT or MRI equipment) may be available only in a limited number of locations or in specific locations, such as special rooms or operating rooms in a hospital or diagnostic centers. The systems, devices, and methods disclosed herein can advantageously expand the availability of augmented reality-assisted or image-guided medical procedures because they involve the use of more readily available intraoperative 2D imaging equipment (such as C-arms and fluoroscopic imaging equipment) that can be used in outpatient treatment rooms, ambulatory surgery centers, or other locations. Medical professionals can navigate preoperative CT scans, for example, by using intraoperative X-rays (e.g., fluoroscopy) to register and match or calibrate the X-rays (e.g., fluoroscopy) with preoperative CT scans (e.g., CT / Fluoro calibration). Registration allows the system to know, for example, which 3D voxels in the CT scan correspond to 2D pixels in the X-ray or fluoroscopic image.

[0008] The systems, devices, and methods described herein can provide a level of accuracy for image-guided or augmented reality-assisted navigation during medical procedures similar to that involving more expensive and bulky intraoperative imaging equipment. The accuracy can result from registration and calibration involving preoperative images of the patient acquired before the procedure and intraoperative images acquired during the procedure. The calibration and registration can include tracking markers that can be imaged by a tracking system of a wearable device (e.g., a head-mounted display and tracking device such as glasses, goggles, a visor, or other head-mounted or non-head-mounted device) worn or that can be worn by the surgeon or other medical professional performing the medical procedure.

[0009] The systems, devices, and methods described herein can provide improved real-time image-guided displays that improve precise augmented reality-assisted navigation during medical procedures, reduce the need for more invasive surgical procedures, have a short learning curve, save time and resources, and reduce safety risks for surgeons and other personnel in the operating room because the systems may advantageously result in less radiation exposure.

[0010] Medical procedures may include spinal surgery procedures, other orthopedic procedures (e.g., procedures involving the hip, knee, ankle, elbow, shoulder, foot, arm, leg, etc.), cranial procedures, dental or oral surgery procedures, ear, nose, and throat (ENT) procedures, or other procedures. The systems and methods described herein may be used in connection with surgical procedures such as spinal surgery, joint surgery (e.g., shoulder, knee, hip, ankle, other joint), orthopedic surgery, cardiac surgery, bariatric surgery, facial bone surgery, dental surgery, cranial surgery, neurosurgery, etc. The surgical procedures may be performed during open or minimally invasive surgery (e.g., surgery in which small incisions are made that are self-sealing or sealed with surgical glue or small sutures). However, the described systems and methods may also be used in connection with other medical procedures (including therapeutic and diagnostic procedures) and other instruments and devices or other non-medical display environments. The methods described herein further include performing medical procedures (including, but not limited to, performing surgical interventions such as treatment of the spine, shoulder, hip, knee, ankle, other joints, jaw, skull, etc.).

[0011] According to some embodiments, the systems, devices, and methods described herein can facilitate acquiring and processing three-dimensional (3D) images (e.g., computed tomography (CT) scans, magnetic resonance imaging (MRI), ultrasound, etc.) of a patient's spine or other anatomical region prior to (e.g., hours, days, weeks, or months) a surgical or other medical or diagnostic procedure and using them in combination with two-dimensional (2D) fluoroscopic images of the patient taken during the procedure (e.g., minutes before or during the actual performance of the medical intervention). Software within the system can include algorithms that segment a preoperative 3D scan (e.g., a CT scan or MRI scan) of an anatomical region (e.g., at least a portion of the spine) into individual anatomical components (e.g., individual vertebrae, sacrum, and ilium of the spine and pelvic region). X-ray calibration can be performed by software algorithms to calculate X-ray detector parameters, distortions, and X-ray source and / or detector positions in a patient marker coordinate system. Registration can then be performed on the 2D intraoperative images to create a transformation for each individual anatomical component (e.g., each vertebra). Registration can involve finding the position and orientation of individual anatomical components (e.g., each vertebra and sacrum and ilium) within the patient marker coordinate system by comparing digitally reconstructed radiographs (DRRs) to the x-ray, where the DRRs are calculated or generated from the pre-operative 3D scan using parameters determined during x-ray calibration.Thus, the systems and methods may enable the adjustment of multiple coordinate systems via programmed online X-ray calibration and alignment algorithms to determine a 3D image relative to patient markers imaged by an imaging device of an augmented reality display device worn by a surgeon or other medical professional, from which a 3D image volume of the anatomical region may be created or generated, and the reconstructed 3D model of the anatomical region may be displayed on the augmented reality display device (e.g., a wearable display device) to enable an operator (e.g., a wearer) to precisely navigate one or more tools to perform a medical procedure (e.g., a therapeutic and / or diagnostic procedure).

[0012] According to some embodiments, the system can include an X-ray calibration fixture that can be coupled to a fluoroscope (e.g., on the detector side of the C-arm machine). The system can further include a head-mounted display and / or other non-head-mounted display that allows a surgeon or other professional to view an overlaid 3D volumetric image of at least a portion of the patient's anatomy (e.g., spine) relative to the patient's body, and can include a plurality of fiducial markers that assist in determining the position and orientation of the C-arm relative to the patient and facilitate the calibration and alignment process.

[0013] The embodiments of the present disclosure described below provide improved methods for image registration and display, as well as devices and software for implementing such methods. The embodiments of the present disclosure described below further provide improved methods for calibration in image-guided medical and / or diagnostic procedures (e.g., surgery or other interventions).

[0014] According to some embodiments, a method for facilitating augmented reality-assisted navigation based on preoperative 3D imaging and intraoperative 2D imaging of at least a portion of a patient's anatomy (e.g., spine or other bony portion) includes receiving a 3D tomographic image (e.g., a CT image) of at least a portion of the patient's anatomy (e.g., spine or other bony portion). The portion of the anatomy may include a portion of the spine including a plurality of vertebrae and / or other bony structures. The method further includes segmenting the 3D tomographic image into a plurality of 3D segments, each including a respective one of the plurality of vertebrae and / or other bony structures. The method also includes receiving two or more 2D fluoroscopic images of at least a portion of the patient's anatomy (e.g., spine). The method also includes registering each of the 3D segments with a respective vertebra or other bony structure in the two or more 2D fluoroscopic images. The method further includes generating a 3D image volume of at least a portion of the anatomy (e.g., spine) based on the registration. The method also includes presenting a 3D image volume of at least a portion of the anatomical structure (e.g., the spine) on an augmented reality display (e.g., a see-through stereoscopic display in a wearable device such as a head-mounted unit, glasses, or a visor).

[0015] The segmentation may be performed fully automatically by one or more processors (e.g., through the application of one or more trained neural networks). The processor may be located on a wearable device worn by a surgeon or other clinical professional and / or on a separate workstation or portable computer. At least a portion of the segmentation may be performed manually by a user, who may be a surgeon or another clinical professional. In some embodiments, the segmentation includes labeling a plurality of vertebrae and / or other bone portions (e.g., sacrum, ilium, or other bones).

[0016] In some embodiments, receiving the two or more 2D fluoroscopic images includes receiving a first 2D fluoroscopic image captured from a first angle or perspective of the C-arm fluoroscope (e.g., an anterior-posterior angle) and receiving a second 2D fluoroscopic image captured from a second angle or perspective of the C-arm fluoroscope (e.g., a lateral or oblique lateral angle) different from the first angle or perspective.

[0017] In some embodiments, presenting the 3D image volume includes overlaying an augmented reality image of the registered 3D segment onto the patient's back or other part of the patient that corresponds to the anatomical part.

[0018] In some embodiments, the method includes calibrating a frame of reference of the 2D fluoroscopic image to the patient's spine prior to registration, and overlaying the augmented reality image can include applying the calibrated frame of reference when overlaying the vertebrae or other bony portions in the 3D segment registered with the patient's spine or other anatomical portion.

[0019] In some embodiments, receiving two or more 2D fluoroscopic images includes receiving two 2D fluoroscopic images captured from different respective angles relative to the patient.

[0020] In some embodiments, registering each of the 3D segments comprises aligning the 3D segments with both of the two 2D fluoroscopic images.

[0021] In some embodiments, calibrating the frame of reference of the 2D fluoroscopic images comprises performing distortion correction of the two or more 2D fluoroscopic images.

[0022] In some implementations, performing distortion correction includes performing one or more spline interpolation techniques.

[0023] In some embodiments, registering each of the 3D segments includes adjusting the position and orientation of each 3D segment to match a respective vertebra in the two or more 2D fluoroscopic images.

[0024] In some embodiments, adjusting each position and orientation includes processing each 3D segment to generate a digitally reconstructed radiograph (DRR) and finding an optimal match between the DRR and each vertebra or other bony portion in one or more 2D fluoroscopic images.

[0025] In some implementations, aligning each of the 3D segments includes performing an initial guess estimation.

[0026] In some embodiments, the 3D image volume includes a reconstructed 3D model of multiple vertebrae or other bone portions.

[0027] According to some embodiments, a method for facilitating augmented reality-assisted navigation based on preoperative 3D imaging and intraoperative 2D imaging of at least a portion of a patient's anatomical structure (e.g., spine) is provided. The method includes receiving a preoperative 3D image (e.g., a CT image, an MR image, a 3D ultrasound image) of at least a portion of the patient's anatomical structure (e.g., spine) (e.g., including a plurality of bony structures). The method also includes segmenting the 3D image into a plurality of 3D segments, each including a respective one of the plurality of bony structures. The method further includes receiving two or more 2D intraoperative images of at least a portion of the patient's anatomical structure (e.g., spine). The method also includes registering each of the 3D segments with a respective vertebra or other bony portion in the two or more 2D intraoperative images. The method further includes generating a 3D image volume of the plurality of bony structures based on the registration.

[0028] According to some embodiments, a method for image processing includes receiving a 3D tomographic image of a patient including at least a portion of a spine composed of a plurality of vertebrae. The method further includes segmenting the 3D tomographic image into a plurality of 3D segments, each including a respective one of the vertebrae. The method also includes capturing two or more 2D fluoroscopic images of at least a portion of the patient's spine. The method further includes registering each of the 3D segments with a respective vertebra in the one or more 2D fluoroscopic images and presenting on a display an image of at least a portion of the spine including the registered 3D segments.

[0029] In some embodiments, presenting the image includes overlaying an augmented reality image of the registered 3D segments on the patient's back. In some embodiments, the method further includes calibrating a frame of reference of the 2D fluoroscopic image to the patient's spine, and overlaying the augmented reality image includes applying the calibrated frame of reference in overlaying the vertebrae with the registered 3D segments on the patient's spine.

[0030] In some embodiments, capturing one or more 2D fluoroscopic images includes capturing two 2D fluoroscopic images from different respective angles relative to the patient, and registering each of the 3D segments can include aligning the 3D segments with both of the 2D fluoroscopic images.

[0031] In some embodiments, registering each of the 3D segments includes adjusting the respective position and orientation of each 3D segment to match a respective vertebra in one or more 2D fluoroscopic images.

[0032] In some embodiments, adjusting each position and orientation includes processing each 3D segment to generate a digitally reconstructed radiograph (DRR) and finding an optimal match between the DRR and each vertebra in one or more 2D fluoroscopic images.

[0033] According to some embodiments, a computer-implemented method for image processing includes receiving a 3D CT image of a patient including at least a portion of the spine, the spine being composed of a plurality of vertebrae, a sacrum, and an ilium. The method also includes segmenting the 3D CT image into a plurality of 3D segments, each of the plurality of 3D segments including a respective one of the vertebrae, the sacrum, or the ilium using one or more neural networks. The method further includes capturing two or more 2D fluoroscopic images of at least a portion of the patient's spine. The method also includes registering each of the 3D segments with a respective vertebra, the ilium, or the sacrum in the two or more 2D fluoroscopic images. The method further includes generating a 3D image of at least a portion of the spine including the registered 3D segments for presentation on a display.

[0034] According to some embodiments, a method for image processing includes receiving a 3D medical image of a patient including at least a portion of a spine composed of a plurality of vertebrae. The method also includes segmenting the 3D medical image into a plurality of 3D segments, each 3D segment including a respective one of the plurality of vertebrae. The method further includes capturing a plurality of 2D medical images of at least a portion of the patient's spine including the plurality of vertebrae. The method also includes registering each of the plurality of 3D segments with a respective vertebra in the plurality of 2D medical images. The method further includes generating a 3D image of the spine including the registered 3D segments for output to a display.

[0035] In some embodiments, the segmenting step includes applying a neural network to the CT image to segment one or more regions of interest of at least a portion of the patient's spine, and correspondingly applying one or more additional neural networks to the one or more regions of interest of the CT image to segment at least each vertebra of the plurality of vertebrae.

[0036] In some embodiments, the method includes resampling the CT image to a first resolution that is coarser than the CT image resolution and resampling the CT image to a second resolution that is finer than the first resolution. A neural network may be applied to the CT image resampled to the first resolution, and one or more additional neural networks may be applied to one or more regions of interest within the CT image and correspondingly resampled to the second resolution.

[0037] According to some embodiments, an imaging system adapted to facilitate navigation guidance during spinal surgery or other medical intervention includes or consists essentially of an X-ray calibration fixture including an X-ray calibration pattern. The X-ray calibration fixture is configured to be mounted, attached, coupled, or secured to a fluoroscope (e.g., the detector portion of a C-arm fluoroscope) used in a hospital, patient care facility, ambulatory surgery center operating room, outpatient procedure room, or the like. The system also includes or consists essentially of a patient marker configured to be attached to a patient's body at or adjacent to a target area where the spinal surgery or other medical intervention will be performed. The system further includes or consists essentially of an alignment target (e.g., alignment marker) configured to be attached (e.g., rigidly) to the X-ray calibration fixture or patient marker. The system also includes or consists essentially of an alignment optical target (e.g., marker) having a predetermined spatial relationship to the alignment target. The system further includes, or consists essentially of, at least one processor configured to execute computer-readable program instructions stored in the memory that, when executed, cause the at least one processor to receive at least one x-ray image captured in an operating room by a fluoroscope of at least a portion of a patient's spine, the at least one x-ray image including an x-ray calibration pattern and an alignment target; receive optical images of both the patient markers and the alignment optical target; and process the x-ray image and the optical image to calibrate and align a frame of reference of the fluoroscope with at least a portion of the patient's spine.

[0038] According to some embodiments, the imaging system includes or essentially consists of an X-ray calibration fixture including an X-ray calibration pattern. The X-ray calibration fixture is configured to be attached, mounted, coupled, or otherwise secured to the fluoroscope. The system also includes or essentially consists of patient markers configured to be coupled, adhered, or secured to the patient's body. The system further includes or essentially consists of alignment targets configured to be attached to the X-ray calibration fixture or the patient markers. The system also includes or essentially consists of alignment optical targets having a predetermined spatial relationship to the alignment targets. The system further includes or essentially consists of a processor that, upon execution of program instructions stored on a computer-readable medium, receives a plurality of intraoperative medical images including i) X-ray images captured by the fluoroscope including the X-ray calibration pattern and the alignment target, and ii) optical images of both the patient markers and the alignment optical target, and calibrates and aligns a frame of reference of the fluoroscope with the patient's body based at least in part on the X-ray images and the optical images.

[0039] According to some embodiments, the imaging device includes or consists essentially of an X-ray calibration fixture including an X-ray calibration pattern configured to be attached, mounted, coupled, or fixed to a fluoroscope (e.g., a detector portion of a C-arm fluoroscope). The system includes or consists essentially of an alignment target (e.g., an alignment marker) configured to be attached or coupled (e.g., rigidly attached) to the X-ray calibration fixture or a patient marker. The system also includes or consists essentially of at least one processor, the processor configured, upon execution of stored program instructions, to receive one or more images captured in a procedure room including at least one X-ray image captured by a fluoroscope including the X-ray calibration pattern and the alignment target, and the spatial relationship of the alignment target to the patient marker, the patient marker configured to be fixed to the body of a patient undergoing surgery or other medical intervention in a procedure room (e.g., an operating room). The at least one processor is configured to process the X-ray image and the optical image to calibrate and align a frame of reference of the fluoroscope with the patient's body.

[0040] In some embodiments, the device further includes an augmented reality (AR) display, and the at least one processor is configured to apply the calibrated and aligned frame of reference when presenting images of the patient's internal anatomy on the AR display.

[0041] In some embodiments, the processor is configured to calculate a first transformation between the fluoroscope's frame of reference and the alignment target, calculate a second transformation between the alignment target and the patient's body, and combine the first transformation and the second transformation to align the fluoroscope's frame of reference with the patient's body.

[0042] In some embodiments, the plurality of images includes first and second x-ray images captured by a fluoroscope at different first and second angles relative to the body, and in some embodiments, the at least one processor is configured to process both the first and second x-ray images to calibrate and align a frame of reference of the fluoroscope with the patient's body.

[0043] In some embodiments, the registration target is configured to be fixed to a patient's bone in a predefined spatial relationship relative to the patient marker.

[0044] In some embodiments, the system includes an alignment optical target having a predetermined spatial relationship to the alignment target, and one or more images captured in a procedure room (e.g., an operating room) include optical images of both the alignment optical target and the patient marker, and the patient marker is configured to be fixed to the body of a patient undergoing surgery in the operating room.

[0045] In some embodiments, the method includes an alignment marker. The alignment marker may include an alignment target and an alignment optical target. The alignment marker may be configured to be fixed to a surface of the patient's body.

[0046] In some embodiments, the X-ray calibration fixture includes an alignment target, and the alignment optical target is fixed to the X-ray calibration fixture. In some embodiments, the alignment target is configured to be fixed in position during acquisition of the X-ray images and then removed during surgery.

[0047] In some embodiments, the alignment target (eg, alignment marker) comprises a radiopaque pattern.

[0048] In some embodiments, the radiopaque pattern comprises radiopaque elements arranged in a plurality of different planes.

[0049] In some embodiments, the fluoroscope includes an x-ray source and an x-ray detector. In some embodiments, the x-ray calibration fixture includes at least one ring, the ring including the x-ray calibration pattern and configured to be mounted over the x-ray detector.

[0050] In some embodiments, the at least one ring includes or consists essentially of first and second rings that are parallel to one another and spaced apart along the optical axis of the fluoroscope and include respective sub-patterns of radiopaque elements, hi some embodiments, the processor is configured to compare the sub-patterns in the x-ray image to calibrate the frame of reference of the fluoroscope.

[0051] In some embodiments, the X-ray calibration fixture includes a plurality of pads disposed around the circumference of the at least one ring and configured to lock against a periphery of the X-ray detector.

[0052] In some embodiments, the pads are configured to shift radially to engage and lock onto the periphery of the x-ray detector.

[0053] In some embodiments, the x-ray calibration fixture includes an auto-centering mechanism configured to shift the pads together to center the at least one ring relative to the periphery of the x-ray detector.

[0054] In some embodiments, the X-ray calibration fixture includes a safety strap configured to secure the at least one ring to the X-ray detector.

[0055] In some embodiments, the X-ray calibration fixture includes a flexible band configured to clamp around a periphery of the X-ray detector.

[0056] According to some embodiments, a method for image-guided surgery or other medical intervention includes receiving a 3D MR image of a patient's body including a target region including one or more bones where surgery will be performed. The method further includes processing the MR image to generate a segmented 3D image including bone segments and soft tissue proximate to the bone segments. The method also includes registering the segmented 3D image with the patient's body by aligning the bone segments in the segmented 3D image with one or more bones in the target region of the body. The method further includes presenting the registered segmented 3D image on a display.

[0057] In some implementations, presenting the registered segmented 3D image includes overlaying an augmented reality image including bone segments and soft tissue onto the target area of the body.

[0058] In some embodiments, the one or more bones include vertebrae.

[0059] In some embodiments, the one or more bones include a hip bone, a knee bone, an ankle bone, a skull bone, an arm bone, a leg bone, or a facial bone, and / or other bones.

[0060] In some embodiments, processing the MR image includes segmenting the MR image to identify both bone segments and soft tissue within the MR image.

[0061] In some embodiments, processing the MR image includes receiving and segmenting the CT image to identify bone segments, segmenting the MR image to identify soft tissue, and registering the MR image with the CT image to generate a segmented 3D image.

[0062] In some embodiments, registering the segmented 3D image with the patient's body includes capturing two or more fluoroscopic images of the target area, calibrating a frame of reference of the 2D fluoroscopic images with respect to the patient's body, and registering bone segments in the segmented 3D image with one or more bones in the 2D fluoroscopic images.

[0063] In some implementations, the display is an augmented reality display on a head-mounted unit or other wearable device, such as a pair of augmented reality glasses, a visor, a headset, or the like.

[0064] According to some embodiments, a method for image-guided surgery includes, or consists essentially of, receiving a 3D anatomical image of a target area of a patient's body where surgery is to be performed, the 3D anatomical image including one or more bones; processing the 3D anatomical image to generate a segmented 3D image including bone segments and soft tissue proximate to the bone segments; and registering the segmented 3D image with the patient's body by aligning the bone segments in the segmented 3D image with one or more bones in the target area of the body.

[0065] According to some embodiments, a display method based on the registration of 3D magnetic resonance images and 2D fluoroscopic images includes, or essentially consists of, receiving 3D MR images of a patient's body, including a target region of the spine, including one or more vertebrae, where surgery or other medical intervention will be performed. The method also includes, or essentially consists of, processing the MR images to generate a segmented 3D image including vertebral segments and soft tissue adjacent to the vertebral segments. The method further includes, or essentially consists of, receiving two 2D fluoroscopic images of the target region. The method also includes, or essentially consists of, receiving initial input associating vertebral segments in the segmented 3D image with the same vertebral segments in the two 2D fluoroscopic images. The method further includes, or essentially consists of, estimating the orientation of the spine, associating all vertebrae in the 2D fluoroscopic image with corresponding vertebral segments in the segmented 3D image based on the initial input and the estimated orientation, and generating a digitally reconstructed x-ray image from each segmented vertebral segment in multiple orientations. The method further comprises, or essentially consists of, determining an optimal orientation and position of the digitally reconstructed x-ray image to match the vertebrae in the 2D fluoroscopic image. The method also comprises, or essentially consists of, reconstructing a spine model using the segmented 3D image at the determined optimal orientation and position. The method further comprises, or essentially consists of, generating a spine model for display.

[0066] According to some embodiments, a method for image-guided surgery on a patient includes receiving a first preoperative 3D image of soft tissue of a target region within the patient's body where surgery or other medical intervention will be performed, the first image including one or more bones. The method further includes receiving a second 3D image of the one or more bones of the target region registered intraoperatively with the patient's anatomy. The method also includes aligning the first 3D image with the second 3D image by aligning the one or more bones in the first and second images. The method further includes generating a third 3D image including the one or more bones and soft tissue proximate to the one or more bones. The third 3D image is registered with the patient's anatomy. The method also includes displaying the third 3D image.

[0067] In some embodiments, the second 3D image is a CT image.

[0068] In some embodiments, the second 3D image is an intraoperative image that is registered with the patient's anatomy.

[0069] In some embodiments, the second 3D image is a pre-operative image that is registered with one or more intra-operative 2D fluoroscopic images of one or more bones.

[0070] In some embodiments, the second 3D image is a segmented image that includes a segmentation of one or more bones.

[0071] In some embodiments, registering the second 3D image with the one or more 2D fluoroscopic images includes aligning one or more segmented bones in the second 3D image with one or more bones in the 2D fluoroscopic image.

[0072] In some embodiments, the second 3D image is the first 3D image registered with one or more intraoperative 2D fluoroscopic images of one or more bones.

[0073] In some embodiments, the second 3D image is a segmented image that includes a segmentation of one or more bones.

[0074] The use of any of the devices, systems, or methods for treatment of the spine by surgical intervention is also described and contemplated herein.

[0075] Also described and contemplated herein is the use of any of the devices, systems, or methods for treating orthopedic joints, including the shoulder, knee, ankle, hip, or other joints, optionally by surgical intervention.

[0076] The use of any of the devices, systems, or methods for treatment of the skull by surgical intervention is also described and contemplated herein.

[0077] The use of any of the devices, systems, or methods for treatment of the jaw by surgical intervention is also described and contemplated herein.

[0078] Also described and contemplated herein is the use of any of the devices, systems, or methods for diagnosing spinal abnormalities or degeneration or deformation.

[0079] The use of any of the devices, systems, or methods for the diagnosis of spinal cord injury is also described and contemplated herein.

[0080] The use of any of the devices, systems, or methods for the diagnosis of joint damage is also described and contemplated herein.

[0081] Also described and contemplated herein is the use of any of the devices, systems, or methods for the diagnosis of orthopedic injuries.

[0082] According to some embodiments, any of the methods described herein may include diagnosing and / or treating a medical condition, the medical condition comprising one or more of back pain, spinal deformity, spinal stenosis, herniated disc, joint inflammation, joint injury, ligament or tendon rupture or tear.

[0083] According to some embodiments, methods are described and / or illustrated herein for presenting one or more images on a wearable display during a medical procedure, such as an orthopedic surgical procedure, a spinal surgical procedure, a joint repair procedure, a joint replacement procedure, a facial bone repair or reconstruction procedure, an ENT procedure, a cranial procedure, or a neurosurgical procedure.

[0084] For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features of embodiments of the present disclosure have been described herein. It should be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the disclosure disclosed herein. Thus, the embodiments disclosed herein may be implemented or performed to achieve or optimize one advantage or advantages as taught or suggested herein, without necessarily achieving other advantages as taught or suggested herein. The systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein. While the methods summarized above and described in further detail below describe specific actions taken by a practitioner, it should be understood that they may also include direction of those actions by another party. Thus, an action such as "capturing one or more 2D fluoroscopic images" includes "directing the capture of one or more 2D fluoroscopic images."

[0085] The present disclosure will be more fully understood from the following detailed description of the embodiments thereof, taken in conjunction with the drawings. [Brief explanation of the drawings]

[0086] [Figure 1]1 is a schematic diagram of a system for image-guided surgery, according to an embodiment of the present disclosure;

[0087] [Figure 2A] FIG. 2 is a schematic diagram of a head-mounted unit for use in the system of FIG. 1.

[0088] [Figure 2B] FIG. 2 is a schematic diagram of another head-mounted unit for use in the system of FIG. 1.

[0089] [Figure 3A] 1 is a flowchart of an augmented reality assisted navigation workflow including intraoperative imaging only.

[0090] [Figure 3B] 1 is a flowchart of an augmented reality assisted navigation workflow that includes both pre-operative and intra-operative imaging.

[0091] [Figure 4A] FIG. 1 is a perspective view of an X-ray calibration fixture without an alignment marker attached. [Figure 4B] FIG. 1 is a perspective view of a calibration jig with alignment markers attached.

[0092] [Figure 5] 4A and 4B show examples of bead plates for the X-ray calibration fixture.

[0093] [Figure 6] FIG. 10 is a perspective view of another example of the configuration of the X-ray calibration jig to which alignment markers are attached.

[0094] [Figure 7] 10 shows another example of the configuration of the X-ray calibration jig.

[0095] [Figure 8] 4C shows an example of the configuration of the alignment marker shown in FIG. 4B.

[0096] [Figure 9] 7 shows an example of the configuration of the components on the lower side of the alignment marker shown in FIG. 6.

[0097] [Figure 10A] FIG. 10 is a schematic rear view of the slide mechanism of the mounting assembly of the X-ray calibration fixture. [Figure 10B] FIG. 10 is a schematic rear view of the slide mechanism of the mounting assembly of the X-ray calibration fixture.

[0098] [Figure 11] FIG. 1 is a schematic diagram of an X-ray calibration fixture attached to a fluoroscope.

[0099] [Figure 12A] FIG. 1 is a schematic diagram of an x-ray calibration fixture with a mechanism for securing the fixture to the detector portion of an x-ray machine or fluoroscope. [Figure 12B] FIG. 1 is a schematic diagram of an x-ray calibration fixture with a mechanism for securing the fixture to the detector portion of an x-ray machine or fluoroscope. [Figure 12C] FIG. 1 is a schematic diagram of an x-ray calibration fixture with a mechanism for securing the fixture to the detector portion of an x-ray machine or fluoroscope.

[0100] [Figure 13] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 14] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 15] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 16] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 17] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 18] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 19] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope. [Figure 20] 1 is a schematic diagram of an exemplary mounting mechanism for mounting an X-ray calibration fixture to an X-ray machine or fluoroscope.

[0101] [Figure 21A] FIG. 1 is a schematic diagram of an alignment target attached to a patient's back by pins. [Figure 21B] FIG. 1 is a schematic diagram of an alignment target attached to a patient's back by pins.

[0102] [Figure 22] FIG. 1 is a schematic cross-sectional view of an alignment target attached to a pin.

[0103] [Figure 23] 1 is a schematic cross-sectional view of an alignment target attached to a clamp configured to be attached to a patient's spine.

[0104] [Figure 24] FIG. 2 is a schematic diagram of an alignment target.

[0105] [Figure 25] FIG. 1 is a schematic diagram of a multimodal alignment target attached to a patient's back.

[0106] [Figure 26] FIG. 1 is a schematic diagram of a multimodal alignment target attached to a patient's back.

[0107] [Figure 27] FIG. 10 is a schematic diagram of another exemplary configuration of a multimodal alignment target.

[0108] [Figure 28] 1 is a flow chart that schematically illustrates a display method based on registration of a three-dimensional (3D) image and a two-dimensional (2D) image.

[0109] [Figure 29] FIG. 1 is a schematic diagram of the CT-Fluoro calibration process.

[0110] [Figure 30] 1 is a flow chart that schematically illustrates an example of a calibration method.

[0111] [Figure 31A] 1 is a flow chart of flow chart steps illustrating bead detection. [Figure 31B] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating bead detection.

[0112] [Figure 32A] 1 is a flow chart of flow chart steps illustrating grid association. [Figure 32B] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32C] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32D] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32E] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32F] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32G] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association; [Figure 32H] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating grid association;

[0113] [Figure 33A] 1 is a flow chart of flow chart steps illustrating marker association. [Figure 33B] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating marker association. [Figure 33C] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating marker association. [Figure 33D] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating marker association. [Figure 33E] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating marker association. [Figure 33F] 10A-10C are corresponding schematic diagrams of flow chart steps illustrating marker association.

[0114] [Figure 34] 1 is a flow chart that schematically illustrates a method for reconstructing and displaying an augmented reality image of a spine.

[0115] [Figure 35] FIG. 1 is a schematic diagram of a segmented three-dimensional image of a vertebra.

[0116] [Figure 36A] 1 is a screenshot of an exemplary embodiment of a graphical user interface (GUI) display for segmenting a CT image of the spine. [Figure 36B] 1 is a screenshot of an exemplary embodiment of a graphical user interface (GUI) display for segmenting a CT image of the spine. [Figure 36C] 1 is a screenshot of an exemplary embodiment of a graphical user interface (GUI) display for segmenting a CT image of the spine.

[0117] [Figure 37] FIG. 1 is a schematic diagram of a method for registering two-dimensional and three-dimensional anatomical images.

[0118] [Figure 38A] 36A-36C are screenshots of an exemplary embodiment of a GUI display for registering a fluoroscopic image with the segmented CT image of FIGS. 36A-36C. [Figure 38B] 36A-36C are screenshots of an exemplary embodiment of a GUI display for registering a fluoroscopic image with the segmented CT image of FIGS. 36A-36C. [Figure 38C] 36A-36C are screenshots of an exemplary embodiment of a GUI display for registering a fluoroscopic image with the segmented CT image of FIGS. 36A-36C.

[0119] [Figure 39A] 38A-38C are screenshots of an exemplary embodiment of a GUI display showing different views of an aligned vertebra (L3) in a segmented CT image overlaid on the fluoroscopic image of FIGS. 38A-38C (or vice versa). [Figure 39B] 38A-38C are screenshots of an exemplary embodiment of a GUI display showing different views of an aligned vertebra (L3) in a segmented CT image overlaid on the fluoroscopic image of FIGS. 38A-38C (or vice versa).

[0120] [Figure 40A] 10 is a screenshot of an exemplary GUI display showing different views of an aligned vertebra (L4) in a segmented CT image overlaid on a fluoroscopic image (or vice versa). [Figure 40B] 10 is a screenshot of an exemplary GUI display showing different views of an aligned vertebra (L4) in a segmented CT image overlaid on a fluoroscopic image (or vice versa).

[0121] [Figure 41]1 is a flowchart that schematically illustrates an exemplary method for generating and displaying a three-dimensional (3D) model based on registration of a three-dimensional image (e.g., an MR image) and a two-dimensional (2D) image (e.g., a fluoroscopic image).

[0122] [Figure 42] FIG. 1 is a schematic diagram of a segmented 3D image for display in image-guided surgery.

[0123] [Figure 43A] 1 is a flow chart that schematically illustrates a modality for image registration, fusion, and display. [Figure 43B] 1 is a flow chart that schematically illustrates a modality for image registration, fusion, and display. [Figure 43C] 1 is a flow chart that schematically illustrates a modality for image registration, fusion, and display.

[0124] [Figure 44A] 1 is a schematic diagram of an image of a bead plate showing distortions that occur in an X-ray image. [Figure 44B] 1 is a schematic diagram of an image of a bead plate showing distortions that occur in an X-ray image. [Figure 44C] 1 is a schematic diagram of an image of a bead plate showing distortions that occur in an X-ray image.

[0125] [Figure 45] 1 is a flowchart that schematically illustrates an exemplary method for refining image data as part of a distortion correction process.

[0126] [Figure 46] 1 is a flowchart that outlines an exemplary method for interpolating data as part of a distortion correction process. DETAILED DESCRIPTION OF THE INVENTION

[0127] Embodiments of the present disclosure described below provide apparatus, methods, and software for image calibration, alignment, and display to facilitate image-guided and augmented reality-assisted navigation, particularly during medical and / or diagnostic procedures (e.g., open surgery or minimally invasive surgery such as laparoscopic or endoscopic surgery).

[0128] In some systems for image-guided surgery or other medical interventions, anatomical images of structures within a patient's body are overlaid on the surgeon's actual view of the patient's body, generating an augmented reality view that can be used to facilitate navigation by a viewer of the augmented reality view (e.g., a wearer of a head-mounted AR display device). Such displays of 3D anatomical images, such as computed tomography (CT) or magnetic resonance (MR) images, can be particularly useful in allowing the surgeon to visualize structures hidden from the actual view by overlying layers of tissue or bone. For example, during orthopedic surgery, an augmented reality (AR) display can show 3D images of bone segments overlaid on the corresponding bone locations in the target area of the patient's body. For example, during spine surgery, 3D images of vertebrae may be overlaid on the skin of the patient's back in the case of minimally invasive surgery, or on the actual vertebrae in the case of open surgery.

[0129] In image-guided surgery or other medical interventions, it is important that the anatomical images displayed to the surgeon to provide guidance within the patient's body and / or facilitate navigation (e.g., of medical tools and instruments) correspond to the patient's current anatomical structure (e.g., pose and / or structure). Furthermore, for this type of AR to be clinically useful, it can be important that the overlaid 3D images be properly aligned with the patient's actual anatomical structure within the body. When 3D images are acquired during surgery, for example, using an intraoperative medical imaging scanner such as a CT or MRI scanner, proper alignment is maintained as long as the patient remains motionless. However, in most surgeries or other medical interventions, the 3D images are acquired in a different room prior to the surgery or other medical intervention, and for example, the patient's position on the operating table is often different from the preoperative images (e.g., tomographic, ultrasound, or MR images). Preoperatively acquired 3D images typically lack fiducials (e.g., fiducial markers) that allow the preoperative 3D images to be aligned with the patient's anatomical structure during surgery. Furthermore, there may be changes in the relative positions of individual anatomical components (e.g., vertebrae of a patient's spine) between the time the 3D images are acquired and the time of surgery or other medical intervention. Such changes may be due to changes in the patient's posture, the insertion of an implant, or any other reason. For spine surgery, surgeons typically acquire 2D images intraoperatively using a fluoroscope in the operating room and use these 2D images for intraoperative guidance while viewing pre-acquired medical images (e.g., tomographic images) of the spine offline.

[0130] The embodiments of the present disclosure described herein provide methods, systems, and computer software products that can be used to register a pre-acquired 3D medical image (e.g., a tomographic or MR image) with an intraoperative 2D fluoroscopic or X-ray image. In the disclosed methods, systems, and computer software products, the 3D image is segmented into multiple 3D segments, each including, for example, a respective one of the vertebrae for a spinal embodiment. In a spinal embodiment, each of these 3D segments is registered with a respective vertebra in the fluoroscopic image. Specifically, the respective positions and orientations of each 3D segment correspond to the respective vertebrae in the fluoroscopic image and can therefore be adjusted to account for changes in the relative positions of the vertebrae (e.g., due to changes in the patient's position on the operating table relative to their position in the 3D image). In the disclosed embodiments, two fluoroscopic images captured from different angles or viewpoints are used together in this registration process; alternatively, a larger number of fluoroscopic images (e.g., three, four, or more) or viewpoints can be used. Similar techniques may be used for other types of surgery or medical interventions. For example, in some embodiments, the 3D image may be segmented into other bone portions or components.

[0131] In a spine surgery context, once the registration process is complete, an image of the spine including the registered 3D segments is presented on a display, for example, by overlaying an AR image of the registered 3D segments onto the patient's back to generate an AR view. In some embodiments, to ensure proper registration between the AR image and the patient's body, the reference frame of the 2D fluoroscopic image is calibrated to the patient's body (e.g., a portion of the patient's back corresponding to the target treatment area of the spine), for example, using calibration markers described below. This calibrated reference frame can then be applied to the generated 3D image volume or model of the spine to properly align the vertebrae in the registered 3D segments with the patient's spine. Similar techniques can be used in non-spine embodiments, where the calibration and registration and display can be tailored to the specific anatomical structures associated with a particular medical intervention (e.g., other orthopedic surgery or intervention, neurosurgery or other intervention, ENT surgery or other intervention, oral surgery or other intervention).

[0132] According to some embodiments, it may be desirable to combine image information from multiple different imaging modalities and present the combined image information to a surgeon or other clinical professional, for example, on an augmented reality display of a wearable device such as a head-mounted unit or eyewear (e.g., goggles, visors, or glasses), and / or on a non-wearable device such as a tablet, portal monitor, or workstation display. Each imaging modality and device can have its own frame of reference that is separate and independent from the other modalities and devices and typically suffers from different types of distortion. These imaging modalities can include, for example, optical cameras used to capture visible and / or infrared images of a patient's body, fluoroscopes that capture 2D x-ray images of a patient's body in an operating room or diagnostic room, and medical imaging scanners (e.g., tomography scanners such as CT and MRI scanners that can be used to capture preoperative or intraoperative 3D scans of the body). Ultrasound scanners or other 3D or 2D imaging modalities can also be used. In some aspects, the imaging modality must be capable of imaging bone tissue.

[0133] According to some embodiments, in order to combine image information from such different sources in a manner that can provide useful guidance to a surgeon or other clinical professional, it is desirable for all imaging frames of reference to be calibrated and aligned with the reference frame of the patient's body (and thus all imaging frames to be aligned with each other). In certain embodiments of AR displays in which image information from these sources is visually overlaid on the body itself, accurate calibration and alignment can be important to facilitate accurate and precise navigation by a surgeon or other clinical professional relying on the AR display.

[0134] The embodiments of the present disclosure described herein provide apparatus and methods that address various advantages, particularly for operating rooms where fluoroscopic x-ray and optical imaging are used together in image-guided surgery. For these purposes, an x-ray calibration fixture (e.g., a ring adapter) containing an x-ray calibration pattern may be fixed (e.g., attached, mounted, or otherwise coupled) to a fluoroscope (e.g., the detector portion of a C-arm fluoroscope) used in the operating room. Patient markers may be fixed to the body of the patient undergoing surgery, and alignment targets (e.g., alignment markers) may be securely attached to the x-ray calibration fixture or the patient or another location, such as the operating table. The alignment targets may be used to align the x-ray reference frame of the fluoroscope with the optical reference frame. One or more alignment targets, typically one or two targets, may be utilized, each or some of which may be located at different locations (e.g., securely attached to the x-ray calibration fixture or attached to the patient or elsewhere). The alignment targets may be in the form of alignment markers. Each alignment target or marker can include an optical pattern and / or a radiopaque pattern, all depending on the system configuration, as described below.

[0135] A processor (e.g., one or more processing devices or units) may be configured to receive images captured in the operating room, including one, two, or more x-ray images captured by a fluoroscope (including an x-ray calibration pattern) and an optical image of the patient markers. In some embodiments, at least one of the images (e.g., either the x-ray image or the optical image, or both) includes an alignment target (e.g., one or more alignment markers). In disclosed embodiments, the processor receives and uses two or more x-ray images captured by the fluoroscope at different angles or perspectives relative to the body (e.g., anterior-posterior and lateral). In some embodiments, the processor processes the x-ray image or images along with the optical image to calibrate and align the fluoroscope's frame of reference with the patient's body. To this end, the processor typically calculates a first transformation between the fluoroscope's reference frame and the alignment target (e.g., marker), and a second transformation between the alignment target (e.g., marker) and the patient's body, and then combines these two transformations to align the fluoroscope's reference frame with the patient's body.

[0136] In some embodiments, the processor applies the calibrated and registered fluoroscopic frame of reference when presenting images of anatomical structures within the patient's body (e.g., individualized vertebrae, portions of the spine (lumbar, sacrum, lumbar, cervical, thoracic), entire spine, pelvic bones, leg bones, arm bones, hip joints, knee joints, ankle or foot bones, hand bones, brain tissue, skull, oral and maxillofacial bones, bone joints such as the sacroiliac joint, organs, or other soft tissues, etc.) on a display, such as an AR display. Additionally or alternatively, other types of information may be integrated into the AR images. For example, the display may incorporate information from pre-acquired 3D tomographic or other medical images, such as CT or MRI images. To this end, it is desirable that the tomographic or other medical images also be registered with the patient's body. This type of registration can be achieved, for example, by registering the pre-acquired 3D tomographic or other medical images with intraoperative 2D fluoroscopic images, as described below.

[0137] As used hereinafter, the term "image" can include two-dimensional images and / or three-dimensional images, including computer-generated two-dimensional or three-dimensional renderings or models.

[0138] Some embodiments are particularly advantageous because they include one, some, or all of the following benefits: (i) improved visualization of internal body structures during surgery, (ii) increased precision in planning and executing surgical procedures, (iii) increased availability of AR-assisted navigation without sacrificing accuracy, (iv) improved surgeon understanding of 3D features of the patient's anatomy, (v) reduced patient exposure compared to 3D-CT intraoperative imaging, and / or (vi) reduced overall procedure time due to the typical availability of C-arm machines compared to O-arm or other CT machines.

[0139] Fluoroscopy Calibration FIG. 1 is a schematic diagram of an AR system 20 for image-guided surgery or other medical intervention using AR-assisted navigation, according to one embodiment of the present disclosure. In the illustrated scenario, a surgeon or other clinical professional 22 is preparing to operate on the spine of a patient 24 reclining on an operating table 26. The surgeon 22 views the patient's back through a head-mounted AR display unit 28, an example of which is shown in more detail in FIG. 2A or 2B . Before and / or possibly during surgery, a fluoroscope 30 is used to acquire two-dimensional images of at least a portion of the patient's 24 spine (and potentially other bones, blood vessels, and / or soft tissue or other internal structures) from multiple different angles (e.g., anterior-posterior and lateral views). The fluoroscope 30 includes an X-ray source 32 and an X-ray detector 34, which are held on either side of the patient's body by a C-arm 36.

[0140] For purposes of providing image guidance during surgery or other medical intervention, particularly for purposes of augmented reality image display and assisted navigation, it can be important that the frame of reference of images captured by the fluoroscope 30 be calibrated and aligned with the physical frame of reference of the patient's 24 body. According to some embodiments, this calibration has two aspects: (1) correcting distortions in images captured by the fluoroscope 30 itself and calculating internal parameters of the fluoroscope 30 (e.g., focal length, principal point, skew), and (2) aligning the fluoroscope 30 with the patient's body. For these purposes, an X-ray calibration fixture 38 (e.g., a fluoroscopy ring adapter) can be fitted or otherwise mounted or attached to the X-ray detector 34. In the illustrated embodiment, the fixture 38 includes or consists essentially of an X-ray calibration pattern in the form of an array of radiopaque beads or other fiducial elements 40 in a predetermined layout, as described further below. This bead pattern appears in fluoroscopic images captured by the X-ray detector 34. The processor 50, which may include one or more processing devices or units, receives and processes the X-ray images to correct distortions, including external and internal parameters of the fluoroscope 30 (e.g., the X-ray detector 34), and determines the position and orientation of the optical axis of the fluoroscope 30.

[0141] 1, the X-ray calibration fixture 38 includes or consists essentially of an alignment target in the form of an optical marker 42 that includes an optical pattern and is fixed to the fixture 38 in a known position and orientation relative to the pattern of the bead 40. Alternatively or additionally, the alignment target 42 may comprise a radiopaque pattern and may be fixed to the body of the patient 24, or may be fixed to the patient table or another location.

[0142] In some embodiments, the processor 50 uses the optical marker 42 in conjunction with optical patient markers or other fiducial markers on the body of the patient 24 when aligning the optical axis of the fluoroscope 30 with the body of the patient 24. In some embodiments, the surgeon 22 can attach the patient marker 44 to a bone within the patient 24, such as the patient's spine, using an appropriate clamp (e.g., a spinous process clamp) or pin (e.g., an iliac pin). Markers of this type are described, for example, in U.S. Pat. No. 10,939,977, the disclosure of which is incorporated herein by reference. Additionally or alternatively, the surgeon 22 can secure the alignment marker 46 to the surface of the patient's body, as shown, for example, in FIG. 1. Alignment procedures utilizing markers attached to the patient's back are described, for example, in U.S. Patent Application Publication No. 2021 / 0161614, the disclosure of which is also incorporated herein by reference. Alternatively, the surgeon 22 can use alignment markers attached to the patient's spine via support or attachment structures, such as clamps or pins, as shown in FIGS. 21A-23. Alignment procedures utilizing markers attached to the patient's spine via such support or mounting structures are disclosed, for example, in U.S. Patent Application Publication No. 2022 / 0142730, the disclosure of which is also incorporated herein by reference. Note that the use of alignment markers, such as alignment marker 46, can obviate the need for optical marker 42. In some implementations, camera 48 (e.g., an infrared camera or other optical camera) captures images including both optical marker 42 on calibration fixture 38 and markers 44 and / or 46 attached to patient 24. Note, however, that camera 48 in FIGS. 1 , 2A, and 2B is mounted on head-mounted AR display unit 28; these images may alternatively be captured by one or more suitable optical cameras (e.g., infrared cameras) mounted elsewhere on surgeon 22's head or body, or mounted elsewhere in the operating room (e.g., in a stationary manner).

[0143] According to some embodiments, the processor 50 processes images captured by the X-ray detector 34, and according to some embodiments, also by the optical camera 48 (e.g., an infrared camera), to calculate the position and orientation of the fluoroscope 30 relative to the patient's body, thus calibrating and aligning the fluoroscope reference frame with respect to the patient's body reference frame. Specifically, the processor 50 calculates a first transformation between the reference frame of the fluoroscope 30, as represented by the fixture 38, and the optical reference frame of an alignment target (e.g., optical marker 42) fixed to the fixture 38 in this embodiment. In this case, the geometric relationship between the fixture 38 and the optical marker 42 is fixed and known in advance, simplifying the calculation of the transformation. Alternatively, the processor 50 can calculate a first transformation between the X-ray calibration pattern of the bead 40 on the fixture 38 and the radiopaque pattern on the alignment marker (e.g., alignment marker 46) in a calculated, determined, or predefined spatial relationship to the patient marker 44, as illustrated in some of the following figures. According to some embodiments, the processor 50 calculates a second transformation between the alignment target (e.g., the alignment marker 46) and the patient's body (e.g., using the optical pattern of the alignment marker 46 and the optical pattern of the patient marker 44). The processor 50 can then combine these two transformations to align the frame of reference of the fluoroscope 30 with the body of the patient 24 (e.g., the patient's spine, skull, mouth, orthopedic joints, or other portions of the patient's anatomy relevant to the medical intervention being performed).

[0144] In addition to receiving two-dimensional x-ray images from the fluoroscope 30, the processor 50 can also receive three-dimensional tomographic or other medical images (e.g., CT or MRI images) of the patient 24 and store these three-dimensional images in the memory 52. In the case of a spinal interventional procedure, the processor 50 can segment the three-dimensional image and align the three-dimensional segments with respective vertebrae in the two-dimensional fluoroscopic image. The processor 50 can then present an image of the spine, including the aligned 3D segments, on the head-mounted AR display unit 28 so that the vertebrae in the 3D image are aligned with the actual vertebrae of the patient's spine. Such presentation can facilitate AR-assisted navigation during a surgical procedure or other medical intervention (e.g., therapeutic and / or diagnostic intervention). Details of this process are described herein. Similar processes can be performed for other joints, bones, or tissues.

[0145] Alternatively or additionally, the processor 50 may present the image information on a different type of display, for example, an AR display mounted elsewhere in the operating room, such as on the operating table 26 above the patient 24 or surgical site, or on a fixed display (e.g., a workstation display) located in the operating room.

[0146] Processor 50 may comprise one or more general-purpose computer processors programmed with software (via computer-readable program instructions) to perform the segmentation, calibration, alignment, and / or display functions described herein. This software may be stored on a tangible, non-transitory computer-readable medium, such as an optical, magnetic, or electronic memory medium. Additionally or alternatively, at least some of the functions of processor 50 may be performed using dedicated computing hardware, such as, for example, a graphics processing unit (GPU), which may include multiple units.

[0147] FIG. 2A is a schematic diagram illustrating details of a head-mounted AR display unit 28 according to one embodiment of the present disclosure. The head-mounted display unit 28 is in the form or substantially in the form of glasses, goggles, or other eyewear. The head-mounted display unit 28 includes a see-through display 60, such as that described in the aforementioned U.S. Pat. No. 9,928,629 or PCT Publication No. WO 2022 / 053923. The see-through display 60 may include an optical see-through display, a video see-through display, or a hybrid combination of both. The see-through display 60 may also include a stereoscopic display. The head-mounted display unit 28 may include eyewear (e.g., glasses or goggles) as shown in FIG. 2A. Alternatively, the head-mounted display unit 28 may include a headset configured to be worn on the head of the surgeon 22, rather than just worn on the ears and nose (and / or forehead) of the surgeon 22, as shown in the unit of FIG. 2B. The display 60 can be controlled by the processor 50 (e.g., by a processor unit of the processor 50 located in the head-mounted AR display unit 28, not shown) to display AR images to the surgeon 22 wearing the head-mounted AR display unit 28. In some embodiments, the AR images are projected onto an overlay area 62 of the display 60 in alignment with the anatomical structures of the patient's 24 body as viewed by the surgeon 22 via the display 60. The AR images can include, for example, images or 3D models taken from cross-sectional or volumetric images of the patient's interior and / or graphical representations of tools or anatomical features such as bone representations, as well as surgical guidance and planning data or other information. The AR images can be overlaid on the actual locations of the anatomical features of the patient 24 as viewed by the surgeon 22. In some embodiments, the AR images are presented directly in or on the retina of one or both of the patient's eyes.

[0148] To align the AR image with the patient's anatomy, one or more cameras 48 (e.g., infrared or other optical cameras) can be configured to capture respective images of a field of view (FOV) that includes marker 42, and also, for alignment purposes, images that include markers 46 and / or 44. In some embodiments, processor 50 processes the images of one or more of the markers to align the position and orientation of display unit 28 with the patient's body. Based on this alignment, processor 50 can select appropriate features to display in the AR image within overlay region 62 (which may be displayed directly on the wearer's retina) and set the appropriate magnification, translation, and orientation to match the underlying structures of the patient's anatomy from the perspective of surgeon 22 or other clinical professional.

[0149] FIG. 2B is a schematic diagram showing details of a head-mounted display (HMD) unit 70 according to another embodiment of the present disclosure. The HMD unit 70 may be worn by the surgeon 22 or may be used in place of the HMD unit 28 (FIG. 2A). The HMD unit 70 includes an optical housing 74 incorporating a camera 78, and in the particular embodiment shown, an infrared camera. Accordingly, the housing 74 also includes an infrared-transparent window 75, and one or more (e.g., two) infrared projectors 76 are mounted within the housing (e.g., behind the window). A pair of augmented reality displays 72 are mounted on the housing 74, allowing the surgeon 22 to view entities, such as part or all of the patient 24, through the displays 72, which are also configured to present AR images or any other information to the surgeon 22.

[0150] HMD unit 70 includes a processor 84 mounted in a processor housing 86 that operates the elements of the HMD unit. An antenna 88 may be used for communication with processor 50 (e.g., a processor mounted on a workstation). Processor 84 may be the processing unit of processor 50 or may be in communication with processor 50. HMD unit 28 of FIG. 2A may also include one or more processors, similar to HMD unit 70.

[0151] Optionally, a flashlight 82 can be mounted on the front of the HMD unit 70. The flashlight 82 can project visible spectrum light onto objects so that the surgeon 22 can clearly see the objects through the display 72. The elements of the HMD unit 70 are typically powered by a battery (not shown) that provides power to the elements via a battery cable input 90. The HMD unit 28 of FIG. 2A may also include a flashlight, similar to the HMD unit 70.

[0152] FIG. 3A is a flowchart of an augmented reality-assisted navigation workflow in which each step, including the acquisition of a 3D scan of a patient 24, is performed intraoperatively during surgery or other medical intervention. In intraoperative step 300, one or more fiducial markers are attached to the patient 24 and / or medical tools. In step 302, a 3D scan of the patient 24 is acquired and stored in memory (e.g., imported into a head-mounted unit and / or workstation memory). In step 304, registration is performed between the markers and the 3D scan, and in step 306, a 3D image model is created and displayed based on the registration to facilitate AR-assisted navigation. In step 308, the surgeon 22 navigates (e.g., inserts screws into the patient 24) based on the AR display. In some cases, intraoperative 3D scanning can be an expensive and time-consuming operation, taking approximately 30-35 minutes. This time can include transport and positioning of the intraoperative imaging device (e.g., an O-arm machine), draping, imaging, instrument disassembly, and re-scrubbing. In some instances, intraoperative 3D scanning equipment may not be readily available at the location where the procedure is desired to be performed.

[0153] Figure 3B is a flowchart of a CT-Fluoro navigation workflow, illustrating a process that can eliminate the need for expensive and time-consuming intraoperative 3D scans (e.g., CT or MRI scans). This process involves acquiring a preoperative 3D scan (e.g., a CT scan) using more readily available and common intraoperative imaging devices, i.e., a C-arm fluoroscope or other 2D X-ray machine. In this workflow, the patient 24 undergoes a preoperative 3D imaging scan (e.g., a CT scan) in step 310. In step 312, the 3D imaging scan (e.g., the CT scan) is stored in memory (e.g., imported into a workstation's memory) and undergoes segmentation. Segmentation can be performed automatically by software or artificial intelligence (AI), such as by a trained neural network, or the user can make manual adjustments during segmentation as needed (e.g., using manual visualization and adjustment techniques). The process then enters the intraoperative phase. In step 314, fiducial markers are attached to a calibration fixture (e.g., a fluoroscopy ring adapter) coupled to the C-arm of the fluoroscope (e.g., the X-ray detector portion of the fluoroscope). In step 316, two or more intraoperative 2D images are acquired using the C-arm or other fluoroscope or 2D imaging device. A user (e.g., surgeon 22 or other clinical professional) generates initial guess markings in the two-dimensional images in step 318 by selecting one of the segmented vertebrae or other bone portions and marking its location in each of the fluoroscopic images. The initial guess markings can also be performed automatically by processor 50. In some embodiments, the initial guess is calculated by obtaining the Z direction of the marker (e.g., an alignment marker coupled to the fluoroscope), which should correspond to the patient's waist-to-back direction, and the Y direction of the X-ray emitter or X-ray source, which should correspond to the patient's legs-to-head direction. Combining these two directions can define a coordinate system that is approximately parallel to the patient's orientation in the preoperative CT or other preoperative images.In step 320, the processor, upon execution of the stored program instructions, aligns each vertebral body or other bony structure or portion. In some embodiments, a user can manually assist in this alignment step. Next, in step 322, a user (e.g., surgeon 22 or other clinical professional) visually verifies each vertebral body alignment. Finally, in step 324, the user (e.g., surgeon 22 or other clinical professional) performs the procedure by navigation using the generated 3D volume of the vertebrae. Because the 3D scan is performed preoperatively, intraoperative imaging, including intraoperative fluoroscopic imaging, is expected to take less time (e.g., approximately 10-15 minutes) and does not require the use of 3D intraoperative imaging machines, saving time and money.

[0154] X-ray calibration fixture FIG. 4A is a perspective view illustrating one embodiment of an X-ray calibration jig 38. The X-ray calibration jig 38 comprises or consists essentially of two bead plates: an upper bead plate 406 and a lower bead plate 408. Each of the bead plates includes or consists essentially of a pattern of radiopaque or radiopaque beads 40 (e.g., metal beads). A strap holder 402 can be positioned on top of the chassis of the X-ray calibration jig 38 outside the upper bead plate 406 and can accommodate straps for coupling the X-ray calibration jig 38 to the X-ray detector 34 on the C-arm 36 to provide support and / or stability. In some embodiments, the chassis of the jig is manufactured as a single piece to improve accuracy of placement of the beads and markers 412. The bead plates 406, 408 may be configured and adapted to mate with the chassis of the jig 38 such that the bead plates 406, 408 are arranged according to a known or predetermined configuration.

[0155] The jig 38 further includes a ring tightening device 400 and a static clamp 404, which further includes a pad 416. The jig 38 may be adjustable to accommodate different sized fluoroscopes (e.g., a 9-inch version or a 12-inch version). The pad 416 includes a non-slip material (e.g., silicone) for stable attachment of the jig 38 to a fluoroscope (e.g., the detector portion of a C-arm or another fluoroscope). The X-ray calibration jig 38 includes or consists essentially of a marker holder 410. In some embodiments, the marker holder 410 can accommodate a marker 412 via a three-pin (e.g., three-screw) attachment mechanism to facilitate improved accuracy and ease manufacturing repeatability. In some configurations, the marker holder 410 can accommodate a marker via a variety of other attachment mechanisms. For example, in one configuration, the marker holder 410 can accommodate a marker via a two-pin attachment mechanism, a four-pin attachment mechanism, a snap-fit mechanism, a latch mechanism, or the like. In some embodiments, the X-ray calibration fixture 38 includes a Quick Response (QR) Code® element 414 or other machine-readable or information element. The QR code element 414 can store information such as parameter information regarding the location of the markers, or fluoroscopic camera parameters, or manufacturing parameters of the markers for the particular X-ray calibration fixture 38 (e.g., if manufacturing tolerances or repeatability are not sufficiently accurate or achievable). The QR code element 414 can include other information as desired and / or needed. The QR code element 414 can be scanned or read by a suitable imaging device or camera of the head-mounted display unit 28, 70 or a separate imaging device or camera in communication with the processor 50, and the information can be stored in memory of the head-mounted display unit 28, 70 and / or memory 52.

[0156] 4B shows the x-ray calibration fixture 38 of FIG. 4A with the marker 412 attached to the marker holder 410 via a three-pin attachment mechanism. In some embodiments, the marker 412 is configured or adapted for use as an "over-the-drape" marker. In this embodiment, a surgical drape can be placed on the fixture 38 (e.g., for purposes of maintaining a sterile field), and the marker 412 is connected to the marker holder 410 through the drape, such that a pin or screw on the marker 412 is pushed through the drape to couple the marker 412 to the marker holder 410.

[0157] FIG. 5 shows an example of the lower bead plate 408 and the upper bead plate 406 of the jig 38 of FIGS. 4A and 4B. In some implementations, the upper bead plate 406 is used for distortion correction, and the lower bead plate 408 is used to calculate camera or detector parameters (internal and external parameters) of a C-arm or other imaging device (e.g., a fluoroscopy or other X-ray imaging device or other 2D imaging device). The grid pattern and size of the beads 40 or other elements may vary as desired and / or needed, as long as they are different between the upper bead plate 406 and the lower bead plate 408. For example, the upper bead plate 406 may have more beads 40 than the lower bead plate 408, and the beads 40 of the upper bead plate 406 may be smaller than the beads 40 of the lower bead plate 408 for differentiation. In some embodiments, the grid of the upper bead plate 406 can form a complete grid layout with constant bead sizes and the same gap distance between vertical and horizontal lines. In some embodiments, the beads 40 are formed using a radiopaque material (e.g., titanium, stainless steel, tungsten, etc.). The use of radiopaque beads facilitates detection of various bead patterns. In some embodiments, the plate on which the beads are disposed is formed using a material that is radiolucent and durable under X-ray radiation. In some embodiments, the plate material is plastic or polymer (e.g., polyethylene terephthalate (PET)), or glass or ceramic. The beads may be replaced with other types of radiopaque elements other than beads. The cutouts in the bead may have shapes other than circular. Alternatively, the X-ray calibration fixture 38 may include a single ring or three or more rings, as well as other suitable types of geometric structures other than rings.

[0158] FIG. 6 illustrates another configuration of the X-ray calibration jig 38. Unless otherwise noted, the components in FIG. 6 are the same as or substantially similar to the components in FIGS. 4A-4B. FIG. 6 illustrates an example of a marker 600 adapted and configured for use under a drape, meaning that a sterile surgical drape can be completely draped over the jig 38, including over at least a portion of the marker 600. The surgical drape may be transparent. In some embodiments, a mechanism can be implemented that allows a portion of the drape to be stretched over the marker 600 so that the portion of the drape surrounding the marker 600 does not fold in on itself. In some embodiments, there is a groove 660 between the outer component of the marker 600 and the inner component of the marker 600. The groove 660 can facilitate the use of an elastomeric ring to stretch the drape snugly over the marker 600 without creases or bias that could affect image or motion quality. Additionally, the attachment mechanism may be a snap-fit attachment mechanism, where the outer component of marker 600 snaps into the inner component. In some embodiments, there are no separable outer and inner components, but rather an integrated component that snaps into appropriate mounting structure on calibration fixture 38. According to some embodiments, the snap-fit attachment mechanism may allow for quick attachment and removal of marker 600 from calibration fixture 38. In some embodiments, the snap-fit attachment mechanism may provide an audible click when proper attachment is achieved and includes a release button or latch that can be activated to cause easy removal of marker 600.

[0159] FIG. 7 is a schematic diagram showing details of an X-ray calibration jig 38 according to another embodiment of the present disclosure. Any of the structural and operational features described in connection with FIG. 7 can be incorporated into the calibration jig 38 of the previous figure. The X-ray calibration jig 38 of FIG. 7 comprises, or consists essentially of, two rings 140, 142 mounted across the X-ray detector 34 (as shown in FIG. 1) and containing different respective X-ray calibration subpatterns composed of radiopaque beads 40 or other types of radiopaque elements. Alternatively, the X-ray calibration jig 38 may comprise a single ring or three or more rings, as well as other suitable types of geometric structures other than rings. In the illustrated embodiment, the beads 40 are contained in a substrate 152 that is relatively transparent to X-rays, such as a glass or polymer substrate. The bead pattern of the jig 38 of FIG. 7 differs from the bead pattern of the jig 38 of FIGS. 4A and 4B. As shown, the bottom plate has a circular pattern as opposed to a more square or rectangular grid pattern, and the top plate also has a radial pattern as opposed to a more rectangular or square grid pattern.

[0160] Rings 140 and 142 are parallel to one another and spaced a known distance apart along the optical axis of fluoroscope 30, such that their respective subpatterns overlap in the x-ray image captured by detector 34. According to some embodiments, the distortion of each of the subpatterns and the relationship between the projection of the two subpatterns on the fluoroscopic x-ray image advantageously indicates the aberrations and distortions of fluoroscope 30.

[0161] The processor 50 can be configured or programmed to compare subpatterns in the x-ray image with their ideal shapes and with each other to calibrate the fluoroscope's frame of reference. This calibration procedure can include both calculating the position and orientation of the fluoroscope's 30 optical axis (e.g., calculating the extrinsic rotational and translational parameters and the fluoroscope's 30 internal parameters) and calculating and correcting distortions in the fluoroscopic image of the patient's body. Distortion correction can be performed, for example, based on principles (e.g., spline interpolation methods) described in "Calibration and Gradient-Based Rigid Registration of Fluoroscopic X-ray to CT, for Intra Operative Navigation" by Harel Livyatan, available at https: / / www.cs.huji.ac.il / labs / casmip / wp-content / uploads / 2015 / 08 / msc-thesis-2003-harel-livyatan.pdf, which is incorporated herein by reference.

[0162] To secure the X-ray calibration fixture 38 to the X-ray detector 34, the fixture 38 may include multiple pads 144, 146, 148 that are disposed around a ring 142 and lock onto the periphery of the X-ray detector 34. The pads 144, 146, 148 in this example comprise elastomeric friction pads inserted into a polymer base to firmly grip the X-ray detector 34. The pads 144, 146, 148 are attached to a slide 150, allowing the pads 144, 146, 148 to move radially to engage X-ray detectors of different diameters (e.g., 9 inches or 12 inches). A locking button 154 on the slide 150 can be released to allow the pads 144, 146, 148 to move along the slide 150 and then actuated to secure the pads in a selected position. An adjustment knob 156 advances the pads 144 to securely lock the fixture 38 in place.

[0163] FIG. 8 is a perspective view of a marker 412. The marker 412 includes pins or screws 802 to facilitate accurate and stable placement of the marker 412 in the marker holder 410 on the X-ray calibration fixture 38. According to some implementations, the positional accuracy of the marker 412 is achieved such that the marker angular deviation from its nominal or theoretical position does not exceed 0.2 degrees, 0.18 degrees, 0.16 degrees, 0.15 degrees, or 0.14 degrees in all directions and / or is less than 0.15 mm (root-mean-square value). The marker 412 further includes a reflective element 804 disposed in a plane for reflecting infrared light. The marker 412 may also include a reflective element (e.g., a central reflective element) disposed in a different plane spaced apart from the plane in which the other reflective elements 804 are located. In some embodiments, the marker 412 includes three pins or screws 802. In some embodiments, the marker 412 may include more or less than three pins or screws 802.

[0164] FIG. 9 shows a perspective view of the inner or lower portion or component 900 of the marker 600 shown in FIG. 6. The lower portion or component 900 of the marker 600 includes a reflective material that forms a reflective element created by the pattern of openings in the upper portion or component of the marker, as shown in FIG. 6, thereby facilitating imaging and tracking by the infrared camera or sensor of the head-mounted unit of FIGS. 1, 2A, and 2B. The central well of the lower portion or component is in a second plane that is different from the first plane of the periphery. The central well also includes a reflective material. In various embodiments, the marker 600 is a disposable marker or a reusable marker. As described above, the marker 600 can include a snap-fit attachment mechanism.

[0165] 10A and 10B are schematic rear views of a slide mechanism with attached pad 146, according to one embodiment of the present disclosure. In FIG. 10A, lock button 154 is actuated by inserting lock button 154 into detent 158 of slide 150, thus preventing movement of pad 146. In FIG. 10B, lock button 154 is actuated to allow pad 146 to move radially. Similar or alternative slide and lock mechanisms may be used in other illustrated embodiments.

[0166] FIG. 11 is a schematic diagram showing details of an X-ray calibration fixture 38 according to an embodiment of the present disclosure. The design of fixture 38 is similar to the fixture shown in FIGS. 10A and 10B , with the addition of a safety strap 160 that secures around the back of the X-ray detector 34 and secures the ring of fixture 38 to the X-ray detector 34 to prevent accidental release of fixture 38. Multiple safety straps may be used, attached at various locations around the periphery of fixture 38. Fixture 38 of FIG. 11 may incorporate any of the structural or operational features of the fixtures shown and described in connection with FIGS. 10A and 10B or other previous figures.

[0167] 12A, 12B, and 12C are schematic diagrams of an X-ray calibration fixture 38 with alternative mechanisms 162 for securing the fixture 38 to the X-ray detector 34, according to one embodiment of the present disclosure. FIG. 12A is a top view showing the upper ring 142 of the fixture with three such mechanisms 162 distributed around the periphery of the upper ring 142, while FIGS. 12B and 12C show details of the mechanisms 162 in two different operational configurations for use with X-ray detectors (e.g., C-arms) of different sizes (e.g., diameters).

[0168] Each mechanism 162 includes two pads 164 and 166 attached to a base 168. In Figures 12A and 12B, pad 164 is rotated downward so that pad 164 extends inward on extension arm 169 to engage a smaller diameter camera. In Figure 12C, pad 164 is rotated upward, moving pad 164 out of the way so that pad 166 (without extension arm) will engage a larger diameter camera. The fixture of Figures 12A-12C can incorporate any of the structural or operational features of the fixtures shown and described in connection with Figures 7, 11, or other previous figures.

[0169] 13 is a schematic diagram illustrating a portion of an X-ray calibration fixture 38 that fits over a peripheral lip 170 of an X-ray detector 34, in accordance with one embodiment of the present disclosure. While not all fluoroscopes have such a lip, when lip 170 is present, it can be advantageously used to hold fixture 38 in place. To this end, the X-ray calibration fixture includes a plurality of anchors 172, 174 disposed around ring 142 and that engage lip 170. An adjustment knob 176 can be rotated to move anchors 172 radially so that they engage and lock with lip 170, thus holding the fixture securely in place.

[0170] 14 and 15 are schematic diagrams illustrating an alternative mechanism for radially moving anchor 172 to lock lip 170, according to further embodiments of the present disclosure. In FIG. 14, a linear toggle 178 is pushed inward to lock anchor 172 to lip 170 and pulled outward to release the anchor. In FIG. 15, a spring-loaded latch 180 locks and releases anchor 172.

[0171] 16 is a schematic diagram illustrating an X-ray calibration fixture 38 having a mounting configuration that fits over a peripheral lip 186 of an X-ray detector 34, in accordance with another embodiment of the present disclosure. In this embodiment, the X-ray calibration fixture 38 includes a plurality of anchors 182 that are disposed around a ring 142 and engage with the lip 186. An eccentric locking knob 184 is rotated to press against the surface of the X-ray detector 34, thus holding the fixture securely in place.

[0172] 17 is a schematic diagram illustrating an X-ray calibration fixture 38 having a mounting configuration based on a flexible band 190 that clamps around the periphery of the X-ray detector, according to one embodiment of the present disclosure. Elastomeric pads 192 press against and grip the outer surface of the X-ray detector 34. The pads 192 are attached to slides 194, which allow the pads to move radially to engage X-ray detectors of different diameters. The bands 190 secure the pads 192 in place, ensuring that the fixture 38 remains securely attached to the X-ray detector 34.

[0173] 18 is a schematic diagram illustrating an X-ray calibration fixture 38 having a mounting configuration based on a vertical parallelogram mechanism 196 that is locked by a flexible band 190, in accordance with another embodiment of the present disclosure. In this embodiment, the vertical parallelogram mechanism 196 presses pads 192 inward against the surface of the X-ray detector 34.

[0174] 19 is a schematic diagram illustrating an X-ray calibration fixture 38 having a mounting configuration based on radial locking mechanisms 202, according to one embodiment of the present disclosure. Each locking mechanism 202 includes an elastomeric pad 200 that rotates on a respective arm 204 to engage the outer surface of the X-ray detector, allowing the fixture 38 to be used with cameras of different sizes. A locking knob 206 can be rotated to provide additional adjustment range and to lock the fixture in place.

[0175] FIG. 20 is a schematic diagram illustrating a self-centering mechanism 210 for an X-ray calibration fixture 38, according to one embodiment of the present disclosure. The mechanism 210 includes an outer ring 212 and an inner ring 214, which may be attached to or replace the top ring 142 in the fixture 38. Two knobs 218, connected by a lead screw (not shown), are attached to the outer ring 212 and the inner ring 214, respectively. Moving the knobs 218 together or apart rotates the inner ring 214 relative to the outer ring 212. The pads 216 are attached to the outer ring 212 and rotate inward and outward in response to the relative rotation between the inner ring 214 and the outer ring 212. Thus, manipulating the knobs 218 moves all of the pads 216 together, centering the rings 212 relative to the periphery of the X-ray detector 34 and, therefore, the entire calibration fixture 38.

[0176] This approach can be useful to ensure that fixture 38 is accurately centered regardless of the size of the camera. A flexible band (e.g., as shown in FIG. 18) can be placed around pad 216 to ensure connection of fixture 38 to detector 34.

[0177] In some embodiments, the system comprises various features that exist as a single feature (rather than multiple features). For example, in one embodiment, the system comprises a single camera, a single fixture, a single marker, a single ring, a single anchor, a single pad, etc. In alternative embodiments, multiple features or components are provided.

[0178] Alignment targets or markers As noted above, while the alignment target 46 in the embodiment shown in FIG. 1 includes a radiopaque pattern and an optical pattern, in other embodiments, the alignment target includes only a radiopaque pattern in a predetermined spatial relationship relative to the patient marker. In some embodiments, an optical marker, such as optical marker 42, is positioned in a predetermined spatial relationship relative to the alignment radiopaque pattern, such as bead 40. In some embodiments, the x-ray image captured by fluoroscope 30 includes both the x-ray calibration pattern on jig 38 and the radiopaque pattern of the alignment target. In some embodiments, the alignment target is fixed in position with the patient marker during acquisition of the x-ray image for calibration and alignment purposes, and then removed during surgery so that the patient marker is visible. In other embodiments, the alignment target is fixed to the patient marker at a selected distance from the optical pattern of the patient marker, thereby allowing it to remain in place during surgery. The radiopaque pattern of the alignment target typically includes radiopaque elements, such as beads, arranged in multiple different planes. Examples of alignment targets with such features are shown in the following figures.

[0179] 21A and 21B are schematic diagrams illustrating an alignment target 2800 attached to the back of a patient 24 by pins 2804, according to one embodiment of the present disclosure. The alignment target 2800 includes a pattern of radiopaque elements, such as metal beads 74, which may be arranged in multiple planes: two parallel planes 2802 and 78 that are generally horizontal and axially offset from one another along the plane normal, and two parallel inclined planes 80 and 82 that are also axially offset from one another. In FIGS. 21A and 21B, the pattern of beads 74 in all of the planes 2802, 78, 80, and 82 is identical, but in other embodiments, the patterns in some or all of the planes may be different from one another. Also, while in FIGS. 21A and 21B, the alignment target 2800 includes two pairs of parallel planes, in alternative embodiments, the planes need not be parallel.

[0180] To calculate the transformation between the fluoroscope 30 and the alignment target 2800, the X-ray detector 34 can capture fluoroscopic images from two different angles relative to the patient's body, e.g., one anterior-posterior (AP) image and one lateral (LT) image (FIGS. 21A and 21B show the X-ray detector positioned to capture an LT image). Alternatively, a single fluoroscopic image may be sufficient to calculate the transformation between the fluoroscope 30 and the alignment target 2800. Each image includes both the pattern of beads 2400 in two different planes of the alignment target 2800 and the beads 40 in the pattern on the calibration fixture 38. For example, the LT image includes the pattern in planes 80 and 82, and the AP image includes the pattern in planes 2802 and 78. After calibrating the x-ray detector 34 to compensate for distortions as described above, the processor 50 can compare the positions of the patterns of beads 40 and 2400 in the fluoroscopic image with the known geometric layout of the patterns and thus calculate a geometric transformation between the frame of reference of the fluoroscope 30 and the alignment target 2800. The transformation includes coefficients for 3D translation and rotation between the two frames of reference. The coefficient values are optimized to best fit the relative positions of the patterns of beads 40 and 2400.

[0181] 22 is a schematic cross-sectional view illustrating another configuration of an alignment target 2800 attached to a pin 2804, according to one embodiment of the present disclosure. In this example, the pin 2804 is surgically inserted into the iliac crest 2900 of the patient 24, thus providing a stable platform for the alignment target 2800 that is stationary relative to the patient's anatomy. As in the previous embodiment, the alignment target 2800 includes a pattern of radiopaque elements, such as metal beads 2400, which can be arranged in multiple planes, i.e., two parallel horizontal planes axially offset from each other along the plane normal, as well as two parallel inclined planes also axially offset from each other. According to some embodiments, after the fluoroscopic calibration procedure is completed, the alignment target 2800 is removed from the pin 2804 and an optical patient marker is attached in its place.

[0182] 23 is a schematic cross-sectional view showing an alignment target 2800 attached to a clamp 84, according to an alternative embodiment of the present disclosure. The clamp 84 is secured to the spinous process 3000, which also provides a stable platform. Again, the alignment target 2800 may be removed from the clamp 84 after the fluoroscopic calibration procedure is complete and replaced with an optical patient marker.

[0183] 24 is a schematic diagram of an alignment target 100 according to another embodiment of the present disclosure. As in the previous embodiment, the target 100 comprises radiopaque beads 2400 arranged in a predetermined pattern in a plurality of different planes 102, each of which can be imaged from a different angle relative to the patient's body (e.g., from an AP angle and a LT angle). Mounting holes 104 allow the alignment target 100 to be stably secured to pins or clamps that then hold optical patient markers in a known orientation.

[0184] 25 is a schematic diagram illustrating a multimodal target 110 attached to the back of a patient 24, according to another embodiment of the present disclosure. The target 110 comprises an optical pattern 112 that allows for alignment of an X-ray reference frame with an optical reference frame, and an X-ray pattern 116 of radiopaque elements that serves as an alignment target. Both the optical pattern 112 and the X-ray pattern 116 are fixed to a frame 114 in a predetermined spatial relationship. The frame 114 includes a mount 118 that is fixed to the body surface of the patient 24, for example, using a suitable adhesive.

[0185] During operation, the fluoroscope 30 captures images of the patient 24, including the target 110, from two different angles, as described above. The images include both the x-ray pattern 116 and the calibration pattern on the fixture 38, and are therefore used by the processor 50 ( FIG. 1 ) in both calibrating the fluoroscope and aligning the fluoroscope with the target 110. Images of both the patient markers (not shown) and the optical pattern 112 can be captured by the camera 48. Because the spatial relationship between the optical pattern 112 and the x-ray pattern 116 is known and fixed, a transformation between the x-ray and optical reference frames can be calculated based on the geometry of the target 110 and the images of the optical pattern 112 and patient markers. The x-ray and optical images can then be used by the processor in aligning the fluoroscope with the body of the patient 24. The target 110 can then be removed from the patient 24.

[0186] FIG. 26 is a schematic diagram illustrating a multimodal target 120 attached to the back of a patient 24, according to another embodiment of the present disclosure. The target 120 is connected to the patient's 24 skeleton via a flexible extension arm 122, for example, by a pin or clamp 2804. Alternatively, the extension arm 122 may be attached to the operating table or another stable fixed point near the patient 24. The extension arm 122 has a geometric configuration that can be adjusted and then locked into place, allowing for multiple degrees of freedom in positioning the target 120. The arm 122 may be moved or removed from the surgical field after the alignment procedure is complete. The flexible extension arm mechanism may be incorporated into any of the other alignment target embodiments described herein.

[0187] FIG. 27 is a schematic diagram illustrating a multimodal target 121, according to one embodiment of the present disclosure. The multimodal target 121 includes an optical pattern 126. A patient marker 124 with the optical pattern is attached to the patient via pins 73. The camera 48 can capture images of both the optical markers 124 and 126 to determine the position of the multimodal target 121 relative to the patient marker 124. The optical pattern of the patient marker 124 indicates the position of the pins 73, and the pattern 126 indicates the position of an alignment target 128 that is displaced from the pins 73 by a rigid extension arm 123 (or by another stable fixation point). The extension arm 123 has a geometric configuration that can be adjusted and then locked into place. The alignment target 128 includes multiple X-ray patterns 130, 132, 134 of radiopaque beads 2400 arranged in different planes. According to some embodiments, processor 50 processes the fluoroscopic image including alignment target 128 along with the optical pattern of patient marker 124 and the optical image of optical pattern 126 to align the position and orientation of fluoroscope 30 relative to the body of patient 24. In other embodiments, more than two optical patterns may be used. The multiple x-ray patterns may include two, three, four, or more than four patterns.

[0188] Alignment In image-guided surgery, it is important that the anatomical images displayed to the surgeon or other clinical professional to provide guidance and / or facilitate navigation (e.g., of medical tools and instruments) within the patient's body correspond to the patient's current anatomy (e.g., pose and / or structure). Furthermore, for this type of AR to be clinically useful, it may be important that the overlaid 3D images be properly aligned with the actual anatomical structures within the body. When 3D images are acquired during surgery using an intraoperative 3D medical imaging scanner, such as a CT or MRI scanner, proper alignment is maintained as long as the patient remains motionless. However, in most surgeries, 3D images are acquired before surgery in a different room, and the patient's position on the operating table, for example, is often different from that of the tomographic images. Preoperatively acquired 3D images typically lack fiducials (e.g., fiducial markers) that allow the preoperative 3D images to be aligned with the patient's anatomy during surgery. Furthermore, there may be changes in the relative positions of the vertebrae of the patient's spine between the time the 3D images are acquired and the time of surgery. Such changes may be due to changes in the patient's posture, the insertion of an implant, or any other reason. For spine surgery, surgeons typically use a fluoroscope in the operating room to acquire 2D images during surgery and use these 2D images for intraoperative guidance while viewing pre-acquired medical images (e.g., cross-sectional or volumetric images) of the spine offline.

[0189] The embodiments of the present disclosure described herein provide methods, systems, and computer software products that can be used to register a pre-acquired 3D medical image (e.g., a tomographic or MR image) with an intraoperative 2D fluoroscopic image. In the disclosed methods, systems, and computer software products, the 3D image is segmented into multiple 3D segments, each including a respective one of the vertebrae or other bony parts. Each of these 3D segments is registered with a respective vertebra or other bony part in the fluoroscopic image. Specifically, the respective position and orientation of each 3D segment matches the respective vertebra or other bony part in the fluoroscopic image and can therefore be adjusted to account for changes in the relative position of the vertebrae or other bony part (e.g., due to changes in the patient's position on the operating table relative to their position in the 3D image). In the disclosed embodiments, two fluoroscopic images captured from different angles are used together in this registration process; alternatively, a larger number of fluoroscopic images (e.g., three, four, or more than four images) can be used.

[0190] In the context of spine surgery, once the alignment process is complete, an image of the spine including the aligned 3D segments is presented on a display (e.g., to facilitate AR-assisted navigation), for example, by overlaying an AR image of the aligned 3D segments onto the patient's back to generate an AR view. In some embodiments, to ensure proper alignment between the AR image and the patient's body, the reference frame of the 2D fluoroscopic image is calibrated to the patient's body (e.g., a portion of the patient's back corresponding to the target treatment area of the spine), for example, using calibration markers described below. This calibrated reference frame can then be applied to the 3D image to properly align the vertebrae or other bony portions in the aligned 3D segments with the patient's spine.

[0191] Tomographic and fluoroscopic image registration For AR image display, it may be important that the reference frame of the images captured by the fluoroscope 30 be calibrated relative to the physical reference frame of the patient's 24 body. This calibration has two aspects: (1) correcting distortions in the images captured by the fluoroscope 30 itself, and (2) aligning the fluoroscope 30 to the patient's body. For these purposes, according to the exemplary system shown in FIG. 1 , a calibration fixture (e.g., a ring adapter) 38 is attached to the X-ray detector, in this case, the X-ray detector 34. In the illustrated embodiment, the calibration fixture (e.g., ring) comprises an array of radiopaque beads 40 in a predetermined pattern and optical markers 42 in a known position and orientation relative to the pattern of beads 40. This bead pattern appears in fluoroscopic images captured by the X-ray detector 34. In some embodiments, a processor 50 receives and processes these images to correct for distortions in the X-ray images and to determine the position and orientation of the optical axis of the fluoroscope 30.

[0192] According to some embodiments, the optical markers 42 are used in conjunction with optical markers on the body of the patient 24 when aligning the optical axis of the fluoroscope 30 with the body. For example, the surgeon 22 can attach the markers 44 to the patient's spine using appropriate bone clamps or percutaneous pins. Additionally or alternatively, the surgeon 22 can secure the markers 46 to a surface of the patient's body (e.g., the skin). The markers 46 may be secured via a self-adhesive backing on the markers 46 or via a separate adhesive (e.g., adhesive tape or glue). In some embodiments, the camera 48 (e.g., an infrared camera or other optical camera) captures an image that includes both the optical markers 42 on the calibration ring 38 and the markers 44 or 46 attached to the patient 24. 1 and 2A-2B are mounted on head-mounted AR display unit 28, these images may alternatively be captured by a suitable camera mounted elsewhere on the head or body of surgeon 22, or mounted (e.g., in a stationary manner) elsewhere in the operating room or diagnostic imaging suite. In some embodiments, processor 50 processes the images to calculate the position and orientation of fluoroscope 30 relative to the patient's body, and thus calibrate the fluoroscope frame of reference relative to the body's frame of reference.

[0193] This calibration process is described herein as well as descriptions of other configurations of calibration rings and optical markers that can be used in place of the configuration shown by way of example in Figure 1. Alternatively, other devices and methods can be used to calibrate the fluoroscopic frame of reference relative to the patient's body for AR display of 3D medical image data (e.g., tomographic image data), and other devices and methods are considered within the scope of this disclosure.

[0194] In addition to receiving two-dimensional x-ray images from the fluoroscope 30, the processor 50 is configured to receive three-dimensional medical images of the patient 24, such as CT or MRI images, typically acquired before surgery or other medical intervention, and store these three-dimensional images in the memory 52. The processor 50 is configured to segment the three-dimensional images and align the three-dimensional segments with respective vertebrae in the two-dimensional fluoroscopic images. The processor 50 is then configured to present an image of the spine, including the aligned 3D segments, on the head-mounted AR display unit 28 so that the vertebrae in the 3D images are aligned with the actual vertebrae in the patient's spine. Details of this process are described with reference to the following figures. Alternatively, the aligned 3D segments may be presented on a different type of display, for example, an AR display mounted on the patient 24 or operating table 26 above the surgical site or another local and / or remote display device. Additionally or alternatively, the aligned 3D segments may be presented on a non-AR display, such as the display of a workstation or handheld computer.

[0195] According to some embodiments, the processor 50 comprises a general-purpose computer processor, which is programmed with software to perform the calibration, alignment, and display functions described herein. This software (e.g., executable program instructions) may be stored on a tangible, non-transitory computer-readable medium, such as an optical, magnetic, or electronic memory medium. Additionally or alternatively, at least some of the functions of the processor 50 may be performed using dedicated computing hardware, such as a graphics processing unit (GPU). The processor 50 may include one or more processors. The processor 50 may be located in a workstation and / or in the head-mounted AR display unit 28.

[0196] FIG. 28 is a flowchart that schematically illustrates a method for generating an AR display based on the registration of three-dimensional and two-dimensional images, according to one embodiment of the present disclosure. For concreteness and clarity, the method is described herein with reference to system 20 ( FIG. 1 ), assuming that processor 50 receives a 3D CT image of the spine of patient 24 and two X-ray images from fluoroscope 30 captured by X-ray detector 34 at two different angles. Alternatively, the method may be applied, mutatis mutandis, using other types of medical images, such as MRI images or other cross-sectional images processed to segment bone from soft tissue, as well as using a greater or fewer number of 2D X-ray images. The principles of this method may also be applied in generating images of other bones within a patient's skeleton (e.g., hip joints, pelvic bones, leg bones, arm bones, ankle bones, foot bones, shoulder bones, skull, oral and maxillofacial bones, sacroiliac joints, etc.). With respect to the spine, the vertebrae may include lumbar, sacral, cervical, and / or thoracic vertebrae, or other bony structures, portions, elements, or components.

[0197] As described above and shown in FIG. 28 , both the three-dimensional CT image and the two-dimensional X-ray image are pre-processed (at blocks 3500 and 3504) to enable image-to-image registration and display (e.g., on display 60) of vertebrae or other bony structures from the CT image aligned with the patient's spine. As noted above, processor 50, upon execution of the computer-readable program instructions, is configured to calibrate the X-ray image online at block 3506 to correct for distortion and to align fluoroscope 30 with the body of patient 24, as described herein. As further described below with reference to FIG. 35 , the CT image is segmented at step 3502 into a plurality of 3D segments, each of which includes a respective one of the vertebrae and / or sacrum and / or ilium and / or other bony structures. The 3D segments are then registered with the calibrated 2D fluoroscopic image (block 3508), as described below with reference to FIGS. 34 and 37 .

[0198] According to some embodiments, to present the AR images on the display 60 in alignment with the patient's anatomy, the processor 50 aligns the display unit 28 (or correspondingly, the display unit 70) with the patient's body using images of the markers 44 and / or 46, as described above (block 3512). Surgical tools (not shown), such as drills, introducers, cannulas, curettes, stylets, screwdrivers, and insertion instruments, may be provided with similar types of markers (directly or indirectly) to allow the processor 50 to calibrate and align their positions (block 3510), so that virtual images of the tools can also be incorporated into the AR display. With all elements calibrated and aligned in this manner, the surgeon 22 can perform the desired surgical or other medical procedure with the aid of the AR images of the patient's vertebrae or other bony structures or portions presented on the display 60 (block 3514). The method of FIG. 28 applies, mutatis mutandis, when a non-AR system is used. When a non-AR system is used, the step of aligning the display with the body should be omitted.

[0199] 29 is a schematic diagram of a calibration process, according to one embodiment of the present disclosure, in which parametric information about the imaging system is calculated to determine a 3D to 2D mapping (e.g., mapping or finding the correspondence of each 3D voxel from a preoperatively acquired 3D scan to a 2D pixel from an intraoperatively acquired 2D fluoroscopic image, as shown by 3600 in FIG. 29). To map each 3D voxel to a 2D pixel, the intrinsic and extrinsic parameters of the fluoroscope are calculated as P1...P n n points P1...P with known coordinates and known positions in the image n It is estimated using a mathematical relationship that takes into account p=K[RT]P w

[0200] K[RT] represents the transformation matrix that maps each 3D voxel to a 2D pixel. K represents the intrinsic parameters of the fluoroscope or other device, [RT] represents the extrinsic parameters, R is the rotation matrix, and T is the translation matrix.

number

[0201] The calibration process involves finding the transformation matrix K[R,T] in the x-ray calibration fixture or ring, coordinate system, and transforming to the fixture or ring marker's coordinate system 3602 (e.g., marker 412, marker 600-C attached to fixture 38, marker[R,T]), and then transforming to the patient's coordinate system 3604 (e.g., patient[R,T]). The calibration process results in obtaining the camera's (e.g., fluoroscope's) transformation matrix K[R,T] in the patient marker's coordinate system.

[0202] According to some embodiments, the goal of the X-ray calibration process is to find the intrinsic parameters (e.g., focal length and principal point) of a fluoroscope or other imaging device and its extrinsic parameters (translation and orientation) relative to a patient marker coordinate system. To calculate the intrinsic parameters, a dual-layer calibration fixture with plates having different fiducial elements or bead patterns on each layer is attached to the detector of the C-arm or other fluoroscope or imaging device. First, the beads on the top layer appearing in the image are detected, and then each bead is associated with its pattern. The system now contains a batch of 3D-2D point correspondences (3D points in the calibration fixture or C-arm coordinate system and 2D points in the image) and can calculate the intrinsic parameters.

[0203] According to some embodiments, there are several different options for calculating the extrinsic parameters. One option is to connect the registration marker on the clamp to the patient marker with a fixed offset. The registration marker can be captured and displayed on the X-ray image, and in a similar process (e.g., detection and correlation), the translation and orientation of the X-ray detector can be calculated in the registration marker's coordinate system. The system can then mechanically transform from the registration marker's coordinate system to the patient marker's coordinate system based on the known offset.

[0204] A second option is to connect the optical markers on the calibration jig 38 to the jig's coordinate system with a fixed offset, as shown in Figures 1-7. After capturing one or more X-ray or fluoroscopic images, a head-mounted unit (e.g., an infrared camera or other imaging device) can capture images of the patient and optical markers. According to some embodiments, marker positions must be captured for each X-ray or fluoroscopic image before proceeding to the next X-ray. These images can be used to calculate the transformation between the optical markers and the patient markers. Because the position and orientation of the optical markers in the jig's coordinate system are known, the transformation from the jig to the patient markers is known. A third option is to place the alignment markers of the first option on the patient's body without connecting them to the clamps. The workflow for calibration may be similar to the first option, but the transformation between the alignment marker's coordinate system and the patient marker's coordinate system may be calculated using the optical images, as in the second option.

[0205] FIG. 30 is a flowchart that outlines a method for calibration, according to one embodiment. One or more images including radiopaque beads 40 are first acquired. In step 3700, the beads 40 are detected in the images. In step 3702, grid correlation is performed, where images of beads 40 located on the lower bead plate 408 undergo a process that results in correlating each bead on each line of the bead pattern to correct indices. In step 3704, marker correlation is performed, where images of beads 40 located on the upper bead plate 406 undergo a process that results in correlating each bead on each line of the bead pattern to the correct world point. Intrinsic and extrinsic parameters of the fluoroscope 30 (e.g., X-ray detector 34) are calculated in step 3706. Step 3708 indicates that the upper bead plate (or marker beads) are utilized for X-ray image distortion correction as described herein.

[0206] 31A-31B are flowcharts and corresponding schematic diagrams of steps of the flowchart according to one embodiment of the present disclosure. FIG. 31A provides an overview of a process (e.g., a sub-process, method, or algorithm that may be stored in memory 52 and executed by processor 50) for bead 40 detection. In step 3700a, as shown in FIG. 31B, an image 3800 including grid and marker beads 40 is acquired. Specific information about the bead plates 406, 408 is known prior to capturing the image of these bead plates 406, 408. This information includes the expected bead 40 radius, the spacing between beads 40, and the expected number of beads 40. Using this information, in step 3700b, a bead template 3808 is created, which is used to create a correlation image 3810 by moving the bead template 3808 through every pixel of the original image and finding local maxima. For example, moving the bead template 3808 over a section of the original image that includes beads (e.g., bead image 3806 in FIG. 31B) will result in a higher correlation value than moving the bead template over a section of the original image that does not include beads. Following this template matching, the detected beads 40 are divided into groups according to size in step 3700c to distinguish beads on the lower bead plate 408 from beads on the upper bead plate 406. Subpixel centers are then determined for each of the beads in step 3700d.

[0207] 32A-32H are flowcharts and corresponding schematic diagrams of steps in a flowchart illustrating grid association, according to one embodiment. Following bead detection, grid association occurs. In step 3702a, duplicate beads are removed. In one embodiment, the duplicate bead removal step of 3702a includes analyzing all detected beads and further examining beads located within a certain distance of other beads. This step can reveal whether adjacent beads are of different types (and thus belong to different bead plates, as shown in FIG. 32B, which shows beads 3802 and 3804) or the same type (and thus likely duplicate beads). In step 3702b, a direction vector is calculated for each pair of beads (e.g., beads detected for the lower bead plate 408) separated approximately by the expected grid bead separation distance. This is shown in FIG. 32C. The grid vectors are then used in step 3702c to group beads into unique lines. In one embodiment, as shown in FIG. 32D, grouping can be performed by starting with one grid bead and sorting the beads through a process of searching for another bead in the grid direction using a grid vector (e.g., 3902 shows the identification of a first grid bead, 3904 shows the grid vector being used to identify a second grid bead along a line, and so on for 3906 and 3908). Outliers are removed in step 3702d. In one embodiment, as shown in FIG. 32E, the distance between the determined bead lines 3910 can be calculated and used to remove bead lines based on the expected distance. In step 3702e, each bead on each line is associated with the correct index. As shown in FIG. 32F, indexing can be performed through sub-process 3912, where one bead is selected to have index (0,0) and the remaining beads are indexed relative to this initial bead.In some embodiments, if a row of beads appears to be missing where expected based on prior knowledge of the imaged bead plate, the beads can be indexed to account for the missing beads, as shown in sub-process 3914 of Figure 32G. The bead index can be shifted in sub-process 3916 of Figure 32H to remove negative indexes.

[0208] 33A-33F are flowcharts and corresponding schematic diagrams of steps of a flowchart illustrating marker association according to one embodiment of the present disclosure. Grid association is followed by marker association. In step 3704a, overlapping beads are removed in the same or similar manner as in step 3702a for grid association. In step 3704b, beads are grouped into unique lines according to grid angles (or direction vectors), as shown, for example, by zoomed-in subfigures 4000, 4002, 4004, and 4006 in FIG. 33B. In some embodiments, if no markers are detected, grouping beads into unique lines can accommodate those undetected marker beads, as shown by subfigures 4004 and 4006 in FIG. 33B. As shown in FIG. 33C, in step 3704c, a label is determined for each line based on the spacing between the unique lines and the distance between each pair of beads on each unique line. Examples of single lines or groups of beads are shown in subfigures 4008, 4010, and 4012. The labeled lines are fitted to a known pattern in step 3704d (FIG. 33D), and each bead on each line is associated with the correct world point in step 3704e. As shown in FIG. 33E, each marker bead or top plate bead on each line is associated with the correct world point by converting from pixel (e.g., bead position as pixel 4014 in FIG. 33E) to world point [x, y, 0] (e.g., bead position as world point 4016 in FIG. 33E). Similarly, bottom plate bead 3804 can also be associated with the correct world point, as shown schematically in FIG. 33F. An example of a single line or grouping is shown in subfigure 4018.

[0209] FIG. 34 is a flowchart that schematically illustrates details of a method for registering two-dimensional and three-dimensional images, according to one embodiment of the present disclosure. While described with reference to vertebrae of the spine, the method can be used for other bones, joints, or tissues as well. To begin the process of registering a 3D image segment with a 2D image of the spine, processor 50 receives an initial input (block 4100) that associates one of the vertebrae in the 3D image segment with the position of the same vertebra in the two 2D images. For example, a user of system 20 can use a cursor to mark the location of a selected 3D vertebra or other bony structure on the 2D image. Furthermore, according to some implementations, processor 50 uses external cues to make an initial estimate of the orientation of the patient's 24's spine (block 4102). For example, the position of marker 44 or 46 relative to the patient's skeleton indicates the Z direction (e.g., the sagittal axis), and the position of x-ray source 32 and detector 34 indicates the Y direction (e.g., the longitudinal axis). Based on the initial input and the estimated orientation of the spine, processor 50 may associate each 3D image segment with a corresponding vertebra and / or other bony structure in each 2D image (block 4104). Processor 50 may also estimate and utilize known ranges of movement of the vertebrae and / or other bony structures relative to each other in estimating the registration parameters.

[0210] To accurately align vertebrae in the 3D image segments with associated vertebrae in the 2D images, processor 50 generates (e.g., calculates) digital reconstruction radiographs (DRRs) or other simulated X-ray images based on the 3D images of the vertebrae over a range of vertebral motion and rotation about the estimated axis of the 2D images relative to the spine (block 4106). In some embodiments, the intensity of each pixel in a given DRR is calculated by calculating the cumulative radiodensity of voxels along the path of a ray between the X-ray source and the pixel. In some implementations, the DRRs can be generated using Siddon's algorithm.

[0211] In the illustrated embodiment, the processor 50 applies an optimization process to find the orientation of each 3D vertebra or other bony structure relative to the 2D image by comparing the gradient of pixel values in the DRR P1(i,j) with the actual gradient of pixel values in the 2D X-ray image P2(i,j) (block 4108). The optimization uses mask functions M1(i,j), M2(i,j) for the DRR and the X-ray image, respectively, to mitigate the effects of artifacts such as foreign objects in the X-ray image (e.g., for pixels containing beads of the calibration fixture 38, the mask value may be set to 0). For purposes of optimization, the processor 50 calculates a similarity, or metric Sim(P1,P2), between each DRR and the corresponding X-ray image, for example, using the following equation:

number

[0212] The similarity is averaged over all pairs of (X-ray image, DRR). The best orientation and position belongs to the pair with the highest average similarity, or metric.

[0213] Various search strategies can be used to find the optimal orientation and position while avoiding the excessive computational burden of an exhaustive search. For example, the orientation and position search space can be divided into smaller regions and the similarity can be calculated for one sample in each region. The processor 50 can then perform a fine-grained search only within the region with the highest similarity measure. The search can be performed initially at a coarse CT resolution (e.g., 1 mm, 1.5 mm, 0.5 mm, 2 mm, or other value), and then refined using a finer CT resolution (e.g., 0.3 mm, 0.2 mm, 0.15 mm, 0.1 mm, 0.05 mm, or other value). In some embodiments, the search space can be sampled to avoid the optimization getting stuck in a local minimum. After the first vertebra is registered, its neighbors can be registered using the registration of the first vertebra as an initial guess. Other bony structures can be registered in a similar manner.

[0214] Once the optimal positions and orientations of all vertebrae or other bony structures within the 3D image segment have been found in this manner, processor 50 uses the results to reconstruct a complete 3D model of the spine from the individual 3D vertebrae and / or other bony structures (block 4110). According to some embodiments, the positions and orientations of the vertebrae and / or other bony structures in this 3D model match the actual spine (e.g., the actual pose of the spine) of patient 24 on operating table 26. In block 4112, processor 50 may then display the three-dimensional model (e.g., generate output for display to facilitate navigation of medical tools during the procedure). If AR is utilized, processor 50 may then use the relative position and orientation of head-mounted AR display unit 28 or head-mounted AR display unit 70 with respect to patient 24 to calculate views of the vertebrae and / or other bony structures that are overlaid on the actual anatomical structure of patient 24 and projected onto display 60 in the appropriate position and orientation.

[0215] 35 is a schematic illustration of a segmented 3D image of vertebrae, according to one embodiment of the present disclosure. To arrive at this image, the 3D image of the spine in a CT scan is segmented into individual 3D vertebrae. This segmentation operation can advantageously be performed by deep learning techniques using one or more trained convolutional neural networks (CNNs). The sacrum and ilium may similarly be segmented in this manner using a neural network separate from or the same as that used for the vertebrae.

[0216] In some embodiments, a combination of three networks is used for this purpose: The networks are fully convolutional networks (e.g., based on the U-Net architecture).

[0217] The first network can receive as input a CT image or other tomographic or volumetric measurement of the spine or portion of the spine that has been resampled to a coarse resolution relative to the resolution of the CT image (e.g., the original unprocessed CT image). For example, the CT image may have a resolution of 8 mm per voxel (e.g., 8*8*8 mm). 3 ) resolution. According to some embodiments, the coarse resolution may be in the range of 5-15 mm. According to some embodiments, the coarse resolution may be in the range of 6-10 mm. In some cases, the resolution of a CT image may be as low as 2 mm. Such a coarse resolution may allow the entire image to be fed to the network (e.g., as one block), conserving computing resources. The output of the first network may be two values for each voxel: one indicating whether the voxel contains a vertebral portion, and a second indicating whether the voxel contains a portion of the sacrum or ilium. The goal is to define a region of interest within the image. According to some embodiments, processing images of smaller size advantageously allows for the use of fewer computing resources and faster processing. Furthermore, identifying the region of interest may prevent errors, such as identifying other bony structures adjacent to the spine (e.g., the shoulder) as a region of interest (e.g., as the spine).

[0218] The second network may receive as input the CT image regions identified by the first network as regions of the sacrum and ilium resampled to a finer resolution relative to the resolution used by the first network (e.g., 1 mm). In some embodiments, the finer resolution may be between 0.3 and 1.5 mm. In some embodiments, the finer resolution is finer relative to the resolution of the CT image. In some embodiments, the finer resolution may be substantially equal to the resolution of the CT image. In some embodiments, the finer resolution may be 50% to greater than 50% of the resolution of the CT image. The resampled relevant image portions may then be divided into patches of a predetermined voxel size. The network is fed one patch at a time. The output may include two values per voxel: one indicating whether a portion of the sacrum is contained in the voxel, and the other indicating whether a portion of the ilium is contained in the voxel. According to some implementations, the goal is to segment the sacrum and ilium.

[0219] A third network can then be applied to the region of interest in the CT image identified by the first network as containing a vertebra. The third network can receive two inputs: a patch of the portion of the CT image resampled to a finer resolution (e.g., 1 mm). This finer resolution can be equal to or substantially equal to the finer resolution used in the application of the second network, and the same patch contains information about previously segmented vertebrae. In some embodiments, the first such patch can contain information about the segmented sacrum and ilium. The output of the network can be a patch containing a value for each voxel indicating whether the voxel contains part of a vertebra following the previously segmented vertebra (or the first ilium and sacrum) identified in the input patch (the second input described above). That is, the output is the segmentation of the next adjacent (in a given direction) vertebra. In some implementations, the network is trained to identify the next vertebra in the first patch based on the second patch containing information identifying the previously segmented vertebra, and to reposition the patch in the resampled CT image along a predetermined spinal direction (inferior-superior or vice versa) until the entire next vertebra is identified and centered within the patch. In this example, an inferior-superior direction is used, starting with the ilium and sacrum.

[0220] The orientation of the CT scan (e.g., patient body orientation: legs-head) can be determined by the DICOM standard data provided for the scan. Alternatively, it may be determined by identifying the sacrum and / or ilium. The operation of the third network can terminate when at least one of the following occurs: the entire region of interest has been processed or 28 vertebrae have been identified. If the sacrum or ilium is not identified in the CT image by the first network (e.g., if the CT scan does not include the sacrum and ilium), the second network may not be applied. The output of the network can be converted to binary values (e.g., "0" and "1"), and a mask image of the CT image can be generated based on these values to correspondingly indicate the segmented vertebrae and the ilium and sacrum, if present in the CT image. According to some embodiments, training of the network is supervised. Thus, spine images are labeled for each network training and serve as ground truth. In some embodiments, augmentation can be used (e.g., by applying a transformation to the training images to increase the number of distinct training images). According to some embodiments, the training CT images used to train the first network include labeling voxels within the intervertebral spaces as vertebrae (vertebral "blurring") to facilitate identification and segmentation of vertebral regions.

[0221] According to some embodiments, the processor 50 divides the segmented spine into 3D image segments (e.g., according to the methods described above), each containing a single vertebra. This segmentation step calculates and crops a 3D bounding box around each vertebra, as illustrated by the three diagrams shown in FIG. 35. Portions of soft tissue and adjacent vertebrae remaining within the bounding box are removed. All voxels within the image segment that do not belong to the vertebral bone are assumed to belong to soft tissue and are set to 0 HU (Hounsfield Units). The resulting separated 3D images of the vertebrae are then used to align the 3D and 2D images. The segmentation output may be vectors of small CT image portions (e.g., one for each vertebra, sacrum, and ilium).

[0222] 36A-36C and 38A-40B show screenshots of an exemplary embodiment of a GUI and user workflow for registering a CT image of a patient's spine and two fluoroscopic images of the patient's spine captured from two different angles (e.g., anterior-posterior (AP) and lateral or oblique). The described concepts can be implemented for images of other bones, joints, or other tissues (e.g., other non-spine orthopedic sites, skull, ear, nose, throat, mouth, shoulder, hip, knee, arm, foot, ankle).

[0223] Reference is now made to FIGS. 36A-36C, which illustrate screenshots of segmentation displays in the GUI. As described above, a CT image may be automatically segmented. The automatically segmented CT image can then be displayed to the user in a segmentation display (e.g., via segmented image views or images 4300 and 4302). The automatically segmented CT image can be displayed in various views, including CT slice views, such as sagittal view 4300 or coronal view, view 4302, and / or 3D views, such as x-ray or "x-ray-like" views. In some embodiments, the CT slice views relate to different anatomical planes (e.g., coronal, sagittal, and axial). Sagittal view 4300 shows the entire spine segmentation, while in some embodiments, only a portion of the spine is segmented. Slide 4310 or other GUI input element can allow the user to slide, toggle, or otherwise transition between different slices or segments, which can be activated when a slice view is displayed. GUI input element 4304 can allow the user to switch between various portions or segments of a vertebra or other bone segment. In some embodiments, one or more GUI elements may be included to allow a user to adjust the visualization of the CT slices, such as sliders 4330 (e.g., window center and window level). The 3D view 4302 may display a 3D model or 3D rendering of the spine 4306 generated from a CT scan. In some embodiments, the 3D model or rendering of the 3D view may be manipulated by the user (e.g., rotated in various or all directions). An X-ray or "X-ray-like" view may be generated from the CT scan (e.g., via a DRR). In some embodiments, an X-ray view or virtual X-ray view may be generated to substantially match or resemble the perspective of the captured fluoroscopic image. In some embodiments, the virtual X-ray view is in a predetermined fixed view (e.g., AP and lateral or oblique lateral).In some embodiments, one or more GUI elements may be generated or provided to allow a user to adjust the threshold at which the DRR image is generated and / or to allow a user to adjust the visualization of the DRR image display (e.g., via adjusting pixel intensity or opacity values). In some embodiments, the user may select the view to be displayed (e.g., via a drop-down menu 4305). The segmentation GUI display may optionally include one or more windows, including two view windows, for simultaneously displaying one or more views according to user selection, and as shown in FIGS. 36A and 36C. In some embodiments, spine segmentation may be further visualized by coloring different segmented vertebrae with different colors. Spinal segmentation may additionally or alternatively be visualized by different shading or hatching patterns. In some embodiments, and as shown in FIGS. 36A-36B, different segmented vertebrae may be automatically labeled (e.g., by activating GUI element 4340), which may be a toggle switch or other GUI element. In some embodiments, the segmented CT image may be manually segmented, or both manual and automatic segmentation may be possible or performed. In some embodiments, a GUI element (e.g., slider 4345) may be generated to allow a user to control label transparency (e.g., to display labels without obscuring essential image information). In some embodiments, manual editing of the automatic segmentation may be allowed (e.g., by activating GUI element 4350 (e.g., a toggle switch)). At the end of the segmentation phase of CT-Fluoro image registration, a registration procedure, phase, or step may be initiated (e.g., by pressing the command button "Start Procedure" or other GUI element).

[0224] FIG. 37 is a schematic diagram of the 2D / 3D registration process, according to one embodiment of the present disclosure. For purposes of illustration, this diagram shows a portion of the spine, but the same principles apply to the registration of each of the vertebrae or other bones. "Image 1" and "Image 2" represent 2D fluoroscopic views of the bone captured by the X-ray detector 34 at two different angles (Viewpoint 1 and Viewpoint 2), so that each 2D image represents a projection of the 3D shape of the bone onto a different plane. To find the actual position and orientation of the bone (whether it be a femur, vertebra, or other bone), the processor 50 performs a coordinate transformation (T CT )1 and (T CT )2. The coordinate transformation may be calculated before the registration.

[0225] 38A-38C are screenshots of an exemplary embodiment of a GUI display for registering a fluoroscopic image with a segmented CT image. The segmented display includes a FluoroView window 4400 and a CT View window 4410. In the FluoroView window 4400, the captured fluoroscopic images may be displayed; specifically, the first and second fluoroscopic images may be displayed in two views, designated "1" and "2," respectively. In some embodiments, the CT window 4410 displays a virtual X-ray image from a viewpoint or viewing angle corresponding to the viewpoint or viewing angle from which the currently displayed fluoroscopic image was captured. In FIGS. 38A-38B, a first fluoroscopic image is displayed in window 4400, and a first corresponding virtual X-ray image is displayed in window 4410. In FIG. 38C, a second fluoroscopic image is displayed in window 4400, and a second corresponding virtual X-ray image is displayed in window 4410. In some embodiments, window 4410 can include additional CT views, such as sagittal and coronal slice views, and the user can switch between different views, as shown in FIGS. 38A-38C. In some embodiments, the segmented portions (e.g., vertebrae) of the virtual x-ray image can be labeled. The user can then enter an initial guess or indicate the matching vertebrae to begin automatic segmentation. The user can indicate the segmented vertebrae (e.g., via blue highlighting or other coloring 4420 of vertebra L3) in the virtual x-ray image (or any other CT-based image). The indication can be performed using a user input device (e.g., a keyboard, mouse, control pad, joystick, touchscreen user interface, etc.). The user can then place a mark, such as a target 4450, on the matching vertebra in one of the fluoroscopic images, as shown in FIG. 38B.The second fluoroscopic image can then be displayed with a mark (e.g., line 4470, which may be an epipolar line calculated from the mark on the first fluoroscopic image) indicating where the selected vertebra (e.g., L3) is located in the second fluoroscopic image, as shown in FIG. 38C. The user can then place another mark or representation, such as a blue highlighting or coloring 4460 of the shape of the selected vertebra, along line 4470 (e.g., the epipolar line) to mark the corresponding vertebra in the fluoroscopic image (e.g., select the vertebra location), as shown in FIG. 38C. In some embodiments, the alignment representation can include a GUI element, such as a slider 4430, to allow the user to adjust the threshold used in generating the virtual X-ray image. In some embodiments, GUI elements (e.g., window center and window level) may be included to allow the user to adjust the image display or visualization. The user can optionally interact with the user interface to rotate the vertebra for better alignment. In some embodiments, a user can mask a fluoroscopic image (e.g., by activating GUI element 4480). For example, a user can mask portions of an image that contain noise or data that may interfere with the registration process. Thus, a user can mask, for example, metal elements such as screws in a fluoroscopic image displayed in window 4400, as shown.

[0226] Reference is now made to FIGS. 39A-39B, which are screenshots of a GUI display showing different views of the aligned vertebra (L3) in a segmented CT image overlaid on a fluoroscopic image (or vice versa). FIGS. 39A-39B display an augmented or composite image of the aligned vertebra, consisting of a fluoroscopic image of the aligned vertebra and a corresponding virtual X-ray image generated by assuming the perspective of the fluoroscopic image. According to some embodiments, such a composite image can visualize the alignment and advantageously allow for relatively easy evaluation, or at least aid in the assessment of the alignment. The X-ray virtual image can be blended with the aligned segmented vertebra. In some implementations, each of the aligned segmented vertebrae may be positioned and rotated according to the alignment output, meaning that the position and orientation relationship of the vertebrae is not required as on the CT image. In some embodiments, a GUI element (e.g., a sliding element) can be included that allows the user to adjust the CT transparency or opacity in either continuous or discrete increments. In some embodiments, when the CT transparency is adjusted to a minimum value, the composite image displays only or mostly the X-ray virtual image, as shown in FIGS. 4500 and 4510. In some embodiments, when the CT transparency is adjusted to a maximum value, the composite image displays only or mostly the fluoroscopic image, as shown in FIGS. 4520 and 4525. Image intensity and contrast levels can also be adjusted by interaction with GUI user input elements, such as slide bars or adjustment elements. In some embodiments, the user can select (e.g., via GUI element 4520, which can be a toggle button or switch) a display in flash mode in which the images (e.g., virtual X-ray and fluoroscopic images) of FIGS. 39A and 39B are presented sequentially and repeatedly in a flash manner (e.g., during which each type of image is presented at very short, predetermined time intervals). Such a display can facilitate review of the alignment. In some embodiments, the user may be required to approve each vertebra alignment.As shown in FIGS. 39A-39B , in some embodiments, a user can confirm vertebral alignment by pressing a green button 4530 or other GUI user input element, or reject vertebral alignment by pressing a red button 4535 or other different GUI user input element. Elements such as CT transparency adjustment and flash display mode can facilitate the alignment confirmation process. In some embodiments, landmarks may be added by the user to a composite image or an augmented image (e.g., visualized to display only or nearly a fluoroscopic image or virtual x-ray image). The user can then change the visualization of the image (e.g., by adjusting the CT transparency level of the image to receive a virtual x-ray image or fluoroscopic image, respectively), receive a second, different view of the composite or augmented image, and review the landmarks in the second view of the image or in flash mode. Landmarks can, for example, indicate identified and / or interesting anatomical structures. Landmarks can facilitate the alignment confirmation process.

[0227] 40A and 40B provide additional examples of GUI displays for registering a fluoroscopic image with a segmented CT image, with FIG. 40B showing different views 4604, 4606 of registered vertebra L4 in a segmented CT image overlaid on a fluoroscopic image. The GUI displays of FIGS. 40A and 40B can include similar operational features and elements as the previously described GUI displays.

[0228] Magnetic resonance and fluoroscopic image registration MR images contain a wealth of data about soft tissues, such as the location of muscles and nerves, which can be beneficial to surgeons when performing image-guided surgery. However, traditional MR images do not clearly show bone (as opposed to, for example, X-ray-based CT images). Therefore, it can be difficult to locate soft tissues within an MR image with sufficient accuracy in an AR display to allow surgeons to visualize both bone and soft tissue together.

[0229] In addition to pre-operative or previously acquired CT images, the embodiments of the present disclosure described herein provide methods, systems, and computer software products that can be used to align previously acquired MR images with actual bones in a target region of a patient's body and fuse MR image data containing soft tissue segments with a 3D image of the bone segments on a display. The calibration and alignment techniques described herein with respect to CT can be similarly applied to MR image data (e.g., where similarity measures or metrics may be different and segmentation neural networks may be trained differently). In these embodiments, a 3D MR image of a target region containing one or more bones where surgery will be performed is processed to generate a segmented 3D image containing both the bone segments and the soft tissue adjacent to the bone segments. This segmented 3D image is then aligned with the patient's body by aligning the bone segments in the segmented 3D image with one or more bones in the target region of the body. The aligned segmented 3D image is presented on a display, for example, with an AR image containing the bone segments and soft tissue overlaid on the target region of the body.

[0230] Although the embodiments described below relate specifically to the registration and presentation of images of vertebrae for purposes of spinal surgery, the principles of the present embodiments may be applied mutatis mutandis to other bones and target regions of the body, such as the shoulder, hip, knee, arm, leg, head, brain, skull, jaw, ankle, foot, or target region containing an organ. Additionally, the principles of the present disclosure may be applied mutatis mutandis to soft tissue surgery.

[0231] In some embodiments, the MR images are processed to identify and segment both bone segments and soft tissue within the MR images. According to some embodiments, when the same MR image is segmented to identify both bone segments and soft tissue, this approach is advantageous because the bone and soft tissue segments are inherently aligned with one another. Any suitable method for acquiring and processing MR images can be used for this purpose. One method that can be used is described, for example, in U.S. Pat. No. 10,748,309, the disclosure of which is incorporated herein by reference. (References incorporated by reference into this patent application should be considered an integral part of this application, but if terms are defined in these incorporated references in a manner that contradicts definitions expressly or implicitly made herein, only the definitions herein should be considered.) As another alternative, image processing methods using artificial intelligence, such as neural network-based methods, can be applied to segment the MR images.

[0232] Alternatively or additionally, a CT image can be received and segmented to identify bone segments, while an MR image can be segmented to identify soft tissue. The MR and CT images can then be registered with each other to generate a segmented 3D image that includes both the bone segments and the soft tissue. Registration of the MR and CT images can be performed by identifying and aligning landmarks in the two images, such as anatomical or artificial landmarks attached to the patient's body.

[0233] In some embodiments, 2D fluoroscopic images of the target area are used in registering the MR image data with the patient's body. To this end, two or more 2D fluoroscopic images of the target area of the patient's body are captured. The reference frames of the 2D fluoroscopic images are calibrated to the body (as described herein), and bone segments in the segmented 3D images are registered with the bones appearing in the 2D fluoroscopic images, for example, using digitally reconstructed radiographs (DRR).

[0234] In addition to receiving 2D X-ray images from the fluoroscope 30 of FIG. 1 , the processor 50 is configured to receive 3D medical images of the patient 24, including MR images and possibly CT images. These 3D images are typically acquired prior to surgery and stored in the memory 52. Alternatively or additionally, according to some embodiments, CT images may be generated intraoperatively, in which case the fluoroscope 30 and 2D X-ray images are not required. While the preoperative images are typically described herein as 3D images, 2D, 4D, or other preoperative images may also be used. In some embodiments, the processor 50 is configured to segment the 3D images to identify bone segments and / or soft tissue and align the 3D bone segments with their respective vertebrae in the 2D fluoroscopic images. The processor 50 may further be configured to present an image of the spine, including the aligned 3D bone segments and, optionally, soft tissue, on the head-mounted AR display unit 28, such that the vertebrae in the 3D images are aligned with the actual vertebrae of the patient's spine. Details of this process are described with reference to FIG. 41 .

[0235] Alternatively, the registered 3D bone segments and soft tissue may be presented on a different type of display, for example, an AR display attached to the patient 24 or operating table 26 above the surgical site or another local and / or remote display device. Additionally or alternatively, the registered 3D segments may be presented on a non-AR display, such as the display of a workstation or handheld computer. Other techniques for image registration and / or image fusion are described below with reference to Figures 43A, 43B, and 43C.

[0236] According to some embodiments, the processor 50 comprises a general-purpose computer processor, which is programmed with software to perform the segmentation, calibration, alignment, and display functions described herein. This software (e.g., executable program instructions) may be stored on a tangible, non-transitory computer-readable medium, such as an optical, magnetic, or electronic memory medium. The software may be stored in memory 52. Additionally or alternatively, at least some of the functions of the processor 50 can be performed using dedicated computing hardware, such as a graphics processing unit (GPU). The processor 50 may include one or more processors. The processor 50 may be located on the workstation and / or the head-mounted AR display unit 28 (FIG. 2A), or may be located remotely (e.g., on a cloud platform on one or more remote servers).

[0237] In one embodiment, the AR image includes one or more vertebrae or other bony structures segmented from a 3D medical image (e.g., MR or CT), as well as soft tissues adjacent to the vertebrae segmented from the MR or CT image. This AR image is projected onto an overlay region 62 ( FIG. 2A ) of the display 60 in alignment with the anatomical structures of the patient's 24 body visible to the surgeon 22 via the display 60. The overlay region 62 may be transparent, semi-transparent, or opaque. In other words, the image of the vertebrae is overlaid on the actual location of the corresponding vertebrae in the patient's 24 spine (either on the skin for minimally invasive surgery or on the actual vertebrae for open surgery).

[0238] To align the AR image with the patient's anatomy, one or more cameras 48 (e.g., infrared cameras or other optical cameras) capture respective images of a field of view (FOV), which may include, for example, marker 44, markers 42 and 44, or marker 44 and alignment marker 46. Processor 50 processes the images of one or more of markers 42, 44, 46, for example, to align marker 44 with the patient's body and determine the position and orientation of display unit 28 relative to the patient's body. Based on this alignment and the determination of the current position of display unit 28, processor 50 can select the appropriate vertebrae or portion of the spine to display in the AR image within overlay region 62 and set the appropriate magnification, translation, and orientation of the vertebrae and soft tissue in the AR image to match the underlying structures of the patient's spine from the surgeon's 22's perspective. One or more cameras 48 may also be used to optically track the position of patient 24, for example, via markers 44. The one or more cameras 48 may include two cameras (e.g., a left camera and a right camera) as shown in FIG. 2A or two additional cameras to provide a stereoscopic display of at least a portion of the surgeon's field of view captured by the two cameras. Accordingly, the one or more cameras 48 may consist of a single camera or may include three or more cameras. The head-mounted display unit 28 may be provided in the form of eyewear, such as glasses or goggles, as shown in FIG. 2A. Alternatively, the head-mounted display unit 28 may be provided in the form of an over-the-head or forehead-mounted headset 70, as shown in FIG. 2B.

[0239] 41 is a flowchart that schematically illustrates a method for generating and displaying a 3D model including bone and soft tissue information based on the registration of preoperative 3D MR images and intraoperative 2D fluoroscopic images, according to one embodiment of the present disclosure. The method is described here with reference to vertebrae of the spine, but can be used for other bones as well, such as shoulder bones, hip bones, knee bones, leg bones, arm bones, foot bones, ankle bones, skull bones, etc. Further details of this method are described herein.

[0240] To begin the process of registering the preoperative 3D MR image segments with the 2D image of the spine, the processor 50 segments the MR image of the patient's back into bone segments and soft tissue adjacent to the bone segments (block 4700), for example, as shown in FIG. 42. In block 4702, the processor 50 receives an initial input associating one of the vertebrae in the 3D image segment with the position of the same vertebra in the two 2D images. For example, a user of the system 20 can use a cursor to mark the position of the selected 3D vertebra on the 2D image. Additionally, in some embodiments, the processor 50 uses external cues to make an initial estimate of the orientation of the patient's 24 spine. For example, the position of the markers 44 or 46 relative to the patient's skeleton indicates the Z direction (e.g., the sagittal axis), and the positions of the X-ray source 32 and detector 34 indicate the Y direction (e.g., the longitudinal axis). Alternatively, the user reorients the 2D image to match the orientation of the 3D image segment. Based on the initial input and the estimated orientation of the spine, the processor 50 may associate each 3D image segment with a corresponding vertebra in each 2D image (block 4704). The processor 50 may also estimate and utilize known ranges of movement of the vertebrae relative to each other in estimating the registration parameters.

[0241] To accurately align vertebrae in the 3D MR image segments with associated vertebrae in the 2D image, processor 50 generates (e.g., calculates) digital reconstruction radiographs (DRRs) based on the 3D images of the vertebrae over a range of vertebral motion and rotation about the estimated axis of the 2D image relative to the spine (block 4706). In some embodiments, the intensity of each pixel in a given DRR is calculated by calculating the cumulative radiodensity of voxels along the path of a ray between the X-ray source and the pixel. In one example, processor 50 applies an optimization process to find the orientation of each 3D vertebra relative to the 2D image by comparing the gradient of pixel values in the DRR with the actual gradient of pixel values in the 2D X-ray image.

[0242] Once the optimal positions and orientations of all vertebrae in the 3D MR image segments have been found in this manner, processor 50 uses the results to reconstruct a complete 3D model of the spine from the individual 3D vertebrae (block 4708). According to some embodiments, the positions and orientations of the vertebrae in this 3D model match the actual spine (e.g., the actual pose of the spine) of patient 24 on operating table 26. The positions and orientations of the soft tissue in the segmented MR images proximate to the vertebrae can be reconstructed using the same transformation parameters generated in reconstructing the 3D model of the spine, resulting in the patient's entire anatomy being properly rendered and aligned with the underlying tissue.

[0243] The processor 50 may then display (or generate as output for display) a 3D model including both bone and soft tissue (e.g., to facilitate navigation of medical tools during a procedure) in block 4710. Image registration and fusion modes that may be used for this purpose are shown by way of example in FIGS. 43A, 43B, and 43C. If AR is utilized, the processor 50 may then use the relative position and orientation of the head-mounted AR display unit 28 or head-mounted AR display unit 70 with respect to the patient 24 to calculate views of the vertebrae and soft tissue that are overlaid on the actual anatomical structure of the patient 24 and projected onto the displays 60, 72 at the appropriate position and orientation.

[0244] 42 is a schematic illustration of a segmented 3D image for display in image-guided surgery, according to one embodiment of the present disclosure. The image is generated by processing and segmenting a patient's MR images and includes vertebrae 4800 surrounded by muscle tissue 4802. The image may also be enhanced (e.g., with virtual graphics) to show, for example, the location of the spinal cord 4804 passing through and between the vertebrae 4800, as well as peripheral nerves (also not shown) branching off from the spinal cord or tumor (also not shown).

[0245] 42 can be presented in registration with the underlying anatomical structures, for example, on display 60 (FIG. 2A) or display 72 (FIG. 2B). Alternatively or additionally, the image may be presented on a separate display (local and / or remote).

[0246] Fusion of multiple imaging modalities 43A-43C are flow charts that schematically illustrate modalities for image registration, fusion, and / or display using the tools and techniques described above, in accordance with embodiments of the present disclosure.

[0247] In the example of FIG. 43A, preoperative MR images are captured (block 4900) and converted by processor 50 or another processor or processors into 3D images showing bone structures, simulating and / or mimicking CT images. The bone MR images, shown in FIGS. 43A-43C as "Bone MR Images (block 4902)," may be used in place of CT images. The MR images may be converted into bone MR images by various techniques. Such techniques may include, for example, using BoneMRI® software, deep neural networks (e.g., U-Net or DenseNet), and / or other image processing techniques such as active contours or level sets. Such conversion techniques may include highlighting bone tissue and / or segmenting bone tissue to distinguish bone from surrounding soft tissue. Intraoperative 2D X-ray images are captured, for example, using a fluoroscope 30 (FIG. 1). In block 4906), the processor aligns bone segments or portions in the bone MR image with corresponding bone segments or portions in one or more 2D X-ray images, as described above, and the resulting aligned image data displaying the latest bone tissue data is presented on a display, for example, but not limited to, an augmented reality (AR) display (block 4908).

[0248] The embodiment of Figure 43B uses a similar process to generate an updated bone MR image by registering it with a preoperative 2D X-ray image (block 4904), as described with respect to Figure 43A. Processor 50 or another processor generates a registered fused MR image (including both bone and soft tissue) in block 4906 by utilizing the registered bone MR image for the updated bone tissue data and the original MR image for the soft tissue data, while the bone MR image and the original MR image are inherently registered with each other. The processor can then present the fused image on a display, optionally an AR display (block 4908).

[0249] In the example of FIG. 43C, registered CT-MR fusion image data is displayed. The CT image is used to provide bone data, and the MR image is used to provide soft tissue data. The CT bone data and the MR soft tissue data are fused to generate a fused image. The MR images are typically generated preoperatively. The CT images can be generated intraoperatively (block 4914) or preoperatively (block 4912). If the CT images are also generated preoperatively, an intraoperative 2D X-ray image can be captured and registered with the CT preoperative image to provide updated bone data (block 4916), for example, as described herein above with respect to FIG. 41. The intraoperative CT image or the registered preoperative CT image can then be fused with the MR image (block 4918). In some embodiments, to facilitate fusion of the MR and CT images, a bone MR image can be generated, for example, as described with respect to FIG. 43A. In some embodiments, the CT image (preoperative or intraoperative) can be segmented to define the bone tissue and facilitate CT image fusion with the MR image. Processor 50 or another processor or multiple processors can be utilized to perform the above-described steps. In some embodiments, the processor can register the preoperative MR image with the segmented CT image so that the soft tissue in the MR image is properly aligned with the bone in the CT image (if the MR image has been segmented to identify bone segments, these bone segments can be used to facilitate accurate registration of the MR and CT images). Once the MR and CT images are registered with each other, the processor fuses the data from the images and presents the resulting fused image, including both bone and soft tissue, on a display, optionally an AR display. Those skilled in the art can implement different methods for registering and fusing CT image data with MR image data.

[0250] The display of the fused image may include, for example, different colors for different tissues, color for one type of tissue and black and white for another type of tissue, or the tissue colors may be a grayscale corresponding to pixel intensity. Alternatively or additionally, it may be provided to switch between display modes displaying different types of tissue (e.g., bone tissue images vs. soft tissue images) while the images are displayed in registration and, optionally, alignment.

[0251] Displaying such fused images can be advantageous in various types of medical procedures. For example, in bone-related procedures, soft tissue information can provide information about important structures, such as nerves in spinal procedures. As another example, in soft tissue-related procedures, such as tumor removal, bone information can facilitate access and navigation.

[0252] In some embodiments, the fused image may be presented as a 3D image, for example, via a 3D model. In some embodiments, a 2D image including 2D slices of the fused image may be displayed. The 3D and / or 2D images may be presented in various views, including axial, sagittal, lateral, and / or anterior-posterior (AP) views. The display may provide necessary information and / or facilitate navigation during a medical procedure. In a head-mounted AR system, such as the head-mounted displays shown in FIGS. 2A and 2B, the fused image may be displayed from the perspective of a medical professional wearing a head-up display. In some embodiments, the fused image may be displayed from the perspective of the tip of a medical tool inserted into and navigating the patient's body.

[0253] In some embodiments, fusion images generated solely based on preoperative MRI utilizing the generation of bone MR images, or generated based on registration between preoperative MR images and CT images, as disclosed herein, can be used in the planning stages of a medical procedure or intervention.

[0254] Distortion Correction X-ray images have distortions that most closely resemble a combination of s-distortion and pincushion distortion. This type of distortion is shown schematically through the images of a bead plate in FIGS. 44A-44C. FIG. 44C further illustrates the presence of distortions in a zoomed-in row of images of a bead 3802 fitted to an undistorted line 5000. Distortion correction algorithms and processes that typically work with regular camera images do not work to correct distortions in X-ray images. In some embodiments, a two-stage approach is utilized to correct distortions in X-ray images: (1) image data refinement and (2) spline interpolation.

[0255] 45 is a flowchart that schematically illustrates an exemplary method for refining image data as part of a distortion correction process or algorithm that may be executed by one or more processors (e.g., processor 50). In some examples, the image data after the bead detection algorithm has been executed may include outliers and / or missing beads that may result in artifacts. In some embodiments, a refinement algorithm is used to refine the image data. In one embodiment, the refinement algorithm includes a first refinement setting, a first refinement pass, a second refinement setting, and a second refinement pass.

[0256] In some embodiments, the first refinement setting can include steps 5100, 5102, 5104, and 5106. In some embodiments, a bead detection algorithm such as those described herein is run on an X-ray image of the beads (e.g., beads of the upper bead plate 406), and the resulting detected beads constitute a grid of source points (e.g., a grid of observed bead points or control points (for purposes of spline calculation or interpolation)). A grid of target points (e.g., an ideal or expected grid of bead points) is created using prior knowledge, including information such as bead volume, size, and spacing. Using the grid of source points (e.g., source grid) and the grid of target points (e.g., target grid), missing splines are removed in step 5100. For example, the two grids are compared, and if the source grid is missing a row or column of source points, the corresponding target point in the target grid is removed. As previously mentioned, not all beads are detected by the bead detection algorithm. In step 5102, straight lines can be calculated from existing source or control points, and if there appear to be gaps indicating missing source points, new source points (e.g., generated source points) can be added to fill the source grid. In step 5104, the grid can be divided into horizontal and vertical grids, and source point lines are excluded if the total number of source points in an individual source point line is below a threshold. These lines are not included in subsequent spline calculations. In some implementations, splines can be constructed using two source points. In some implementations, splines can be constructed using three or more source points. In step 5106, a distance grid is calculated in which grid points are scored based on how far they are from the original source points. For example, a generated source point located one distance away from the original source point can be given a distance value of 1, and a generated source point located two distances away from the original source point can be given a distance value of 2.

[0257] In some implementations, the first refinement pass can include steps 5108, 5110, and 5112. The first refinement pass can occur upon completion of the first refinement setting subprocess. In step 5108, an unrefined spline is set, and using the spline calculated for the source grid, the intersections of the calculated spline and the target line are set as composite source grid points. In step 5110, outlier source points are detected and marked based at least on prior knowledge of the beads and bead patterns. In step 5112, one-way refinement can be performed. This refinement step uses neighboring point data to refine the source points. For example, in one embodiment, the calculated distance grid map determines which neighboring points are closest to each of the original source points, and the derivatives of the neighboring splines are calculated. For each control point, the algorithm determines which neighboring spline has a better score and matches the derivative accordingly. The result is a source grid containing refined source or control points.

[0258] In some embodiments, the second refinement pass can include steps 5114 and 5116. In step 5114, a union grid axis is generated. The refined control points from the first refinement pass are used to determine all vertical and horizontal splines. A new source grid is created, which includes source points including points at intersections between splines and points on target lines where splines are missing. In step 5116, the new combined source grid is used to update the distance and linear grids as done in steps 5106 and 5108.

[0259] In some embodiments, the second refinement pass can include steps 5118, 5120, and 5122. Step 5118 may be performed at the end of the second refinement setting subprocess. In step 5118, a spline for a new source grid is calculated. In step 5120, outliers are detected, and in step 5122, a refinement step is performed. This refinement step is performed by calculating a weighted average of the source point and its neighbors (e.g., neighboring points that are not marked as outliers and have low distance grid values). Following this refinement step, the result is a refined source grid and target grid that are used in the distortion correction algorithm.

[0260] In FIG. 46, a flowchart outlines an exemplary method for interpolating data as part of a distortion correction process according to an embodiment of the present disclosure. In step 5300, an X-ray image is preprocessed. According to one embodiment, a schematic X-ray image of a bead superposition (e.g., beads of the upper bead plate 406 and the lower bead plate 408) is obtained. The image of the source beads (e.g., the source grid) can be rotated and aligned with the (X, Y) axis, and a grid of target beads can be generated and aligned with the center position of the source grid. In step 5302, vertical splines are generated for the source grid and the target grid. In some implementations, the splines for the source points are calculated using source points that reside in the same column of the source grid, and the splines for the target points are calculated using the y-positions of the source grid points and the x-positions of the target grid points. In step 5304, the difference between the vertical spline generated from the source points and the vertical spline generated from the target points and the source points is estimated for each source point and repeated across all vertical splines. The differences are plotted as a function of the source point y-axis values and fitted to a spline. This fitted spline can be used to estimate difference values for all pixel lines on the source spline. This process can be repeated across all vertical splines. This spline interpolation process can then be repeated for each row of the source image in step 5306, and the determined differences can be plotted as a function of the source point x-axis values and fitted to a spline from which difference values for all pixel lines on the source spline are estimated (e.g., interpolated across all x positions in the source image). The spline values can be saved or stored in memory as x-correction amounts (step 5308). In step 5310 of the distortion correction algorithm, steps 5302 through 5308 are repeated for the horizontal spline to obtain a y-correction amount for each pixel.The interpolation process results in an x-axis and y-axis correction for every pixel in the source image, and can create an undistorted image by resampling the target pixels from the source image values.

[0261] Conclusion and Terminology Although examples of the disclosed techniques are given for body parts including spinal vertebrae, the systems, methods, and / or disclosed principles may also be applied to bones and / or body parts other than the spine, including the hip joint, pelvic bones, leg bones, arm bones, ankle bones, foot bones, shoulder bones, skull, oral and maxillofacial bones, sacroiliac joints, etc.

[0262] The disclosed techniques are generally presented in the context of image-guided surgery systems or methods, and therefore the disclosed techniques for medical image visualization should not be considered limited solely to augmented reality and / or head-mounted systems. For example, the techniques are applicable to processing images from different imaging modalities, as described above, for use in diagnosis.

[0263] The terms "top," "bottom," "first," "second," "upper," "lower," "height," "width," "length," "end," "side," "horizontal," "vertical," and similar terms may be used herein, with the understanding that these terms refer only to the structures shown in the figures and are utilized solely to facilitate the description of embodiments of the present disclosure. Various embodiments of the present disclosure are presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the present disclosure. Ranges disclosed herein encompass any and all overlaps, subranges, and combinations thereof, as well as individual numbers within that range. For example, description of a range such as about 5 degrees to about 30 degrees should be considered to specifically disclose subranges such as 5 degrees to 10 degrees, 10 degrees to 20 degrees, 5 degrees to 25 degrees, 15 degrees to 30 degrees, etc., as well as individual numbers within that range (e.g., 5, 10, 15, 20, 25, 12, 15.5, and any whole or partial increments therebetween). Language such as "up to," "at least," "greater than," "less than," and "between" includes the recited numbers. Numbers preceded by terms such as "about" or "approximately" include the recited numbers. For example, "about 2 mm" includes "2 mm." As used herein, the terms "approximately," "about," and "substantially" refer to an amount close to the recited amount that still performs the desired function or achieves the desired result.

[0264] In some embodiments, the system comprises various features that exist as a single feature (rather than multiple features). For example, in one embodiment, the system comprises a single HMD, a single camera, a single processor, a single display, a single marker, a single calibration fixture, a single image, a single bead plate, a single imaging device, a single fluoroscope, etc. In alternative embodiments, multiple features or components are provided.

[0265] In some embodiments, the system comprises one or more of an imaging means (e.g., a camera or a fluoroscope or an MRI machine or a CT machine), a calibration means (e.g., a calibration fixture), an alignment means (e.g., an adapter, a marker, an object, a camera), a fixation means (e.g., an anchor, an adhesive, a clamp, a pin), a segmentation means (e.g., one or more neural networks), a distortion correction means (e.g., a grid of ring markers and beads), and the like.

[0266] The processor described herein may include one or more central processing units (CPUs) or processors or microprocessors. The processor may be communicatively coupled to one or more memory units, such as a random access memory (RAM) for temporarily storing information, one or more read-only memories (ROM) for persistently storing information, and one or more mass storage devices, such as a hard drive, a diskette, a solid-state drive, or an optical media storage device. The processor (or a memory unit communicatively coupled thereto) may include modules containing program instructions or algorithm steps configured to be executed by the processor to perform any of the processes or algorithms described herein. The processor may be communicatively coupled to external devices (e.g., display devices, data storage devices, databases, servers, etc.) over a network via a network communication interface.

[0267] In general, the algorithms or processes described herein may be implemented by logic embodied in hardware or firmware, or by a collection of software instructions written in a programming language such as Python, Java, Lua, C, C#, or C++, possibly having entry and exit points. A software module or product may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It will be understood that software modules may be called by other modules or by themselves and / or in response to detected events or interrupts. Software modules configured to run on a computing device may be provided on a computer-readable medium such as a compact disc, digital video disc, flash drive, or any other tangible medium. Such software code may be partially or completely stored in a memory device of an executing computing device, such as computing system 50, for execution by the computing device. The software instructions may also be embedded in firmware, such as an EPROM. It will be further understood that a hardware module may be composed of connected logic units such as gates and flip-flops, and / or may be composed of programmable units such as programmable gate arrays or processors. The modules described herein are preferably implemented as software modules, but may also be represented in hardware or firmware. In general, any module or program or flowchart described herein may refer to a logical module that may be combined with other modules or divided into sub-modules, regardless of physical organization or storage.

[0268] The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Furthermore, in some implementations, certain method or process blocks or steps may be omitted. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states may be performed in other sequences as appropriate. For example, described blocks, steps, or states may be performed in an order other than the order specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks, steps, or states may be performed serially, in parallel, or in some other manner. Blocks, steps, or states may be added or deleted from the disclosed example embodiments. The example systems and components described herein may be configured differently from that described. For example, elements may be added, removed, or rearranged compared to the disclosed example embodiments.

[0269] Any process description, element, or block in the flow diagrams described in this specification and / or shown in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code that includes one or more executable instructions for implementing a particular logical function or step in the process.

[0270] It will be understood that the systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for or required to achieve the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. The section headings used herein are provided merely to enhance readability and are not intended to limit the scope of the embodiments disclosed in a particular section to the features or elements disclosed in that section.

[0271] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is necessary or essential to every embodiment.

[0272] In particular, conditional language such as "can," "could," "might," or "may," unless otherwise specified or understood otherwise within the context of use, is intended to generally convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are somehow required in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included in or performed in any particular embodiment, with or without user input or prompting.

[0273] Terms such as "comprise," "include," and "have" are synonymous and are used in an inclusive, open-ended manner and do not exclude additional elements, features, acts, operations, etc. Furthermore, the term "or" is used in an inclusive (rather than exclusive) sense; for example, when used to connect a list of elements, the term "or" may mean one, some, or all of the elements in the list. Furthermore, the articles "a," "an," and "the," as used in this application and the appended claims, should be construed to mean "one or more" or "at least one," unless otherwise specified.

Claims

1. A method for facilitating augmented reality-assisted navigation based on preoperative 3D imaging and intraoperative 2D imaging of at least a portion of a patient's spine, A step of receiving a three-dimensional (3D) tomographic image of at least a portion of the spine of the patient, wherein the portion of the spine includes a plurality of vertebrae; The steps include segmenting the 3D tomographic image into a plurality of 3D segments, each of which includes each of the plurality of vertebrae, The steps include receiving two or more two-dimensional (2D) fluorescence images of at least a portion of the patient's spine, The steps include aligning each of the plurality of 3D segments with each of the plurality of vertebrae in the two or more 2D fluoroscopic images, Based on the alignment step, a step of generating a 3D image volume of at least a portion of the spine, The steps include presenting the 3D image volume of at least a portion of the spine on an augmented reality display, Methods that include...

2. The method according to claim 1, wherein the 3D tomographic image is a computed tomography (CT) image.

3. The method according to claim 1, wherein the step of segmenting the 3D tomographic image into a plurality of 3D segments is performed automatically by one or more processors.

4. The method according to claim 1, wherein the step of segmenting the 3D tomographic image into a plurality of 3D segments is performed by applying one or more convolutional neural networks.

5. The method according to claim 1, wherein the step of segmenting the 3D tomographic image into a plurality of 3D segments includes the step of labeling the plurality of vertebrae.

6. The method according to claim 1, wherein the step of receiving two or more 2D fluorescence images includes the steps of receiving a first 2D fluorescence image captured from a first angle or viewpoint of a C-arm fluorescence microscope, and receiving a second 2D fluorescence image captured from a second angle or viewpoint of the C-arm fluorescence microscope that is different from the first angle or viewpoint.

7. The method according to claim 1, wherein the step of presenting the 3D image volume includes the step of overlaying augmented reality images of the aligned 3D segments onto the patient's back.

8. The method according to claim 7, further comprising the step of calibrating a reference frame of the 2D fluoroscopic image to the spine of the patient prior to the alignment step, wherein the step of overlaying the augmented reality image includes applying the calibrated reference frame when overlaying the vertebrae of the aligned plurality of 3D segments onto the spine of the patient.

9. The method according to claim 7 or 8, wherein the step of receiving two or more 2D fluoroscopic images includes the step of receiving two 2D fluoroscopic images captured from different angles relative to the patient.

10. The method according to claim 9, wherein the step of aligning each of the plurality of 3D segments includes the step of aligning each of the plurality of 3D segments with both of the two 2D fluoroscopic images.

11. The method according to claim 8, wherein the step of calibrating the reference frame of the 2D fluorescence image includes the step of performing distortion correction on the two or more 2D fluorescence images.

12. The method according to claim 11, wherein the step of performing distortion correction includes spline interpolation.

13. The method according to claim 1, wherein the step of aligning each of the plurality of 3D segments includes adjusting the position and orientation of each of the plurality of 3D segments so that they match each of the plurality of vertebrae in the two or more 2D fluoroscopic images.

14. The step of adjusting the respective position and orientation includes the step of processing each of the plurality of 3D segments to generate a digitally reconstructed X-ray image (DRR), The method according to claim 13, wherein each of the aforementioned positions and orientations is adjusted to maximize the similarity between the DRR and the two or more 2D fluorescence imaging images.

15. The method according to claim 1, wherein the augmented reality display is a see-through stereoscopic display for a wearable device.

16. The method according to claim 15, wherein the wearable device includes glasses.

17. The method according to claim 15, wherein the wearable device is a head-mounted unit.

18. The method according to claim 1, wherein the step of aligning each of the plurality of 3D segments includes the step of performing an initial estimation.

19. The method according to claim 1, wherein the 3D image volume includes a reconstructed 3D model of the plurality of vertebrae.

20. The aforementioned 3D tomographic image is a computed tomography (CT) image. The augmented reality display is a see-through stereoscopic display for a wearable device. The step of receiving two or more 2D fluorescence images includes the steps of receiving a first 2D fluorescence image captured from a first angle or viewpoint of the C-arm fluorescence microscope, and receiving a second 2D fluorescence image captured from a second angle or viewpoint of the C-arm fluorescence microscope that is different from the first angle or viewpoint. The step of presenting the 3D image volume includes the step of overlaying the augmented reality images of the aligned 3D segments onto the patient's back. The step of aligning each of the plurality of 3D segments includes adjusting the position and orientation of each of the plurality of 3D segments so that they match the respective vertebrae in the two or more 2D fluoroscopic images. The steps of adjusting the respective positions and orientations include processing each 3D segment to generate a digitally reconstructed X-ray image (DRR), and finding the best match between the DRR and each of the vertebrae in the one or more 2D fluoroscopic images. The method according to claim 1.

Images