Systems and methods for visually guiding bone removal during arthroscopic procedures

Abstract

A method for guiding bone removal during a surgical procedure includes receiving a two-dimensional image of at least a portion of a joint during a surgical procedure; determining an alignment of a pre-generated three-dimensional model of the at least a portion of the joint with the two-dimensional image based on one or more features in the two-dimensional image related to the at least a portion of the joint, wherein the pre-generated three-dimensional model contains a representation of planned bone removal; generating an overlay image containing an overlay of at least a portion of the representation of the planned bone removal on the two-dimensional image based on the determined alignment; and displaying the overlay image to guide the bone removal during the surgical procedure.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 979,993, filed on February 21, 2020, the entire contents of which are hereby incorporated by reference.

[0003] field

[0004] This disclosure relates generally to orthopedics, and more specifically to surgical methods and systems for treating joints.

[0005] background

[0006] Orthopedics is a medical specialty focused on the diagnosis, correction, prevention, and treatment of skeletal disorders, including conditions or disorders of the bones, joints, muscles, ligaments, tendons, nerves, and skin that make up the musculoskeletal system. Joint injuries or disorders, such as those of the hip or other joints, can occur due to overuse or overstretching or due to other factors, including genetic factors that can cause deviations from "normal" joint morphology.

[0007] Joints are susceptible to a variety of conditions, including those that can cause deviations from normal joint morphology. These conditions may have both congenital and injury-related origins. In some cases, the condition may be severe from the outset. In others, it may begin mild but can worsen over time if left untreated. More specifically, in many cases, existing conditions may be exacerbated, for example, due to the dynamic nature of the joint, the heavy weight loads applied to the joint, or a combination thereof. The condition may initially or later significantly interfere with a patient's comfort and lifestyle, and may require surgical intervention.

[0008] Current trends in orthopedic surgery include the use of minimally invasive techniques such as arthroscopy to treat joint lesions, in which an endoscope is inserted into the joint through a small incision. Arthroscopic procedures include debridement of the bone lesion, where portions of bone deviating from the "normal" or target shape of the joint are removed. During the debridement procedure, the surgeon uses an endoscopic camera to view the area being debrided; however, due to the limited and somewhat distorted field of view of the resulting endoscopic images, the surgeon cannot view the entire lesion at once. Therefore, it is often difficult for the surgeon to accurately determine the amount of bone that should be removed and whether the remaining bone has the desired geometry.

[0009] Overview

[0010] According to one aspect, the system and method include displaying an overlay of a three-dimensional representation of planned bone removal on a two-dimensional image of the joint. The two-dimensional image may be a preoperative or intraoperative image of the joint, showing the joint in its current state before or during the medical procedure, and the three-dimensional representation of planned bone removal indicates the location in three-dimensional space where bone should be removed from the joint, such that the overlay illustrates the location of bone removal at both the bone contour to be captured in the two-dimensional image and the joint portion extending beyond the two-dimensional imaging plane. By providing a three-dimensional representation of planned bone removal as an overlay on a two-dimensional image of the joint, practitioners can better understand the location of bone to be removed that extends precisely beyond the bone contour captured in the two-dimensional image.

[0011] According to one aspect, the system and method generate an overlay by extracting one or more features related to the location of the target bone from a two-dimensional image and using these features to determine the alignment of a three-dimensional model of the joint with the two-dimensional image. The representation of the planned bone removal in the two-dimensional image can then be rendered as an overlay in the appropriate location and orientation, allowing the practitioner to visualize the location and amount of bone to be removed outside the imaging plane. Optionally, this process can be repeated on newly generated two-dimensional images as the surgery progresses.

[0012] According to one aspect, a method for visualizing planned bone removal associated with a joint includes: receiving a two-dimensional image of at least a portion of a joint; determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image based on one or more features in the two-dimensional image associated with at least a portion of the joint, wherein the pre-generated three-dimensional model contains a representation of planned bone removal; generating an overlay image based on the determined alignment, the overlay image containing an overlay of at least a portion of the representation of planned bone removal on the two-dimensional image; and displaying the overlay image.

[0013] Optionally, the 3D model can be pre-generated based on one or more joint scans.

[0014] Optionally, the joint may include the femur, and one or more features may be associated with at least one of the center of the femoral head, the central line of the femoral neck, and the outer edge of the femoral head.

[0015] Optionally, the joint may include the pelvis, and one or more features may be associated with at least one of the acetabular center, obturator foramen, and pubic symphysis.

[0016] Optionally, the joint may include the tibia, and one or more features may be associated with at least one of the tibial plateau, tibial shaft, and intercondylar eminence.

[0017] Optionally, the joint may include a vertebra, and one or more features may be associated with at least one of the pedicle, facet, superior endplate, and inferior endplate.

[0018] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include translating and rotating the 3D model based on one or more features.

[0019] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include detecting edges in the 2D image that are associated with the bone periphery.

[0020] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include determining the center of the femoral head in the 2D image, determining the centerline of the femoral neck in the 2D image, aligning the center of the model femoral head in the 3D model with the center of the femoral head in the 2D image, and aligning the centerline of the model femoral neck in the 3D model with the centerline of the femoral neck in the 2D image.

[0021] Optionally, the representation of planned bone removal may include at least one of a heat map indicating the location and amount of planned bone removal, a contour map indicating the location and amount of planned bone removal, and a profile of the planned bone removal area.

[0022] Optionally, the three-dimensional model may include representations of bones other than those planned for bone removal, and the representations of bones other than those planned for bone removal may be omitted from the overlay.

[0023] Optionally, the method may further include determining a portion of a three-dimensional model associated with the bone removed during the surgical procedure and omitting that portion of the three-dimensional model from the overlay. Optionally, the portion of the three-dimensional model omitted from the overlay may be a portion of a heatmap associated with the planned bone removal.

[0024] Optionally, the method may further include, after displaying the overlay image, modifying the overlay position of at least a portion of the representation of planned bone removal on the two-dimensional image in response to user input.

[0025] Optionally, the method may further include capturing a new two-dimensional image of the joint portion at the new location, determining an updated alignment between the pre-generated three-dimensional model and the new two-dimensional image, generating an updated overlay image based on the determined updated alignment, and displaying the updated overlay image to indicate the progress of bone removal.

[0026] Optionally, the planned bone removal representation may be three-dimensional, and generating the overlay image may include projecting the planned bone removal representation onto a two-dimensional plane.

[0027] Optionally, the representation of planned bone removal may indicate planned bone removal associated with cam-type femoral-acetabular impingement syndrome.

[0028] Optionally, the representation of planned bone removal may indicate planned bone removal associated with clamp-type femoral-acetabular impingement syndrome.

[0029] Optionally, the indication of planned bone removal may indicate sawing of the tibia and tibial deformity during high tibial osteotomy (HTO) to treat knee deformities.

[0030] Optionally, the indication of planned bone removal may indicate the planned drill holes for placement of pedicle screws during spinal fusion.

[0031] Optionally, two-dimensional images can be received intraoperatively from an X-ray system, such as a C-arm fluoroscopy system.

[0032] Optionally, the three-dimensional model may be based on imaging data from a first type of imaging system, while the two-dimensional image may be received from a second type of imaging system, different from the first type. Optionally, the first type of imaging system may be an MRI system or a CT system, while the second type of imaging system may be a C-arm fluorescence microscope.

[0033] According to one aspect, a system for guiding bone removal during a surgical procedure may include one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a two-dimensional image of at least a portion of a joint; determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image based on one or more features in the two-dimensional image relating to at least a portion of the joint, wherein the pre-generated three-dimensional model contains a representation of the planned bone removal; generating a superimposed image based on the determined alignment, the superimposed image containing a superposition of at least a portion of the representation of the planned bone removal on the two-dimensional image; and transmitting the superimposed image to a display.

[0034] Optionally, the system can be configured to connect communicatively to an intraoperative imaging system that generates two-dimensional images.

[0035] Optionally, one or more procedures may include instructions for receiving user input to reposition at least a portion of the superimposed representation of the planned bone removal on a two-dimensional image.

[0036] Optionally, the system may include a touchscreen display or augmented reality system for displaying two-dimensional images and receiving user input.

[0037] Optionally, the 3D model can be pre-generated based on one or more joint scans.

[0038] Optionally, one or more features may be associated with at least one of the center of the femoral head, the central line of the femoral neck, and the outer edge of the femoral head.

[0039] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include translating and rotating the 3D model based on one or more features.

[0040] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include detecting edges in the 2D image that are associated with the bone periphery.

[0041] Optionally, determining the alignment of a pre-generated 3D model of at least a portion of the joint with a 2D image may include determining the center of the femoral head in the 2D image, determining the centerline of the femoral neck in the 2D image, aligning the center of the model femoral head in the 3D model with the center of the femoral head in the 2D image, and aligning the centerline of the model femoral neck in the 3D model with the centerline of the femoral neck in the 2D image.

[0042] Optionally, the representation of planned bone removal may include at least one of a heat map indicating the location and amount of planned bone removal, a contour map indicating the location and amount of planned bone removal, and a profile of the planned bone removal area.

[0043] Optionally, the three-dimensional model may include representations of bones other than those planned for bone removal, and the representations of bones other than those planned for bone removal may be omitted from the overlay.

[0044] Optionally, one or more procedures may include instructions for determining a portion of a three-dimensional model related to bone removed during the surgical procedure and for omitting that portion of the three-dimensional model from the overlay. Optionally, the portion of the three-dimensional model omitted from the overlay may be a portion of a heatmap related to the planned bone removal.

[0045] Optionally, one or more programs may include instructions for modifying the position of the overlay of at least a portion of the representation of planned bone removal on a two-dimensional image in response to user input after the overlay image has been displayed.

[0046] Optionally, one or more procedures may include instructions for: capturing a new two-dimensional image of the joint portion at a new location, determining an updated alignment between the pre-generated three-dimensional model and the new two-dimensional image, generating an updated overlay image based on the determined updated alignment, and displaying the updated overlay image to indicate the progress of bone removal.

[0047] Optionally, the planned bone removal representation may be three-dimensional, and generating the overlay image includes projecting the planned bone removal representation onto a two-dimensional plane.

[0048] Optionally, the representation of planned bone removal may indicate planned bone removal associated with cam-type femoral-acetabular impingement syndrome.

[0049] Optionally, the representation of planned bone removal may indicate planned bone removal associated with clamp-type femoral-acetabular impingement syndrome.

[0050] Optionally, two-dimensional images can be received intraoperatively from an X-ray system.

[0051] Optionally, the three-dimensional model may be based on imaging data from a first type of imaging system, while the two-dimensional image may be received from a second type of imaging system, different from the first type. Optionally, the first type of imaging system may be an MRI system or a CT system, while the second type of imaging system may be a C-arm X-ray system.

[0052] According to one aspect, a non-transitory computer-readable medium stores instructions for execution by one or more processors of an imaging system to implement any of the methods described above.

[0053] According to one aspect, a computer program product includes computer-executable instructions that, when executed by a programmable computer, cause the computer to perform any of the methods described above.

[0054] It should be recognized that any changes, aspects, features, and options in the system description also apply to the method, and vice versa. It should also be clear that any one or more of the aforementioned changes, aspects, features, and options can be combined. Brief description of the attached diagram

[0056] The invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0057] Figure 1A-1D A schematic diagram illustrating, for example, various aspects of hip movement;

[0058] Figure 2 A schematic diagram of the bone structure in the hip joint region;

[0059] Figure 3 This is a schematic anterior view of the femur.

[0060] Figure 4 A schematic rear view of the top of the femur;

[0061] Figure 5 This is a diagram of the pelvis.

[0062] Figure 6-12 A schematic diagram showing the bone and soft tissue structures in the hip joint region;

[0063] Figure 13 A schematic diagram illustrating cam-type femoral-acetabular impingement syndrome (i.e., cam-type FAI);

[0064] Figure 14 A schematic diagram illustrating clamp-type femoral-acetabular impingement syndrome (i.e., clamp-type FAI);

[0065] Figure 15 A schematic diagram illustrating a labral tear;

[0066] Figure 16 A schematic diagram illustrating the measurement of the α angle of the patient's hip;

[0067] Figure 17 A schematic diagram illustrating the measurement of the central edge angle of the patient's hip;

[0068] Figure 18 A schematic diagram of a surgical kit according to some implementation schemes;

[0069] Figure 19 Description of a method, based on some implementation schemes, for guiding a surgeon to remove bone from a part of a joint during a surgical procedure (1900);

[0070] Figure 20 This describes aspects of method 1900 applied to the femur according to some implementation schemes;

[0071] Figure 21 This describes the implementation of step 1904 for extracting femoral features;

[0072] Figure 22 The illustration shows an example of a two-dimensional X-ray image of a portion of a hip joint captured by an X-ray imaging system, according to some implementation schemes, where a user prompt is placed on the image;

[0073] Figure 23 Examples illustrating the results of femoral head edge detection algorithms based on some implementation schemes;

[0074] Figure 24 This illustrates an example of a circle around the edge of the femoral head detected by edge detection based on a Hough transform according to one implementation scheme;

[0075] Figure 25 This describes a method for determining the alignment of a 3D model with a 2D image, based on some implementation schemes.

[0076] Figure 26 Application description Figure 25 An implementation of the method, the method being used to align a three-dimensional model of at least the upper part of a subject's femur with a two-dimensional image of at least the upper part of the subject's femur captured during a surgical procedure;

[0077] Figure 27 This describes the available degrees of freedom for model alignment according to some implementation schemes;

[0078] Figure 28A and 28B An overlay image illustrating some implementation schemes;

[0079] Figure 29A and 29B This describes a user interface for overlaying and adjusting images, provided according to some implementation schemes;

[0080] Figure 30This describes a method for generating an overlay image according to some embodiments, the overlay image having an updated representation of planned bone removal reflecting bone removed during a surgical procedure;

[0081] Figures 31A-31C Explanation based on an implementation plan Figure 30 The result of the method;

[0082] Figure 32 This describes a system, based on various implementation schemes, for generating and displaying overlay images to a physician to guide bone removal during surgical procedures.

[0083] Figure 33 Illustrate examples of computing systems based on some implementation schemes;

[0084] Figure 34 Explain the back projection technique according to some implementation schemes;

[0085] Figure 35 The description explains how to use machine learning models trained to detect the femoral head and neck to generate scoring bounding boxes according to various implementation schemes;

[0086] Figure 36 Description of a graphical user interface for indicating the clock face position of tools, according to various implementation schemes; and

[0087] Figure 37 This section describes exemplary machine learning algorithms used to determine the alignment of a 3D model with a 2D image, based on various implementation schemes.

[0088] Detailed Explanation

[0089] Detailed reference will now be made to various aspects and variations of the invention, examples of which are illustrated in the accompanying drawings. Various devices, systems, and methods are described herein. Although at least two variations of the devices, systems, and methods are described, other variations may include aspects of the devices, systems, and methods described herein that have all or some of the aspects described, combined in any suitable manner. Exemplary embodiments will now be described more fully with reference to the accompanying drawings; however, they may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey exemplary practices to those skilled in the art.

[0090] According to various implementations, systems and methods based on the principles described herein can provide visualization of at least one three-dimensional region of a joint that deviates from baseline joint morphology. Visualization can be provided before, during, and / or after surgical procedures on the joint. Visualization can indicate bone removal planning to bring the region back to baseline joint morphology. Visualization can be used to guide physicians, such as guiding surgeons in planning and / or performing surgical treatment. Visualization can be overlaid on two-dimensional images of bone captured preoperatively to guide physicians in planning medical procedures intraoperatively; overlaid on two-dimensional images of bone captured intraoperatively to allow surgeons to visualize planned bone removal relative to the intraoperative state of the bone; or overlaid on two-dimensional images of bone captured postoperatively to evaluate the state of the bone relative to the plan. It should be understood that the systems and methods described herein can be used for non-surgical purposes, such as aiding in the evaluation of joint lesions or in evaluating the success of surgical or non-surgical procedures.

[0091] Visualizations of planned bone removal can be extracted from 3D models of the subject's joints generated from one or more preoperative scans. These visualizations can be correlated with deviations of the modeled joint from the baseline joint morphology, identified by comparing the 3D model with baseline data representing the target joint morphology. As discussed further below, visualizations can be overlaid on intraoperatively generated 2D images of the joint by determining the alignment of the 3D model with the 2D images.

[0092] As used in this article, “bone removal” includes any method of removing bone, including bone removal by drilling, sawing, punching, and using a bone chisel.

[0093] In the following description, it should be understood that the singular forms “a,” “an,” and “the” used herein are also intended to include the plural forms, unless the context explicitly indicates otherwise. It should also be understood that the term “and / or,” as used herein, refers to and includes any and all possible combinations of one or more of the related listed items. It should be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used herein, specify the presence of the stated feature, integer, step, operation, element, component, and / or unit, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, units, and / or groups thereof.

[0094] Certain aspects of this disclosure include process steps and instructions described herein in algorithmic form. It should be noted that the process steps and instructions of this disclosure may be embodied in software, firmware, or hardware, and when embodied in software, may be downloaded to reside on and be operated by various operating systems on different platforms. Unless otherwise specifically stated as will be apparent from the following discussion, it should be understood that throughout the description, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” “generating,” etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in computer system memory or registers or other such information storage, transmission, or display devices.

[0095] This disclosure also relates, in some embodiments, to an apparatus or system for carrying out the operations described herein. The apparatus or system may be a general-purpose computer specifically built for the desired purpose, may comprise a computer selectively activated or reconfigured by a computer program stored in the computer, or may comprise any combination thereof. Computer instructions for carrying out the operations described herein may be stored in any combination of non-transitory computer-readable storage media, such as, but not limited to, any type of disk, including floppy disks, USB flash drives, external hard disk drives, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, or optical cards, or any type of media suitable for storing electronic instructions, each connected to a computer system bus. One or more instructions for carrying out the operations described herein may be implemented or executed by one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processing units (DSPs), graphics processing units (GPUs), or central processing units (CPUs). Furthermore, the computer mentioned herein may comprise a single processor or may be an architecture employing multiple processors to increase computing power.

[0096] The methods, apparatus, and systems described herein are not inherently related to any particular computer or other device. Various general-purpose systems may also be used with the procedures taught herein, or may prove convenient for constructing more specialized devices to implement the required method steps. The necessary structures for various such systems will become apparent from the following description. Furthermore, this invention is described without reference to any particular programming language. It should be appreciated that various programming languages ​​can be used to implement the teachings of the invention as described herein.

[0097] While the examples below typically relate to the hip joint, hip pathology, and hip joint features and measurements, it should be understood that the systems, methods, techniques, visualizations, etc., described in this article according to various implementation schemes can be used to analyze and visualize other joints, including the knee, shoulder, elbow, spine, ankle, etc.

[0098] According to some implementation schemes, physicians can be provided with improved guidance regarding the degree of deviation of joint morphology from the target morphology and the amount of bone that should be removed to achieve the target morphology, for example during minimally invasive arthroscopic procedures or open surgical procedures. According to some implementation schemes, visualization can provide physicians with improved guidance regarding hip joint morphology measurements, including the α angle, lateral central edge angle, acetabular tilt and femoral torsion, Tönnis angle, neck-shaft angle, and acetabular coverage, which help practitioners measure the deviation of the object morphology from the target morphology.

[0099] The target joint morphology can be any joint morphology that a given subject might expect. The target joint morphology can be based on the anatomical representation of any reference patient population (e.g., a normal patient population). For example, baseline data can be derived from studies of healthy patient populations and / or from models of “normal” joints generated based on measurements, computer simulations, calculations, etc. The terms target, baseline, and reference are used interchangeably throughout this document to describe the characteristics of a joint morphology compared to the subject's joint morphology.

[0100] A more precise understanding of the anatomy of the joints leads to a better understanding of various joint pathologies and the advantages offered by the various implementation schemes described herein. The hip joint forms at the junction of the femur and the hip. The hip joint is a ball-and-socket joint and is capable of a wide range of movements, such as flexion and extension, abduction and adduction, and internal and external rotation. Figure 1A-1D As shown. Perhaps second only to the shoulder joint, the hip joint is the most flexible joint in the body. For most of the day, the hip joint bears a significant load under both static (e.g., standing and sitting) and dynamic (e.g., walking and running) conditions.

[0101] More specifically, and see Figure 2 The femoral ball is received in the acetabular cup at the hip, where multiple ligaments and other soft tissues serve to hold the bone in its jointed state. Figure 3 As shown, the femur is generally characterized by a slender body that terminates at its apex with an angled neck supporting a hemispherical head (sometimes referred to as a ball). Figure 3 and 4 As shown, a large protrusion, called the greater trochanter, protrudes laterally and posteriorly from the elongated body adjacent to the neck. A slightly smaller protrusion, called the lesser trochanter, protrudes medially and posteriorly from the elongated body adjacent to the neck. The intertrochanteric crest extends along the periphery of the femur between the greater and lesser trochanters.

[0102] See Figure 5 The pelvis is composed of three bones: the ilium, ischium, and pubis. These three bones work together (they typically ossify into a single "hip bone" structure by age 25) to form the acetabular cup. The acetabular cup receives the femoral head.

[0103] Both the femoral head and the acetabular cup are covered by a layer of articular cartilage, which protects the underlying bone and facilitates movement (see [link]). Figure 6 Various ligaments and soft tissues are used to hold the femoral ball in proper position within the acetabular cup. More specifically, see [link to relevant documentation]. Figure 7 and 8 The round ligament extends between the base of the femoral ball and the acetabular cup. See also Figure 9 The labrum is positioned around the outer edge of the acetabular cup. The labrum increases the depth of the acetabular cup and effectively creates a suction seal between the femoral ball and the edge of the acetabular cup, thereby helping to retain the femoral head within the acetabular cup. Also see... Figure 10 The fibrous capsule extends between the femoral neck and the rim of the acetabular cup, effectively sealing the ball-and-socket components of the hip joint from the rest of the body. This structure is surrounded and reinforced by a set of three major ligaments extending between the femur and hip (i.e., the iliofemoral ligament, ischiofemoral ligament, and pubofemoral ligament) (see [link to relevant documentation]). Figure 11 and Figure 12 ).

[0104] The hip joint is susceptible to a variety of different conditions. These conditions may have both congenital and injury-related origins. For example, congenital hip joint lesions involve impingement between the femoral neck and the acetabular cup rim. In some cases, and see [further details omitted] Figure 13 This impingement can occur due to irregularities in the femoral geometry. This type of impingement is sometimes referred to as cam-type femoral-acetabular impingement syndrome (i.e., cam-type FAI). In other cases, see [link to relevant documentation]. Figure 14 Impingement can occur due to irregularities in the geometry of the acetabular cup. This latter type of impingement is sometimes referred to as clamp-type femoral-acetabular impingement syndrome (i.e., clamp-type FAI). Impingement can lead to reduced range of motion, significant pain, and in some cases, significant hip degeneration.

[0105] Another example of congenital hip joint disorders involves defects in the articular surfaces of the ball and / or the acetabular cup. This type of defect sometimes starts quite small but typically increases in size over time, usually due to the dynamic and weight-bearing characteristics of the hip joint. Joint defects can cause significant pain, trigger or exacerbate arthritis symptoms, and in some cases lead to significant hip joint degeneration.

[0106] Examples of injury-related lesions of the hip joint involve trauma to the labrum. In many cases, accident- or sports-related injuries can lead to labral tears, typically accompanied by tears extending through the body of the labrum (see, for example, [link to relevant documentation]). Figure 15 These types of injuries can be painful for patients and, if left untreated, can lead to significant degeneration of the hip joint.

[0107] Current trends in orthopedic surgery favor the use of minimally invasive techniques to treat joint conditions. For example, a common technique is the use of minimally invasive "keyhole" techniques to reattach ligaments in the shoulder joint, which does not require "spreading out" the joint capsule. Additionally, minimally invasive techniques are frequently used to repair torn meniscus cartilage and / or replace ruptured ACL ligaments in the knee. While such minimally invasive approaches may require additional training for surgeons, these procedures typically offer significant benefits to patients and have become standard care for many shoulder and knee conditions.

[0108] In addition to the above, due to the widespread availability of minimally invasive methods for treating shoulder and knee joint diseases, the current trend is to provide such treatment earlier in the life cycle of the disease, thereby resolving patient pain as quickly as possible and minimizing any deterioration of the disease itself. This contrasts sharply with traditional surgical practice, which typically requires postponing surgical procedures for as long as possible to spare patients the significant trauma usually associated with invasive surgery.

[0109] Minimally invasive treatment for hip joint diseases lags behind that for shoulder and knee joint diseases. This may be due, for example, to (i) the geometry of the hip joint itself, and (ii) the characteristics of the lesions that generally must be addressed in the hip joint.

[0110] The hip joint is generally considered a "tight" joint, meaning there is relatively little room for manipulation within its own dimensions. This contrasts with the knee joint, which is generally considered relatively spacious when compared to the hip. Therefore, performing minimally invasive surgery on the hip joint is relatively more challenging for surgeons.

[0111] Furthermore, the natural pathways into the hip joint (i.e., the pathways that naturally exist between adjacent bones) typically impose much greater constraints on the hip joint than on the shoulder or knee joint. This limited access further complicates the effective implementation of minimally invasive procedures on the hip joint.

[0112] In addition to the above, the characteristics and location of hip joint lesions (such as conditions or disorders that may cause the joint to deviate from its baseline anatomy) also complicate the implementation of minimally invasive procedures. For example, in the case of a typical tear of the labrum in the hip joint, an approach suture is often required to guide instruments into the joint space, and in some locations the approach suture is set at an angle of 25 degrees or greater to the repair suture. This makes drilling into the bone, for example, far more complex than in cases where the approach suture is effectively aligned with the repair suture (as is often the case in the shoulder joint). Furthermore, the working space within the hip joint is generally extremely limited, which further complicates repairs where the approach suture is not aligned with the repair suture.

[0113] For the reasons mentioned above, minimally invasive hip joint procedures remain relatively difficult, and patients often have to manage their hip joint disease for as long as possible until partial or total hip replacement can no longer be avoided. The procedure is then usually performed as a highly invasive open procedure, with all the disadvantages associated with highly invasive open procedures.

[0114] As mentioned above, hip arthroscopy is becoming increasingly common in the diagnosis and treatment of various hip lesions. However, due to the anatomy of the hip joint and the lesions associated with it, hip arthroscopy currently appears to be applicable only to selected lesions, and even so, it generally achieves limited success.

[0115] One procedure sometimes attempted arthroscopically involves femoral debridement for the treatment of cam-type femoral-acetabular impingement syndrome (FAI). More specifically, in cam-type FAI, irregularities in the femoral geometry can lead to impingement between the femur and the acetabular cup rim. Treatment for cam-type FAI generally involves debridement of the femoral neck and / or femoral head using instruments such as awls and chisels to remove the bone deformity causing the impingement. It is important to debride the femur carefully, as only bone that does not conform to the desired geometry should be removed to ensure a positive outcome and minimize the possibility of fracture after treatment. Therefore, when debridement is performed as an open surgical procedure, surgeons typically use a debridement template with pre-shaped curvature to guide them in removing an appropriate amount of bone from the femur.

[0116] However, when attempting debridement procedures arthroscopically, conventional debridement templates with pre-shaped curvatures cannot pass through narrow keyhole incisions, and therefore the templates often fail to guide the surgeon in reshaping the bone surface. Thus, debridement usually must be performed "freehand." In addition to the above, the field of vision for cam lesions is generally limited. First, surgeons use endoscopes and cameras to view the resection area, but the endoscopic images have a limited field of view and are somewhat distorted. Furthermore, because the endoscope is placed close to the bone surface, the surgeon cannot view the entire lesion "at once." Second, surgeons also use a fluoroscopic screen to take X-ray images of the anatomical structures. These X-ray images supplement the arthroscopic view from the endoscope, but are still limited to a two-dimensional representation of the three-dimensional cam lesion.

[0117] For the reasons mentioned above, surgeons often find it difficult to determine precisely the amount of bone that should be removed, and whether the remaining bone will have the desired geometry. In practice, surgeons tend to be overly cautious and remove less bone. Importantly, inadequate resection of cam lesions is a major cause of hip arthroscopic revision surgery.

[0118] Sometimes, an example of another procedure attempted arthroscopically involves the treatment of clamp-type femoral-acetabular impingement syndrome (i.e., clamp-type FAI). More specifically, in clamp-type femoral-acetabular impingement syndrome, irregularities in the acetabular geometry can lead to impingement between the femur and the rim of the acetabular cup. Treatment for clamp-type femoral-acetabular impingement syndrome generally involves debridement of the acetabular cup rim using instruments such as a drill bit and a chisel to remove the bone deformity causing the impingement. In some cases, the labrum is released from the acetabulum before debridement of the acetabular cup rim, thus exposing the underlying acetabular cup rim, and then the labrum is reattached to the debridemented acetabular cup rim. It is important to carefully debride the acetabular cup rim, as only bone that does not conform to the desired geometry should be removed to alleviate the impingement while minimizing the possibility of removing too much bone from the acetabular cup rim (which can lead to joint instability).

[0119] However, when attempting debridement procedures arthroscopically, debridement often must be performed manually. In such cases, it is often difficult for surgeons to accurately determine the exact amount of bone to be removed and whether the remaining bone has the desired geometry. In practice, surgeons tend to be overly cautious and remove less bone. Importantly, insufficient removal of the lesion by clamping may necessitate hip arthroscopic revision.

[0120] Two common anatomical measurements used to diagnose femoral-acetabular impingement syndrome (FAI) are the α angle for cam-type impingement. Figure 16 ) and the center edge angle for clamp-type impact ( Figure 17These measurements are typically taken from preoperative images (e.g., preoperative X-ray images). These measurements are used to determine the extent to which a patient's hip anatomy deviates from normal (e.g., baseline) healthy hip anatomy.

[0121] For example, a healthy hip typically has an alpha angle of less than about 42 degrees to about 50 degrees at various locations; therefore, patients with an alpha angle greater than about 42 degrees to about 50 degrees may be candidates for FAI surgery. These are merely exemplary ranges of alpha angle and do not limit the systems and methods described herein to any particular range of alpha angle. During the initial examination of a patient, the surgeon will typically take an X-ray of the patient's hip. If the patient is initially diagnosed with FAI, they may also receive an MRI or CT scan of their hip for further evaluation of any bone lesions leading to the FAI.

[0122] Most modern imaging techniques (such as X-rays, CT scans, and MRI) are digital, and therefore the images can be imported into and manipulated by computer software. Using the imported digital images, surgeons can measure the alpha angle (and / or the central-marginal angle). For example, surgeons import digital images into one of the many available software programs that use the DICOM (Digital Imaging and Communications in Medicine) standard for medical imaging. To measure the alpha angle (or central-marginal angle) using the digital image, the surgeon must first manually create the geometry and overlay it onto the digital medical image.

[0123] For example, and see Figure 16 To measure the α angle, the surgeon manually creates a circle 5 and places it on the femoral head 10, then manually determines the size of the circle so that its edges match the edges of the femoral head. The surgeon then manually creates a line 15 and places it along the midline of the femoral neck 20. The surgeon then manually draws a second line 25, which begins at the center of the femoral head and passes through a location 30 indicating the onset of the cam lesion (i.e., where the bone first extends to the outside of the circle defined around the femoral head). The surgeon then manually selects these two lines and instructs the software to calculate the angle between them; the result is the α angle 35.

[0124] Accordingly, and see Figure 17 To measure the central edge angle, the surgeon manually creates a vertical line 40, which begins at the center of the femoral head and is perpendicular to the transverse axis of the pelvis. The surgeon then manually draws a second line 45, which begins at the center of the femoral head and passes through the location 50, indicating the start of the clamping lesion (i.e., the edge of the acetabular cup). The surgeon then manually selects these two lines and instructs the software to calculate the angle between them; the result is the central edge angle 55.

[0125] These alpha angle measurements (or central-marginal angle measurements) are typically performed around the time the patient is initially examined, usually weeks or months before surgery. During surgery, the surgeon can take a copy of the alpha angle measurements (or central-marginal angle measurements) (e.g., a printout) to the operating room so that the printout can be used as a reference during the procedure. The surgeon can also access these measurements using a computer located at or near the operating room, connected to the hospital's PACS (Picture Archiving and Communication System). Either way, the surgeon can use the preoperative measurements as a reference during surgery.

[0126] However, when surgeons are debridement of bone on a cam (or clamp), preoperative measurements may not be sufficient to adequately guide them regarding the location and amount of bone to be removed, as it is difficult to compare what the surgeon sees in endoscopic images with preoperative measurements. Therefore, as further discussed below with respect to various implementations, systems and methods can guide surgeons during joint procedures by overlaying a three-dimensional image representation of the planned bone removal onto two-dimensional images of the joint captured during the procedure. The three-dimensional representation of the planned bone removal indicates the location of the bone to be removed from the joint in three-dimensional space, allowing the surgeon to better understand how the planned bone removal relates to what the surgeon sees via endoscopic imaging.

[0127] Figure 18 The description includes surgical kits, according to some embodiments, for guiding a surgeon to remove bone from a portion of a joint during a surgical procedure. In a typical arthroscopic surgical kit, the surgeon uses an arthroscope 105 and a monitor 110 to directly view the internal surgical site. Additionally, the surgeon may use a C-arm X-ray machine 115 and a fluorescein monitor 120 to image the internal surgical site. According to various embodiments, the surgical kit may include a visual guidance system 125 for guiding the surgeon during a surgical procedure according to the principles described herein, which generates overlay images in which a representation of bone removal extracted from a three-dimensional model of the bone is overlaid on a two-dimensional image of the bone captured intraoperatively, such as by a C-arm X-ray machine 115.

[0128] According to some embodiments, the visual guidance system 125 includes one or more processors, memory, and one or more programs stored in the memory for enabling the visual guidance system to provide the functions disclosed herein. According to some embodiments, the visual guidance system 125 includes a tablet device, such as a touchscreen, with an integrated computer processor and user input / output capabilities. The visual guidance system 125 may be located at least partially in a sterile area; for example, the visual guidance system 125 may include a touchscreen tablet computer mounted on an operating table or cantilevered tablet support. The visual guidance system 125 may be covered with sterile drapes to maintain the aseptic state of the surgeon while the surgeon operates the touchscreen tablet computer. The visual guidance system 125 may include other general-purpose computers with appropriate programming and input / output capabilities, such as desktop or laptop computers with a keyboard, mouse, touchscreen display, heads-up display, gesture recognition device, voice activation feature, pupil reading device, etc.

[0129] Figure 19 This describes method 1900, based on some implementation schemes, for guiding a surgeon to remove bone from a portion of a joint during surgical procedures. Method 1900 can be performed using a visual guidance system, such as... Figure 18 A visual guidance system 125 is implemented to display a rendered overlay image of a three-dimensional visualization of planned bone removal for a portion of the joint, the three-dimensional visualization overlaid on a two-dimensional image (e.g., a fluorescence image) of that portion of the joint captured during the surgical procedure. The overlay image can indicate to the surgeon the location and amount of bone that should be removed from the joint portion not captured in the two-dimensional image (i.e., outside the imaging plane). This method can be advantageous to the surgeon by allowing the surgeon to better correlate planned bone removal with what the surgeon sees in endoscopic imaging. Because two-dimensional images of a joint only show a two-dimensional outline of the joint at the imaging plane, surgeons often have difficulty identifying the location in endoscopic imaging that corresponds to the outline in the two-dimensional image and therefore the exact location of the bone that should be removed. Method 1900 enables the surgeon to visualize the location of bone to be removed outside the imaging plane, allowing the surgeon to better compare planned bone removal with what the surgeon sees in endoscopic imaging. Method 1900 can be used for any joint in the body, including the hip, shoulder, knee, spinal joints, etc.

[0130] Typically, as discussed further below, the rendering of planned bone removal is part of a preoperatively generated 3D model 1950 of the patient's joint or a portion thereof, generated from a 3D scan of the patient's joint, such as during the preoperative planning phase. The 3D model 1950 models at least a portion of the joint that is the target of surgical treatment, such as the upper part of the femur or acetabulum. The model also includes a representation of planned bone removal that can identify bone portions deviating from the target joint morphology. For example, in some embodiments, the representation of bone removal is a heatmap of the bone portions associated with the lesion target to be treated surgically. The visualization can indicate the location and, in some embodiments, the amount of bone to be removed during the debridement procedure. Method 1900 typically includes determining the alignment of the 3D model with the 2D image when projected onto a 2D plane, and at least overlaying the representation of planned bone removal from the model onto the 2D image based on the alignment.

[0131] Figure 20 This section describes the general principles of method 1900, an implementation method for debridement of CAM lesions on the femoral head of the hip joint, according to some embodiments. In the illustrated example, a three-dimensional model 2002 models the upper portion of the femur and includes a representation of planned bone removal in the form of a heatmap 2004, which covers portions of bone that deviate from the target morphology and are thus identified as to be removed (e.g., debridement). The heatmap 2004 is color-coded according to the planned bone removal depth. As discussed further below, method 1900 includes a step for aligning the three-dimensional model 2002 such that the projection 2006 of the model onto a two-dimensional plane 2008 corresponding to the imaging plane is aligned with the femur 2010 in the two-dimensional image 2012. Conceptually, according to some embodiments, the model 2002 is manipulated according to available degrees of freedom, such as translation in the x, y, and z directions, rotation about these axes, and relative to the observer (in the viewer). Figure 20 The viewpoint (described in the image) is scaled until its projection 2006 is fully aligned with the femur 2010 in the two-dimensional image 2012. Once this satisfactory alignment is achieved, and continuing to the next step... Figure 20 In the example, the three-dimensional heat map 2004 is projected onto the two-dimensional plane 2008 and rendered as an overlay 2014 on the two-dimensional image 2012 displayed to the surgeon during the operation.

[0132] Before describing the steps of method 1900, various aspects of the three-dimensional model 1950 according to various embodiments will be described. Typically, the three-dimensional model is generated preoperatively using three-dimensional imaging data. This model may include one or more parts of one or more bones of a joint. For example, for the hip joint, the patient's hip joint can be imaged, and a model can be constructed from the image including various parts of the hip joint, such as the femoral head, femoral neck, acetabular cup, pelvis, femoral condyle, etc. Any suitable imaging system can be used, including, for example, CT imaging systems, ultrasound tomography systems, O-arms, etc. TM Imaging systems include cone-beam computed tomography (CBCT) and MRI. The 3D model can then be analyzed to identify potentially surgically treatable portions of the imaged anatomical structures, such as those requiring resection of FAI lesions.

[0133] According to some implementations, systems and methods based on the principles described herein can generate a three-dimensional model of at least a portion of a joint, the model including a representation of at least one region of the joint portion that deviates from the target morphology. This model can be used to assist a physician in planning surgical procedures on at least one region of the joint, for example by indicating the location and amount of bone that the physician should remove. For example, a three-dimensional model of a portion of a subject's hip joint can be generated, including information identifying the location of hip joint lesions (e.g., conditions or disorders, such as femoral artery disease) and the amount of bone that may be removed to match baseline anatomy. The three-dimensional model can be used to generate a rendering of the joint portion, including a visual representation of the location and amount of bone to be removed, such as a heatmap overlaying the deviated bone portion and including variations in color, contrast, or other suitable visual cues indicating the degree of deviation from the target morphology.

[0134] A 3D model and information about deviations from the baseline / target anatomy can be used to generate a 3D rendering of the joint, including a visual representation of the deviations from the baseline / target anatomy. In some implementations, this visual representation can indicate the location and amount of bone that should be removed to achieve the target morphology. In some implementations, users, such as surgeons or third parties, can customize the visual representation for surgical planning purposes, for example, by changing one or more parameters that determine the deviation from the target bone morphology. This can increase or decrease the indication of the size of the area to be removed and / or increase or decrease the indication of the amount of bone to be removed.

[0135] In some implementations, a 3D model is pre-generated and stored for use during the surgical session. In other implementations, the 3D model can be generated intraoperatively, for example, using an O-arm. TM Imaging systems. In some implementations, the 3D model can be updated based on images captured during the surgical procedure to reflect the bone removal during the procedure, which will be discussed further below.

[0136] Return to Figure 19 Method 1900 includes receiving a two-dimensional image of at least a portion of the joint during the surgical procedure at step 1902. The two-dimensional image typically includes the bone portion being surgically treated, as well as surrounding portions of the bone, allowing the surgeon to typically compare what is displayed in the image with what is seen through a surgeon's endoscope. For example, in embodiments involving debridement to address CAM lesions, the two-dimensional image typically includes the femoral head and femoral neck. The two-dimensional image may be received from an intraoperative imaging system, such as an X-ray imager (e.g., [missing information]) communicatively connected to a surgical guidance system implementing method 1900. Figure 18 (C-arm X-ray machine 115). In some embodiments, one or more preprocessing operations are applied to the X-ray image, such as one or more scaling operations, cropping operations, downsampling, upsampling, etc. In some embodiments, a de-distortion operation is applied to the X-ray image to correct distortion caused by the imaging system. In some embodiments, X-ray image de-distortion is performed based on a determined relationship between a known pattern of reference markers attached to the imaging system detector and a visible reference marker in the X-ray image. For example, reference markers in the X-ray image can be detected, a non-rigid transformation mapping the known location of the reference markers to the visible markers in the image can be calculated, and the transformation can be applied to the image, resulting in image de-distortion. In some embodiments, the reference markers can then be removed from the image.

[0137] In step 1904, the two-dimensional image is analyzed to extract one or more features related to the location of the target bone in the two-dimensional image, which can be used to align a three-dimensional model of the bone with the two-dimensional image. According to some embodiments, the extracted features include the edges of the bone in the image, which can be extracted using one or more edge detection techniques. In some embodiments, additional features are extracted based on the detected edges, which may include, for example, the size and location of the bone.

[0138] Figure 21 This describes an implementation of step 1904 for extracting femoral features. Method 2100 can be used to provide visual guidance to a surgeon during femoral debridement to treat CAM femoral injuries.

[0139] In step 2102, a two-dimensional image 2130 of a portion of the joint is displayed on a monitor (e.g., on...). Figure 18 The visual guidance system 125 and / or display 110 are shown to the surgeon. Figure 22 This illustrates an example of displaying a two-dimensional X-ray image of a portion of the hip joint captured by an X-ray imaging system (e.g., a C-arm X-ray machine 115). The displayed image 2130 allows the surgeon to determine whether the image 2130 is a sufficient image (e.g., it is clear enough, and it is in the correct position).

[0140] In some embodiments, at optional step 2104, image 2130 may be displayed as part of a user interface configured to receive “tips” from a surgeon (or other operating room personnel) regarding one or more features of the femur in the image. These “tips” can be used to expedite the identification of bone features in the image. One or more tips from the surgeon may be received via any suitable input, including, for example, touch input to a corresponding portion of a touchscreen display showing the two-dimensional image 2130, or a gesture recognition system incorporated into or connected to an augmented reality or virtual reality system of the vision guidance system 125. In some embodiments, and see also... Figure 22 The surgeon may provide one or more cues related to bone landmarks in the image. For example, a femoral head cue "H" may be provided on the femoral head in the image, and a femoral neck cue "N" may be provided on the femoral neck in the image. In some embodiments, a femoral shaft cue "S" may be provided on the femoral shaft in image 2130. The surgeon may be guided through the process by one or more cues on the display indicating what cues are being sought. In some embodiments, the visual guidance system may be configured to display one or more warnings based on the determination that the cues meet one or more predefined criteria. For example, the system may be configured to determine whether the cues provided by the surgeon are outside a predefined range, or whether the cues provided by the surgeon appear to be in an incorrect area of ​​the image, or whether the cues provided by the surgeon do not enable the system to determine the location of one or more anatomical features based on the cues. If one or more predefined criteria for displaying a warning in response to a received cues are met, the system may display a warning in the form of a pop-up window or drop-down window. In some embodiments, the warning may display information about the criteria that led to the warning being displayed. In some embodiments, the warning may prompt the surgeon to re-enter a new cue, as described above, where receiving the user cues is optional.

[0141] In some implementations, the feature recognition process is fully automated and requires no user guidance. For example, an object detection machine learning model trained to detect one or more features of bone in an image can be used. For instance, an object detection machine learning model can be trained to detect the femoral head, femoral neck, greater trochanter, lesser trochanter, and / or femoral shaft. The object detection machine learning model can utilize a convolutional neural network (CNN), such as R-CNN or YOLO architecture, or any other suitable object detection model. In some implementations, the trained machine learning model provides potential bounding boxes and their classifications (e.g., one or more of the femoral head, femoral neck, femoral shaft, etc.). Post-processing of the machine learning model results determines the bounding box with the highest score for each classification. The scores can be compared to one or more thresholds, and if the scores meet the thresholds, the center of each bounding box can be used as a starting point for further feature detection in a manner similar to the user-provided prompts described above. Figure 35 The description describes the use of a machine learning model trained to detect the femoral head and neck to generate scored bounding boxes according to various implementation schemes. The machine learning model provides bounding boxes 3502 forming the boundary of the femoral head 3504 captured in X-ray image 3500 and bounding boxes 3506 forming the boundary of the femoral neck 3508 in image 3500. The femoral head bounding box 3502 scores 97% and the femoral neck bounding box 3506 scores 96%. If these scores meet predetermined thresholds, the centers of the respective bounding boxes can be used as the starting point for further feature recognition steps to identify the femoral head and neck, as discussed further below. In some implementations, if the scores do not meet the thresholds, the system may prompt the user with the prompts described above.

[0142] Method 2100 continues with step 2106, in which edge detection is performed on the two-dimensional image 2130 to identify the edges of at least a portion of the femur in the image, such as the femoral head, femoral neck, greater trochanter, lesser trochanter, and / or any other part of the femur. Various methods exist for implementing this edge detection step, including industry-standard methods such as Sobel, Canny, and Scharr edge detection methods. Figure 23 This section illustrates examples of results from edge detection algorithms used for the upper femur based on certain implementation schemes.

[0143] After edge detection is completed, it may be advantageous to find and remove the edges of any instruments in the search area in optional step 2108, as the presence of instrument edges in the image can complicate subsequent processing steps (e.g., finding the femoral head, finding the femoral neck, etc.). The finding and removal of instrument edges can be implemented using algorithms well-known in the field of image processing.

[0144] In some implementations, in step 2110, the center of the femoral head is determined based on the edges detected in step 2106. According to some implementations, the center of the femoral head can be detected by first detecting the outer edge of the femoral head in the image. According to some implementations, this can be done using a Hough transform, which searches for circles that match the edge of the femoral head. These circles can be limited to the range of the largest and smallest possible femoral heads. The Hough transform produces a list of possible answers, and the best possible answer is selected. Figure 24 This illustrates an example of a circle 2400 around the edge of the femoral head detected by edge detection based on a Hough transform according to an implementation scheme.

[0145] Although the Hough transform is relatively fast, it may not be as accurate as desired because the femoral head may not be a perfect circle. Therefore, in some implementations, an alternative method is used where a center point is selected (e.g., using a hint provided by the surgeon in step 2104) and then the visual guidance system 125 begins tracing along lines searching for the edge between the minimum and maximum possible radii (which are related to the minimum and maximum possible femoral heads). In this method, the visual guidance system 125 selects points with the strongest edges in each ray and then checks whether these points end in a circle. Another point is then selected, and the process is repeated. This is done iteratively, using previous points as guides for the next search, until an optimal point is found. Any other suitable techniques can be used to locate the outer edge of the femoral head, including machine learning models trained on images with similar anatomical structures.

[0146] Once the femoral head is identified, its center in the x and y dimensions can be determined, such as... Figure 24 Finally, as shown in 2402. In some embodiments, the radius of the femoral head can also be determined in step 2112 by measuring the distance from the center of the femoral head to the outer edge of the femoral head, such as... Figure 24 As shown in 2404.

[0147] In some implementations, method 2100 continues with step 2114, in which the midline of the femoral neck is identified. Figure 24 In the example, the midline is indicated by 2406. There are several ways to locate the femoral neck; however, once the femoral head is identified, locating the femoral neck is usually easier. In some implementations, a box sweep method is used to locate the femoral neck. A box is swept around the femoral head (the box's center line passes through the center of the femoral head) until the sides of the box align with the edges of the femoral neck (as identified by edge detection). This process is repeated for boxes of various sizes. The box aligned with the strongest edge of the femoral neck can be selected. The center of the box is then used to determine the midline of the femoral neck.

[0148] The method 2100 generates a set of features 2150 of bone in a two-dimensional image, which can be used to align a three-dimensional model with the two-dimensional image. As described above, the set of features 2150 may include at least the edges of the femoral head and femoral neck in the two-dimensional image, the coordinates of the center of the femoral head in x and y, the radius of the femoral head, and the direction of the midline of the femoral neck. These features are only examples of identifiable features of the superior femur. Various embodiments may include one or more of these identified features and / or one or more additional features, such as the midline of the femoral shaft, the superior trochanter, and / or the inferior trochanter.

[0149] Method 2100 is merely an example of feature detection step 1904 of method 1900 when applied to the femur. According to various embodiments, feature detection can be performed multiple times throughout method 1900. For example, a first feature detection step can be performed at the initial stage of the alignment determination process to provide a starting point for alignment, as discussed further below, and a second feature detection step can be performed at a later stage when refining the model alignment. Therefore, multiple feature detection steps can be performed during iterative feature detection and alignment processes. Feature detection can be performed at different times and in different ways. For example, the initial feature detection step may detect edges, and may also detect features derived from these edges, such as the location of the femoral head center, while subsequent feature extraction steps may only include edge detection. According to some embodiments, feature extraction may be limited to simple edge detection using one or more edge detection algorithms as described above.

[0150] In some embodiments, one or more features of the pelvis may be detected in step 1904. For example, pelvic features may be associated with at least one of the acetabular center, obturator foramen, and pubic symphysis. In some embodiments, method 1900 is applied to the knee joint, and one or more features may be associated with at least one of the tibial plateau, tibial shaft, and intercondylar eminence. In some embodiments, method 1900 is applied to one or more vertebrae, and one or more features may be associated with at least one of the pedicle, facet, superior endplate, and inferior endplate.

[0151] Return to Figure 19Once bone features are detected in the 2D image in step 1904, one or more of the detected features can be used in step 1906 to determine the alignment of the 3D model 1950 with the 2D image. Typically, determining the alignment involves manipulating the 3D model 1950 according to the available spatial degrees of freedom until the projection of the model onto the 2D plane aligns with the bones in the 2D image. According to some embodiments, the features extracted from the 2D model can be as simple as the bone contour in the 2D image (e.g., extracted using edge detection), and the alignment process can include iteratively repositioning the 3D image until the projection of the 3D model aligns with the bone contour. According to some embodiments, additional features detected in the 2D image are used to provide an initial estimate of the model alignment, which can help reduce the processing time required to find the correct alignment.

[0152] Figure 25 This describes method 2500 for determining the alignment of a 3D model with a 2D image according to some embodiments. In step 2502, the initial position and orientation of the 3D model are set as the starting point for an iterative process to determine the alignment of the 3D model with the 2D image when projected onto a 2D plane. Setting the initial position and orientation may include setting the available degrees of freedom of the model, including three positional degrees of freedom, three rotational degrees of freedom, and the scale of the model. In some embodiments, multiple initial positions and orientations of the model are set, allowing for parallel analysis of multiple positions.

[0153] In some implementations, the initial position and orientation of the 3D model are set according to one or more predefined criteria. For example, in some implementations, the initial position and orientation are set according to default values ​​associated with the 3D model, such as by setting all translations and rotations to zero and scaling to a value of one. In some implementations, the initial position may be set according to one or more positions that the anatomical structure can generally take during the surgical procedure. For example, the initial position and orientation of the 3D model may be set according to the position and orientation of the anatomical structure when it is in a position that the body typically takes during the surgical procedure. For example, in some medical procedures, it may be standard practice for the physician to capture anatomical images, such as X-ray images, in a predefined patient position and / or from a predefined angle. For example, for camectomy procedures, imaging is generally performed in six standard patient positions to ensure evaluation of all potential cam lesion locations on the femur, as described in the article entitled “Intraoperative Fluoroscopic Imaging to Treat Cam Deformities: Correlation with 3-Dimensional Computed Tomography” by James R. Ross, published online in The American Journal of Sports Medicine on April 15, 2014, the entire contents of which are hereby incorporated by reference. As described in Ross's article, the six standard positions are: knee and hip fully extended (i.e., 0 degrees of flexion) with the leg (1) 30 degrees of internal rotation, (2) neutral rotation, and (3) 30 degrees of external rotation; and hip and knee in 50 degrees of flexion with the hip in (4) neutral rotation, (5) 40 degrees of external rotation, and (6) 60 degrees of external rotation. Therefore, in some implementations, the initial position and orientation of the 3D model are set according to one or more of these six standard positions.

[0154] In some implementations, the initial position and orientation are set based on one or more features 2550 determined from a two-dimensional image. For example, estimation of the center and / or midline of a bone in the image can be used to center the corresponding bone in a three-dimensional image. In some implementations, the initial position and orientation are set based on a combination of one or more predefined criteria and one or more features determined from a two-dimensional image. For example, the determined features can be used to set one or more, but less than all, degrees of freedom, and the predefined criteria can be used to set the remaining degrees of freedom.

[0155] Once the initial position and orientation of the 3D model are set for the number of initial model alignments (multiple initial alignments can be used to speed up the process), the model is projected onto a 2D plane in step 2504 for each alignment. As used herein, projecting the model onto a 2D plane means finding the contour edges of the model relative to the model position and orientation set in step 2502. Projecting the 3D model onto a 2D plane produces the 2D contour of the bone. For implementations where multiple positions are analyzed in parallel, projections are generated for each initial position and orientation set in step 2502.

[0156] In step 2506, the projection generated in step 2504 is compared with one or more features 2550 determined from the 2D image. In some embodiments, the features determined from the 2D image are edges detected in the 2D image, and the projection is compared with the detected edges to determine the similarity between the edges in the projection and the detected edges. This edge comparison may include generating a similarity metric that compares the projection with the detected edges. In some embodiments, the similarity metric is an edge-based similarity metric. An edge-based similarity metric may take into account any suitable edge attributes detected in the 2D image, such as edge strength, the direction the edge faces, and continuity with neighboring edges, and may compare one or more detected edge attributes with corresponding parts of the 3D model. In some embodiments, the edge-based similarity metric is based on a weighted average of the distances from points on the edges in the projection to the nearest edges extracted from the 2D image, and produces numerical values ​​associated with these distances. In some embodiments, the edge-based similarity metric is based on the strength of the edges extracted from the 2D image for each edge in the projection. In some embodiments, the edge-based similarity metric is based on the matching angle between the edges extracted from the 2D image and the corresponding edges in the projection. In an implementation where multiple locations are analyzed in parallel, each projection is compared with features extracted from a two-dimensional image.

[0157] In step 2508, a decision can be made as to whether to continue searching for a suitable alignment of the 3D model or to complete the alignment step. In some embodiments, this decision may be based on the result of step 2506, such as by determining whether the result of step 2506 indicates a sufficient alignment. For example, in some embodiments, step 2506 may generate a similarity metric that is compared to a predefined threshold to determine whether the alignment is close enough or whether further iterations are expected. In some embodiments, the decision to continue iterations is based on predefined criteria, such as whether a predefined number of iterations has been completed.

[0158] If the decision in step 2508 is to continue, then a new model position and orientation are set in step 2510. The new model position and orientation can be set by moving a fixed amount relative to the previous position and orientation in one or more degrees of freedom. In embodiments where multiple positions and orientations are analyzed in parallel, a position and orientation that best matches the features of the two-dimensional model can be selected in step 2510, and the model can be perturbed relative to that selected position and orientation. For example, a position and orientation that produces the best similarity score in step 2506 can be selected, and the model can be repositioned relative to the selected position and orientation by a predetermined amount in one or more degrees of freedom. In some embodiments, multiple new model positions and orientations are set. For example, the model can be moved a fixed amount in a first degree of freedom relative to the selected position and orientation to provide a first new model position and orientation, and moved a fixed amount in a second degree of freedom to provide a second new model position and orientation.

[0159] In step 2510, given the new model position and orientation (or multiple new positions and orientations), the process returns to step 2504, where each new model alignment is projected onto the two-dimensional plane. This process continues in a loop until a decision in step 2508 is made to stop iteration, for example, because the resulting projections are sufficiently close or because a predetermined number of iterations has been completed, resulting in a determined model alignment 2575 that produces a projection that is fully aligned with the two-dimensional image.

[0160] Figure 26 One embodiment of the application of method 2500 is described, which is used to align a three-dimensional model of at least the upper part of a subject's femur with a two-dimensional image of at least the upper part of the subject's femur captured during a surgical procedure on the femur (e.g., during bone debridement of the femur) to correct CAM-type FAI.

[0161] Starting at step 2602 of method 2600, the initial alignment of the intermediate model 2650 is set based on features 2660 extracted from the two-dimensional image 2630, such features being, for example, derived from... Figure 21 Method 2100 is used to generate the initial alignment. The initial alignment can be set according to seven degrees of freedom, which may include three translational degrees of freedom, three rotational degrees of freedom, and scaling. For illustrative purposes, the translational degrees of freedom related to translation within the projection plane are referred to herein as the x and y directions. The z direction extends orthogonally to the projection plane.

[0162] In some implementations, features used for initial alignment include the location of the femoral head center in the 2D image, the direction of the femoral neck midline, and the radius of the femoral head. The center of the femoral head in the image can be used to set the x and y positions of the model. The model is conceptually manipulated relative to the projection plane such that the femoral head center in the model is positioned relative to the projection plane at the same location as the femoral head center in the 2D image. For example, according to some implementations, if the femoral head center in the 2D image is at the center of the image, the model is manipulated relative to the projection plane such that the femoral head center of the model is projected onto the center of the projection plane.

[0163] The model is positioned in the z-direction to obtain bone dimensions (e.g., in pixels) that match the bone dimensions in the two-dimensional image.

[0164] The model is then manipulated relative to the projection plane so that the midline of the femoral neck extends from the center of the femoral head in the same direction as the midline in the two-dimensional image. This step can be considered as rotating the model around the z-axis until the midline is in the same position relative to the projection plane as the femoral neck midline in the two-dimensional image.

[0165] Through these initial alignment steps, the model's position approximates the bone's position in the two-dimensional image in terms of x, y, and z positions, scale, and rotation about the z-axis. In this example, the degree of freedom not approximated by features determined from the image is rotation about axes extending in the two-dimensional projection plane, i.e., rotation about the x and y axes. By defining one of the x and y axes as extending through the midline of the femoral neck, the two unknown degrees of freedom are rotation about an axis extending through the midline of the femoral neck and rotation about an axis located in the projection plane and orthogonal to the midline axis.

[0166] According to some implementation schemes, the degrees of freedom for model alignment are... Figure 27 The x and y positions are derived from the x and y positions of the femoral head center in the 2D image. The z position is determined from the femoral head dimensions from the 2D image, known parameters of the imaging system, and known femoral head dimensions from the 3D model. For example, since the ratio of the z position to the focal length of the imaging system (the distance from the transmitter to the detector) is equal to the ratio of the actual diameter of the femoral head to the diameter of the femoral head in the image, the z position can be determined by multiplying the focal length by the ratio of the actual diameter of the femoral head in the 3D model to the diameter of the femoral head in the image. The diameter of the femoral head in the image can be determined by scaling the pixel diameter of the femoral head in the image to the ratio of the detector diameter (e.g., in millimeters) to the detector diameter (in pixels). The scale is derived from the femoral head radius from the image. The rotation about the z-axis (perpendicular to the imaging / projection plane, "yaw") is derived from the direction of the femoral neck midline, indicated by reference numeral 2702. The remaining degrees of freedom are rotations about an axis extending through the femoral neck midline. Figure 27The axis marked as y-axis ("pitch") and rotation about an axis extending orthogonal to the midline of the femoral neck ( Figure 27 The middle label is marked as the x-axis ("roll").

[0167] In some implementations, the initial alignment is determined based on one or more objects detected in a 2D image by one or more object detection machine learning models, along with assumptions about the positions of those objects in different joint configurations. In implementations related to the femur of the hip joint, the object detection machine learning model may be configured to detect one or more of the femoral neck, femoral shaft, and lesser trochanter. The femur appears to have an obtuse angle between the femoral neck and femoral shaft in the anterior-posterior view, and this angle becomes more obtuse as the femur moves toward the Dunn view, until the femoral neck and femoral shaft become collinear. Additionally, the lesser trochanter is visible in the anterior-posterior view but not in the Dunn view. Based on this understanding of the patient's anatomy, if the object detection machine learning model does not see the lesser trochanter, it can be assumed that the anterior-posterior view is not an imaging view. If the angle between the femoral neck and femoral shaft is obtuse, then the anterior-posterior view is more likely an imaging view. Therefore, objects detected by one or more object detection machine learning models can be used to provide initial inferences about rotations around the x and y axes. This can accelerate the algorithm by reducing the input used and the search space to achieve a good fit.

[0168] Once the initial alignment based on features extracted from the 2D image is completed, one or more inferences can be made for the remaining degrees of freedom in step 2604. According to some implementations, several model alignments are evaluated in parallel. Therefore, inferences are made for several combinations of the remaining degrees of freedom in step 2604. The inferences for the remaining degrees of freedom can be based on, for example, predictions about how the femur might be positioned. Since the range of motion of the leg is limited, the inferences for the remaining degrees of freedom can be constrained based on range of motion limitations. For example, minimum and maximum values ​​for the remaining degrees of freedom can be assumed, and multiple values ​​within these minimum and maximum values ​​can be selected.

[0169] In some implementations, the values ​​of the remaining degrees of freedom are selected based on predictions of possible bone locations during the surgical procedure. For example, in surgical procedures addressing CAM-type FAI, the leg is typically positioned in one of six locations identified for X-ray imaging of the femoral portion requiring debridement. Therefore, an initial set of predictions can be associated with these six common leg locations.

[0170] Once the initial assumptions for the remaining degrees of freedom are set, one or more alignments of the model are fully defined. As mentioned above, different model alignments can be analyzed in parallel, and therefore, the result of step 2604 can be multiple initial model alignments. From each model alignment, in step 2606, the contour of the model is generated by projecting the aligned model onto a projection plane corresponding to the image. The resulting contour is the two-dimensional contour of the bone in the three-dimensional model.

[0171] In step 2608, each contour is compared with features extracted from the 2D image. For example, the contour may be compared with a group of edges detected in the 2D image by edge detection, such as... Figure 23 As shown. According to some implementations, the comparison can be based on a similarity algorithm that compares one set of edges (contours) with another set of edges (edges detected in a two-dimensional image), and the comparison result can include a numerical value related to similarity (or dissimilarity). Any suitable similarity algorithm can be used, including, for example, edge alignment metrics based on Canny, Sobel, and Scharr edge detection algorithms.

[0172] In step 2610, one or more contours that produced the best similarity score in step 2608 can be selected (multiple contours can be selected to speed up the search). In step 2612, a decision is made as to whether to continue searching for a suitable alignment. In some embodiments, the decision to continue may be based on the similarity of the alignment selected in step 2610, such as whether the similarity meets a predefined threshold, or may be based on completing a fixed number of iterations.

[0173] If a decision is made to continue refining the alignment, the process proceeds to step 2614, where one or more contours selected in step 2610 are shifted in one or more degrees of freedom, resulting in one or more shifted or "refined" contours. In some embodiments, the contours are shifted by different amounts in one or more degrees of freedom to generate multiple shifted contours for each selected contour.

[0174] According to some implementations, the profile is shifted only in one or more degrees of freedom within the projection plane. For example, the x and / or y positions of the shiftable profile (in-plane translational degrees of freedom) and / or the rotation of the shiftable profile about the z-axis (in-plane rotational degrees of freedom). In some implementations, the amount of shift of the profile in a given in-plane degree of freedom can be predefined. For example, the profile can be shifted a predefined distance in the x and / or y directions and / or shifted an predefined angle about the z-axis. In some implementations, the amount of shift of the profile in a given degree of freedom can be based on the comparison result in step 2608. According to some implementations, the comparison algorithm can provide information about how far a given edge in the model is from a given edge in the 2D model, and the amount of shift in step 2614 can be selected based on the distance information.

[0175] In step 2616, each shifted contour generated in step 2614 is compared with features extracted from the 2D image in a manner similar to that in step 2608. For example, a similarity algorithm is used to compare the shifted contour with a set of edges extracted from the 2D image via edge detection. This step may result in a similarity (or dissimilarity) score for each shifted contour.

[0176] In step 2618, the shift profile with the best similarity score is identified, and the model is realigned based on the degrees of freedom values ​​associated with the shift profile (more than one best shift profile may be selected, as in other steps). In other words, the model is aligned based on the x, y, z, roll, pitch, yaw, and scale values ​​associated with the shift profile with the best similarity score. Then, in step 2620, the realigned model is perturbed by a given amount in the roll and pitch degrees of freedom, resulting in a new model alignment. For example, the model may be perturbed by a small amount of roll and / or pitch. The model may be perturbed by different amounts in the roll and / or pitch directions to generate multiple new model alignments. Furthermore, hill climbing can be used to find the best fit in a shorter time.

[0177] Method 2600 then returns to step 2606, where the new model is aligned and projected onto the projection plane to generate a new set of contours. In step 2608, these new contours are compared with features of the 2D image, such as edge groups detected in the 2D image, and in step 2610, the contour with the best similarity score is identified. Then, in step 2612, a decision is made on whether to continue searching for an appropriate alignment of the model to generate contours that match the bones in the 2D image. As described above, this decision may be based on a similarity score that meets predefined criteria (e.g., above a threshold similarity) or on a fixed number of iterations. If the decision to continue searching is made, the process continues, accompanied by shifting the contours in the projection plane in step 2614, comparing these shifts with features of the 2D image in step 2616, selecting the closest match in step 2618, perturbing the roll and / or pitch degrees of freedom to generate a new set of alignments, and so on, until a decision is made to end the search for an appropriate alignment.

[0178] Once it is decided to end the search for a suitable alignment, a set of values ​​for the 7 degrees of freedom associated with the best alignment produced by method 2600 is provided as model alignment 2675 for generating the overlay image.

[0179] In some implementations, a determined model alignment (e.g., model alignment 2675) is used to provide information about the orientation of patient anatomy, such as in a 2D image capture. For example, if the femoral head has been imaged, the orientation of the leg relative to the patient can be determined based on the model alignment, and this information can be provided to the user. Once the model alignment is known, information about the perspective view of the generated 2D image relative to the patient's body can be used in conjunction with the determined model alignment to determine the relative orientation of the anatomical structures when the 2D image was captured. For example, if the 2D imager position is known, the determined alignment of the 3D image of the femoral head can be used to determine the degree of abduction / adduction, flexion / extension, and / or internal / external rotation of the leg. In some implementations, the orientation of the anatomical structures is determined based on predefined assumptions about the imager position (e.g., assuming the C-arm X-ray imager is oriented in an emitter-to-detector direction, vertically upward relative to the patient, and the top of the image is above the patient). In other implementations, information about the imager position when the 2D image was generated is received and used to determine the orientation of the anatomical structures. The information can be received in any suitable manner, including from user input or from the imager itself.

[0180] In some implementations, information about the orientation of determined anatomical structures can be displayed to the user (e.g., to the surgeon during surgery), such as by adding it to an overlay image. Providing information about the orientation of anatomical structures can help the user record the surgical procedure and / or help the user evaluate the accuracy of the determined model alignment. For example, if the orientation of a determined anatomical structure is inaccurate (e.g., because the leg cannot be in the determined location), the user can understand that the determined model alignment is inaccurate and should be repeated, such as on a new 2D image and / or after adjustments by one or more users (see below regarding...). Figure 29A -Discussion on user adjustments for B).

[0181] In some implementations, the alignment of the 3D model with the 2D image in step 1906 can be determined based on the image segmentation performed in step 1904. According to some implementations, in step 1904, a semantic segmentation algorithm (e.g., U-net) trained to detect target bones (e.g., the femur) is used to segment bones (e.g., the femur) in the 2D image, thereby generating a mask. The mask is the feature extracted in step 1904. In step 1906, the 3D model is projected into 2D space and the projection is compared with the mask. Different projections are scored as the intersection of the projection and the mask divided by the union of the projection and the mask. The alignment of the projection with the highest score can be the alignment of the bone in the 2D image.

[0182] In some implementations, steps 1904 and 1906 are functionally implemented by a machine learning algorithm trained to determine the alignment of the 3D model with the 2D image based on the 3D model and the 2D image, as well as known parameters of the imaging system (e.g., focal length, detector diameter, etc.). Figure 37 This describes an exemplary machine learning algorithm 3700 for determining the alignment of a 3D model 3702 (e.g., CT, MRI, segmented 3D model, etc.) with a 2D image 3704 (e.g., X-ray, fluorescent screen, etc.). The machine learning algorithm includes a 3D convolutional network 3706 for the 3D model 3702. The 3D convolutional network 3706 may include one or more 3D convolutional layers, pooling layers, ReLU (Rectified Linear Unit) layers, and / or other convolutional neural network layers. The machine learning algorithm includes a 2D convolutional network 3708 for the 2D model 3704. The 2D convolutional network 3708 may include a series of convolution and subsampling steps. The 3D convolutional network 3706 and the 2D convolutional network 3708 are connected to an output 3710 via a fully connected dense layer 3712. In some embodiments, one or more parameters 3714 associated with the 2D imaging system used to generate the 2D image 3704 may be provided as input to the fully connected dense layer 3712. For example, for a C-arm fluorescence imager, focal length, detector size, etc., can be the inputs to a fully connected dense layer 3712. The output 3710 of the machine learning algorithm is a transformation matrix (x, y, z, u, v, and w transformations) used for aligning the 3D model with the 2D image.

[0183] Return to Figure 19 Method 1900, once the model alignment is determined in step 1906 (e.g., via...) Figure 25 Method 2500 Figure 26 Method 2600 or via Figure 37 If the machine learning algorithm is used, then in step 1908, an overlay image is generated by overlaying the rendering of at least a portion of the 3D model onto the 2D image. A projection of the model aligned according to the one determined in step 1906 is generated, and at least a portion of the projection is overlaid onto the 2D image, which is then displayed in step 1910.

[0184] As described above, the three-dimensional model includes a representation of bone portions deviating from the target bone state. This representation can serve as a plan for surgical procedures, instructing the surgeon on the location and amount of bone that should be removed to achieve the target bone state. In step 1908, at least a portion of the representation of the planned bone removal is included in the overlay to provide the surgeon with information about the bone to be removed. In some embodiments, the model includes bone portions other than those planned for bone removal, and the overlay may include at least some of these bone portions. In some embodiments, the overlay includes only the representation of the planned bone removal.

[0185] Figure 28AThis describes the overlay of a rendering 2802 of a three-dimensional model of the upper femur projected onto a two-dimensional image 2804 of the hip joint, according to one embodiment. The rendering 2802 includes a heatmap representation 2806 of the planned bone removal and 2808 of the bone portion outside the planned bone removal. Figure 28A The heatmap shown is merely an example of a planned bone removal representation. Any suitable representation may be used, including, for example, an outline of the planned bone removal area, a contour map indicating the amount of bone removal, or any combination of such visual aids. Figure 28B This describes the overlay rendering of the projection of the three-dimensional model of the upper femur, where only the planned bone removal representation 2806 is overlaid on the two-dimensional image.

[0186] According to some implementations, additional information may also be displayed in the overlay, including, for example, clock face line 2810, which may be a virtual object from a three-dimensional model. As is known in the art, clock face lines are useful for surgeons to identify positions within the hip joint (e.g., for identifying rotational positions around the femoral head, acetabular cup, etc.). Clock face line 2810 indicates the 12 o'clock position (…). Figure 28A The 3 o'clock position is also shown. In some embodiments, the α-angle line 2812 may also be displayed. The α-angle line 2812 may represent a set of circumferential positions, where the bone first extends outside the best-fit sphere (or multiple best-fit circles) around the femoral head, as described above. In some embodiments, the target α-angle line may also be displayed in the overlay to indicate the target bone morphology. In the 3D model, the projection of a circle representing the α-angle target as it rotates around the femoral head can also be useful.

[0187] For surgeons, knowing the location where a cam lesion begins in the bone portion captured in a two-dimensional image as a function of the clock face (i.e., the clock face position of the point defining the α angle in the two-dimensional image, referred to here as the α angle point or α point) can be beneficial. This information can be determined from an aligned three-dimensional model whose position and orientation correspond to the patient's position and orientation when the two-dimensional image was captured. Therefore, in some embodiments, an indicator 2814 (indicating the start of the cam lesion, see [reference]) may be included in the overlay of the clock face position of the α angle point on the bone edge in the two-dimensional image 2804 relative to a predefined clock face position. Figure 16 (and related discussions). For example, in Figure 28A In the image, indicator 2814 indicates that the α corner point in the two-dimensional image is at -30° to 12 o'clock (in some implementations, the clock face position, such as 11 o'clock, may be displayed).

[0188] According to some implementations, the aligned 3D model can be used in conjunction with measurements from a 2D image to estimate the clock face position of the α-corner point. In some implementations, the α-corner point in the 2D image is determined by analyzing the 2D image, for example by performing edge detection, and using the results to: (1) locate the femoral head in the 2D image (e.g., using Hough transform, ray tracing, and / or active shape modeling); (2) locate the femoral neck and its midline (e.g., using sweep frame method and / or active shape modeling); and (3) locate the location where the femoral neck stops at the beginning of the circle and cam lesion (i.e., the α-corner point). In some implementations, locating the α-corner point may include tracing the strongest edges of the bone surface (e.g., using the results of edge detection) until a deviation from the circle surrounding the femoral head is detected (see, for example...). Figure 16 (α-corner point 35 in the image). Once the α-corner point is located in the 2D image, point 2816 is determined in the aligned 3D model that projects onto (or near) the α-corner point in the 2D image (this point 2816 may correspond to the intersection of α-corner line 2812 and the horizontal line of the 3D model). Next, a plane including point 2816, the center of the femoral head in the 3D model, and the center of the femoral neck in the 3D model is determined. The angle between this plane and the 12 o'clock plane is determined, where the 12 o'clock plane is defined as the plane in the 3D model that includes the highest point of the femoral head in the 3D model, the center of the femoral head in the 3D model, and the center of the femoral neck in the 3D model. Figure 28A In the image, the angle between the 12 o'clock plane and the clock face plane including the α-angle point 2816 is -30°. With this information, the surgeon knows the location on the bone where the cam lesion begins in the two-dimensional image.

[0189] In some implementations, the surgeon may be provided with the clock face position of the tool in a two-dimensional image to help the surgeon understand the position of the tool relative to the portion of bone to be removed. Figure 36 A graphical user interface 3600 is described, in which the clock face position of a tool is displayed. Interface 3600 includes a two-dimensional image 3602 imaged of a bone removal tool 3604 located near the femoral head. Interface 3600 also includes a rendering of a three-dimensional model 3606 of the femur, located according to the position and orientation of the femur in image 3602, based on the principles described above. The rendering also includes a representation 3608 of the bone removal tool located according to the position of tool 3604 in image 3602. A clock face value 3610 of the distal end of tool 3604 may be provided in interface 3600. In some embodiments, the position of the bone removal tool can be determined by segmenting the tool from the image using a semantic segmentation algorithm (e.g., U-net). The distal end of the tool can then be determined and its clock face calculated.

[0190] According to various implementation schemes, interface 3600 can provide additional information about the position and orientation of the femur captured in the image, such as the flexion, abduction, external rotation and / or the clock face of the horizontal line determined according to the above principles.

[0191] In some implementations, the overlay may include a representation of the bone contour from the model to indicate how well the projection of the user model is aligned with the bones in the two-dimensional image. Figure 29A This illustrates the superposition of the outer edge contour 2908, which includes the model projection. Contour 2908 is shown in an exaggerated misaligned position to illustrate that by including contour 2908, surgeons can evaluate the accuracy of the alignment process.

[0192] In some implementations, the user can manipulate the model alignment within the user interface. For example, such as Figure 29A As shown, the overlay image is displayed in a user interface that provides three user controls. A head position control "H" is provided at the center of the femoral head to shift the overlay's x and y positions. The user can select the head position control "H" and drag it around the screen to reposition the model render relative to the 2D image, i.e., to plan the bone removal and representation of the femoral contour. The user can move the model render until the femoral contour aligns with the outer edge of the femur in the 2D image, as shown. Figure 29B As shown. Figure 29A The implementation also includes a rotation tool 2912 that can be used to rotate the model around the center point of the femoral head. A scaling tool "R" is provided to allow the surgeon to increase or decrease the scale of the model.

[0193] In some implementations, the tool manipulates the model to generate a new projection. User input is used to realign the model and generate a new projection that is overlaid on the 2D image. In other implementations, user input is used to modify the projection rendering itself.

[0194] As described above, systems and methods according to various embodiments can be used to overlay a representation of planned bone removal onto a two-dimensional image of the bone generated intraoperatively. The overlay can be displayed to the surgeon to indicate the areas of bone captured in the two-dimensional image and the locations of bone to be removed, both of which extend beyond the captured area. The surgeon can then use the overlay as a guide to remove bone. The surgeon can then capture a new two-dimensional image of the bone to determine the amount and location of bone removed, to evaluate whether sufficient bone has been removed. According to some embodiments, the system can be configured to update the representation of planned bone removal to generate the overlay, wherein the rendering of the representation of planned bone removal in the overlay reflects the removed bone.

[0195] Figure 30The present invention describes a method 3000 for generating an overlay image according to some embodiments, the overlay image having an updated representation reflecting planned bone removal that has been removed during a surgical procedure. In step 3002, a two-dimensional image of the joint is received intraoperatively. In step 3004, the two-dimensional image is analyzed to extract features of the target bone that can be used to align a three-dimensional model of the bone with the two-dimensional image. This step is similar to step 1904 of method 1900, and therefore details of this step are omitted. In step 3006, in a manner similar to step 1906 of method 1900, the alignment of the three-dimensional model with the two-dimensional image is determined based on at least a portion of the features extracted in step 3004. In step 3008, a projection of the model is generated based on the alignment determined in step 3006.

[0196] In step 3010, one or more portions of the projection associated with the removed bone are identified. In some embodiments, this step may include comparing the projection with a set of edges detected in a two-dimensional image and identifying projection portions outside the edges associated with the target bone from the two-dimensional image.

[0197] In step 3012, an updated representation of the planned bone removal is generated based on the projected portion identified in the two-dimensional image as being outside the outer edge of the bone. In some embodiments, the projection is directly manipulated to reflect the removed bone. For example, the outer edge of the bone in the projection may be altered to align with an edge in the two-dimensional image. In some embodiments, only the representation of the bone removal is modified to reflect the removed bone. In some embodiments, the bone removal is represented as a heatmap, and a portion of the heatmap associated with the removed bone is removed from the projection. In other embodiments, the portion of the heatmap associated with the removed bone is visually altered to indicate its possible association with the removed bone.

[0198] In some implementations, an updated representation is generated by first updating the 3D model and then regenerating the projection from the updated 3D model. The 3D model can be updated so that it does not include bones identified as to be removed. In some implementations, the representation of planned bone removal can be updated accordingly. For example, the heatmap portion associated with the removed bone can be changed to a different color to indicate that the bone has been removed and no further bone removal from that area is necessary, or that there may still be bone that needs to be removed from the area to achieve the target morphology. In some implementations, the representation of planned bone removal can be updated based on detected grayscale changes in a 2D image relative to a previously captured 2D image. Grayscale changes associated with bone portions not on the horizontal line can indicate that there is less bone in that portion relative to a previously captured image, thus indicating bone removal. For example, a brightening of a region in a 2D image relative to a previously captured 2D image can be associated with removed bone.

[0199] Then, in step 3014, an overlay image is generated by superimposing the updated representation onto the two-dimensional image. The overlay can then be displayed to the surgeon in step 3016, allowing the surgeon to visualize the locations of bone that have been removed and the locations of bone that still need to be removed.

[0200] Figures 31A-31C Explain the results of method 3000 according to the implementation plan. Figure 31A An overlay image is described, which shows a representation of planned bone removal 3102 overlaid on a two-dimensional image 3104 before any bone removal. Figure 31B This describes an updated overlay generated from a new two-dimensional image captured after the surgeon has removed some bone from the femoral head. The removal of the portion 3106 represented by the planned bone removal 3102 is reflected in the updated overlay. Figure 31C The illustration, as shown in removal section 3108, describes an overlay generated later in the procedure after more bone has been removed. In this example, the portion of the planned bone removal associated with the removed bone in the representation is shaded with a solid color different from the color used to represent the planned bone removal, to provide a clear visual indication that bone has been removed from these areas. In some embodiments, the planned bone removal portion associated with the removed bone may be completely omitted from the overlay, making that portion visible in those areas of the 2D model. In some embodiments, the removed bone can be determined by comparing the outer edge of the 3D model with the outer edge of the bone in the current 2D image to identify missing 2D bone slices in the 2D image. The missing bone slices from the 2D image can then be removed from the 3D model, for example, by “squeezing” the slices from the 3D image. The planned bone removal overlay can then be updated based on the updated 3D model to be overlaid on the 2D image.

[0201] As described above, a 3D model is projected onto a 2D plane to determine the alignment of the model with the 2D image and to overlay a planned bone removal representation onto the 2D image. In some embodiments, the model may be projected using a back-projection technique, which illustrates how the 2D image is generated. Back-projection may be used during the determination of the alignment of the model with the 2D image, such as in step 1906 of method 1900, step 2504 of method 2500, and / or step 2606 of method 2600, and / or during the generation of the planned bone removal overlay, such as in step 1908 of method 1900.

[0202] For example, back projection is suitable for implementations where a two-dimensional image is generated by an X-ray imager (such as a C-arm imager), with the X-ray source located below the patient and the X-ray detector located above the patient. When the X-ray detector is above the patient and below the source, objects closer to the X-ray source appear larger than objects farther away. Since the standard surgical view is from above the patient, the perspective captured in the X-ray image is the opposite of that from the standard surgical view. In other words, objects positioned farther from the observer according to the standard surgical view appear larger in the X-ray image than objects positioned closer to the observer. To accommodate this difference in perspective, according to some implementations, back projection techniques are used to project a three-dimensional model onto a two-dimensional imaging plane.

[0203] Figure 34 This section conceptually explains the use of back projection to project a 3D model onto a 2D plane corresponding to the X-ray imaging plane. Back projection replicates the perspective view created by an X-ray imager by magnifying the portion farther from the observer (also referred to herein as the camera) and shrinking the portion closer to the observer. Therefore, when the observer-facing portion of the 3D model is included in the projection, points on that portion are projected such that the portion farther from the observer is larger than the portion closer to the observer. This is shown in... Figure 34 In the overlay 3402, a heatmap 3404 of the anterior portion 3406 of the left femoral head in the 3D model 3408 is generated by projecting from below (rear) the 3D model 3408 onto a projection plane 3410 (corresponding to the X-ray detector / imaging plane). According to some embodiments, the back projection uses a projection ray 3416 to map points on the 3D model 3408 onto the projection plane 3410, the projection ray 3416 extending from a point 3412 located behind the projection plane 3410 at a distance corresponding to the focal length 3414 of the X-ray imager. This results in the portion of the heatmap 3404 farther from the observer 3416 appearing relatively larger in the overlay 3402 than the portion of the same size closer to the observer 3416.

[0204] As described above, back projection can be used to align the 3D model with the 2D image and to overlay a representation for generating planned bone removal, such as in step 1908 of method 1900. Once the 3D model is properly aligned (e.g., via back projection), the planned bone removal representation (or at least a portion thereof used for overlay) is located on a portion of the 3D model facing the X-ray image observer, and the overlay of at least a portion of the planned bone removal representation is generated via back projection, wherein the portion of the planned bone removal representation farther from the observer is relatively larger than a similarly sized portion of the planned bone removal representation closer to the observer. For example, in generating an X-ray image of the femoral head, where the direction from the X-ray emitter to the detector is back-to-foreground, the front of the 3D model faces the observer direction, and the planned bone removal representation on the front of the model is projected onto a projection plane corresponding to the imaging plane, such that the rearward (farther from the observer) portion is larger than the equally sized frontward (closer to the observer) portion. As described above, this is the opposite of a standard projection in which the portion closer to the observer will be relatively larger than the portion farther from the observer. Therefore, the overlay is generated in a manner corresponding to the capture of X-rays and can provide an overlay image in which the overlay from the 3D model is better aligned with the bones in the X-ray image.

[0205] Figure 32 A system 3200 is described for generating two-dimensional images of a subject's joint, generating an overlay image in which a representation of planned bone removal from a three-dimensional model of the joint is superimposed on the two-dimensional image, and displaying the overlay image to a physician to guide the physician in bone removal during the surgical procedure, according to various embodiments. System 3200 includes a three-dimensional modeling subsystem 3202 for generating a three-dimensional model of the joint including a representation of planned bone removal, and an intraoperative imaging subsystem 3204 for generating two-dimensional images of the joint during the surgical procedure (e.g., ...). Figure 18 The C-arm X-ray machine 115) and the visual guidance subsystem 3206 for generating overlay images from 3D models and 2D images (e.g., the ... Figure 18The visual guidance system 125. Subsystems may be communicatively connected to each other via one or more communication connections 3208, which may be a network connection (such as a local area network, wide area network, a combination of local area networks and wide area networks, or any suitable communication network), one or more direct connections between subsystems, or a combination of direct and network connections. Any of the modeling subsystem 3202, intraoperative imaging subsystem 3204, and visual guidance subsystem 3206 may be located separately from other subsystems, or may have components located separately from other subsystems or components of subsystems. In some embodiments, subsystems or portions of subsystems may be located in the same location, such as within the same operating suite. In some embodiments, the 3D modeling subsystem 3202 and intraoperative imaging subsystem 3204 are the same system or share the same components, such as the same imager.

[0206] The 3D modeling subsystem 3202 may include an imager for generating imaging data on a subject. The imaging data may include, for example, MRI scans, CT scans, X-rays, fluorescence imaging data, or any suitable imaging data for imaging the subject's joints. In some embodiments, the 3D modeling subsystem 3202 may include one or more imaging data processing systems for processing the imaging data generated by the imager. The 3D modeling subsystem 3202 may include one or more data storage systems for storing the imaging data and / or model data. The 3D modeling subsystem 3202 may be configured to transmit imaging and / or model data of the subject's joints to the visual guidance subsystem 3206. For example, after an imaging session in which the subject's joints are imaged, the 3D modeling subsystem 3202 may generate a 3D model of the joint from the imaging data generated during the session. Based on the principles described herein, the 3D model may be transmitted to the visual guidance subsystem 3206 for generating an overlay image. According to some embodiments, the 3D modeling subsystem 3202 may identify at least one region of the imaged joint that deviates from the baseline anatomy by comparing at least a portion of the 3D model to a baseline model. Based on the principles described herein, the 3D modeling subsystem 3202 can use a 3D model and coordinate system to generate one or more measurements of joint features at one or more predefined locations; and generate a 3D rendering of the model. The 3D rendering may include a visual indication of at least one region where the 3D model deviates from a baseline, wherein the visual indication is based on the degree of deviation. This visual indication may represent the planned bone removal for a surgical procedure. The 3D rendering may be a visualization component that includes any other relevant information as described herein.

[0207] The intraoperative imaging subsystem 3204 may include an imager for generating two-dimensional imaging data for the subject during a surgical procedure. The intraoperative imaging subsystem 3204 may include, for example, an x-ray imager, such as a C-arm x-ray imager. In some embodiments, the intraoperative imaging subsystem 3204 may be configured to transmit imaging data of the subject's joints to the visual guidance subsystem 3206. For example, according to the principles described herein, after capturing an image of the subject's joints, the image may be transmitted to the visual guidance subsystem 3206 for generating a superimposed image.

[0208] Based on the principles described herein, the visual guidance subsystem 3206 can be configured to receive imaging data and 3D model data and use some or all of the received data to generate an overlay image.

[0209] Figure 33 This describes examples of computing systems that, according to some implementation schemes, can be used in one or more of the subsystems 3202, 3204, and 3206 of system 3200. System 3300 may be a computer connected to a network, such as one or more communication connection networks 3208 of system 3200. System 3300 may be a client computer or a server. Figure 33 As shown, system 3300 can be any suitable type of microprocessor-based system, such as a personal computer, workstation, server, or handheld computing device (portable electronic device) such as a telephone or tablet. The system may include one or more of, for example, a processor 3310, an input device 3320, an output device 3330, a storage device 3340, and a communication device 3360. Input device 3320 and output device 3330 typically correspond to those described above and can be connected to or integrated with a computer.

[0210] Input device 3320 may be any suitable device that provides input, such as a touchscreen, keyboard or keypad, mouse, gesture recognition component of a virtual / augmented reality system, or voice recognition device. Output device 3330 may be or may include any suitable device that provides output, such as a touchscreen, haptic device, virtual / augmented reality display, or speaker.

[0211] Storage device 3340 may be any suitable device providing storage, such as electrical, magnetic, or optical storage including RAM, cache memory, hard disk drive, removable storage disk, or other non-transitory computer-readable media. Communication device 3360 may include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or device. Computer components may be connected in any suitable manner, such as via a physical bus or wirelessly.

[0212] Software 3350, which may be stored in storage device 3340 and executed by processor 3310, may include programming that embodies, for example, the functions of this disclosure (e.g., embodied in the device described above). For example, software 3350 may include one or more programs for implementing one or more of the steps of method 1900, method 2100, method 2500, method 2600 and / or method 3000.

[0213] Software 3350 may also be stored and / or transmitted in any non-transitory computer-readable storage medium for use or in conjunction with an instruction execution system, apparatus, or device (such as those described above) that can retrieve and execute instructions relating to the software from and execute such instructions. In the context of this disclosure, a computer-readable storage medium may be any medium, such as storage device 3340, which may contain or store programming for use or in conjunction with an instruction execution system, apparatus, or device.

[0214] The software 3350 can also be propagated within any transmission medium for use or in conjunction with an instruction execution system, apparatus, or device (such as those described above) that can retrieve and execute instructions related to the software from and execute such instructions. In the context of this disclosure, the transmission medium can be any medium capable of communicating, propagating, or transmitting programming for use by or in conjunction with an instruction execution system, apparatus, or device. Transmittable media can include, but are not limited to, electrical, magnetic, optical, electromagnetic, or infrared wired or wireless transmission media.

[0215] System 3300 can be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can contain network links in any suitable arrangement that enables network signal transmission and reception, such as wireless network connections, T1 or T3 lines, cable networks, DSL or telephone lines.

[0216] System 3300 can run any operating system suitable for operation over a network. Software 3350 can be written in any suitable programming language, such as C, C++, etc. ++ Java or Python. In various implementation schemes, the application software embodying the functions of this disclosure can be deployed in different configurations, such as a client / server setup or via a web browser as a web-based application or web server.

[0217] For purposes of explanation, the foregoing description has been given with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of these techniques and their practical application. This enables others skilled in the art to best utilize the techniques and various embodiments with various modifications as suited to the particular purpose considered.

[0218] For the purposes of clarity and concise description, features are described herein as part of the same or different embodiments; however, it should be appreciated that the scope of this disclosure includes embodiments having all or some of the described features.

[0219] Although this disclosure and embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications will be understood to be included within the scope of this disclosure and embodiments as defined in the claims. Finally, the full disclosure of any patents and publications mentioned in this application is hereby incorporated by reference.

Claims

1. A method for guiding bone removal during a surgical procedure, the method comprising: receiving a two-dimensional image of at least a portion of a joint during the surgical procedure; determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image based on one or more features in the two-dimensional image related to the at least a portion of the joint, wherein the pre-generated three-dimensional model contains a representation of planned bone removal; generating an overlay image containing an overlay of at least a portion of the representation of planned bone removal on the two-dimensional image based on the determined alignment; and displaying the overlay image to guide bone removal during the surgical procedure.

2. The method of claim 1, wherein the three-dimensional model is pre-generated based on one or more scans of the joint.

3. The method of claim 1 or 2, wherein the one or more features are associated with at least one of a center of a femoral head, a centerline of a femoral neck, and an outer rim of the femoral head.

4. The method of claim 1 or 2, wherein determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image comprises translating and rotating the three-dimensional model based on the one or more features.

5. The method of claim 1 or 2, wherein determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image comprises detecting edges in the two-dimensional image related to a bone perimeter.

6. The method of claim 1 or 2, wherein determining an alignment of a pre-generated three-dimensional model of at least a portion of the joint with the two-dimensional image comprises determining a center of a femoral head in the two-dimensional image, determining a centerline of a femoral neck in the two-dimensional image, aligning a model femoral head center in the three-dimensional model with the femoral head center in the two-dimensional image, and aligning a centerline of a model femoral neck in the three-dimensional model with the centerline of the femoral neck in the two-dimensional image.

7. The method of claim 1 or 2, wherein the representation of planned bone removal comprises at least one of a heat map indicating a location and amount of planned bone removal, a contour map indicating a location and amount of planned bone removal, and an outline of a region of planned bone removal.

8. The method of claim 1 or 2, wherein the three-dimensional model includes a representation of bone outside of the planned bone removal, and the representation of bone outside of the planned bone removal is omitted from the overlay.

9. The method of claim 1 or 2, further comprising determining a portion of the three-dimensional model related to bone that has been removed during the surgical procedure and omitting the portion of the three-dimensional model from the overlay.

10. The method of claim 9, wherein the portion of the three-dimensional model omitted from the overlay is a portion of a heat map related to planned bone removal.

11. The method of claim 1 or 2, further comprising modifying a position of the overlay of at least a portion of the representation of planned bone removal on the two-dimensional image in response to a user input after displaying the overlay image. ​ 12. The method of claim 1 or 2, further comprising capturing a new two-dimensional image of the joint portion at a new location, determining an updated alignment of the pre-generated three-dimensional model to the new two-dimensional image, generating an updated superimposed image based on the determined updated alignment, and displaying the updated superimposed image to indicate progress of bone removal.

13. The method of claim 1 or 2, wherein the representation of planned bone removal is three-dimensional, and generating the superimposed image comprises projecting the representation of planned bone removal onto a two-dimensional plane.

14. The method of claim 1 or 2, wherein the representation of planned bone removal indicates planned bone removal associated with cam-type femoroacetabular impingement.

15. The method of claim 1 or 2, wherein the representation of planned bone removal indicates planned bone removal associated with pincher-type femoroacetabular impingement.

16. The method of claim 1 or 2, wherein the two-dimensional image is received intraoperatively from an x-ray system.

17. The method of claim 1 or 2, wherein the three-dimensional model is based on imaging data from a first type of imaging system, and the two-dimensional image is received from a second type of imaging system different from the first type.

18. The method of claim 17, wherein the first type of imaging system is an MRI system or a CT system, and the second type of imaging system is a C-arm x-ray system.

19. A system for guiding bone removal during a surgical procedure, the system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a two-dimensional image of at least a portion of a joint during the surgical procedure; determining an alignment of a pre-generated three-dimensional model of the at least a portion of the joint to the two-dimensional image based on one or more features in the two-dimensional image associated with the at least a portion of the joint, wherein the pre-generated three-dimensional model contains a representation of planned bone removal; generating a superimposed image based on the determined alignment, the superimposed image containing a superimposition of at least a portion of the representation of planned bone removal on the two-dimensional image; and transmitting the superimposed image to a display for guiding bone removal during the surgical procedure.

20. The system of claim 19, wherein the system is configured for communicative connection to an intraoperative imaging system that generates the two-dimensional image.

21. The system of claim 19 or 20, wherein the one or more programs include instructions for receiving user input to reposition the superimposition of at least a portion of the representation of planned bone removal on the two-dimensional image.

22. The system of claim 21, comprising a touch screen display or an augmented reality system for displaying the two-dimensional image and receiving the user input.

23. The system of claim 19 or 20, wherein the three-dimensional model is pre-generated based on one or more scans of the joint.

24. The system of claim 19 or 20, wherein the one or more features are associated with at least one of a center of the femoral head, a centerline of the femoral neck, and an outer edge of the femoral head.

25. The system of claim 19 or 20, wherein determining an alignment of the pre-generated three-dimensional model of at least a portion of the joint to the two-dimensional image includes translating and rotating the three-dimensional model based on one or more features.

26. The system of claim 19 or 20, wherein determining an alignment of the pre-generated three-dimensional model of at least a portion of the joint to the two-dimensional image includes detecting edges in the two-dimensional image that are associated with a perimeter of a bone.

27. The system of claim 19 or 20, wherein determining an alignment of the pre-generated three-dimensional model of at least a portion of the joint to the two-dimensional image includes determining a center of the femoral head in the two-dimensional image, determining a centerline of the femoral neck in the two-dimensional image, aligning a model femoral head center in the three-dimensional model with the femoral head center in the two-dimensional image, and aligning a centerline of a model femoral neck in the three-dimensional model with the centerline of the femoral neck in the two-dimensional image.

28. The system of claim 19 or 20, wherein the representation of the planned bone removal includes at least one of a heat map indicating a location and amount of planned bone removal, a contour map indicating a location and amount of planned bone removal, and an outline of a region of planned bone removal.

29. The system of claim 19 or 20, wherein the three-dimensional model includes a representation of bone outside of the planned bone removal, and the representation of bone outside of the planned bone removal is omitted from the overlay.

30. The system of claim 19 or 20, wherein the one or more programs include instructions for determining a portion of the three-dimensional model that is related to bone that has been removed during the surgical procedure and omitting the portion of the three-dimensional model from the overlay.

31. The system of claim 30, wherein the portion of the three-dimensional model that is omitted from the overlay is a portion of a heat map related to planned bone removal.

32. The system of claim 19 or 20, wherein the one or more programs include instructions for modifying a position of an overlay of at least a portion of a representation of planned bone removal on the two-dimensional image in response to user input after the overlay image is displayed.

33. The system of claim 19 or 20, wherein the one or more programs include instructions for capturing a new two-dimensional image of the joint portion at a new position, determining an updated alignment of the pre-generated three-dimensional model to the new two-dimensional image, generating an updated overlay image based on the determined updated alignment, and displaying the updated overlay image to indicate a progress of bone removal.

34. The system of claim 19 or 20, wherein the representation of the planned bone removal is three-dimensional, and generating the overlay image includes projecting the representation of the planned bone removal onto a two-dimensional plane.

35. The system of claim 19 or 20, wherein the representation of the planned bone removal indicates planned bone removal related to cam-type femoroacetabular impingement.

36. The system of claim 19 or 20, wherein the representation of planned bone removal indicates planned bone removal associated with a pincer-type femoroacetabular impingement.

37. The system of claim 19 or 20, wherein the two-dimensional images are received intraoperatively from an x-ray system.

38. The system of claim 19 or 20, wherein the three-dimensional model is based on imaging data from a first type of imaging system, and the two-dimensional images are received from a second type of imaging system different from the first type.

39. The system of claim 38, wherein the first type of imaging system is an MRI system or a CT system, and the second type of imaging system is a C-arm x-ray system.

40. A non-transitory computer readable medium storing instructions for execution by one or more processors of an imaging system to implement the method of any of claims 1-18.

Images