SYSTEM AND METHOD USING PHOTOGRAMMETRY FOR ALIGNING SURGICAL ELEMENTS INTRA-SURGICAL OPERATIONS - Patent application

Abstract

A system and method for verifying a position of an orthopedic element in space includes capturing first and second images of the orthopedic element in different reference frames using radiographic imaging techniques, detecting spatial data defining anatomical landmarks on or within the orthopedic element using a deep learning network, applying a mask to the orthopedic element defined by the anatomical landmarks, projecting the spatial data from the first and second images to define volumetric data, applying the deep learning network to the volumetric data to generate a reconstructed three-dimensional model of the orthopedic element, and mapping the three-dimensional model of the orthopedic element to the spatial data to determine a position of the three-dimensional model of the orthopedic element in three-dimensional space.

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 250,906, filed September 30, 2021. The disclosure of this related application is incorporated herein in its entirety.

[0002] The present disclosure relates generally to the field of orthopaedic joint replacement surgery, and more specifically to the use of photogrammetry and three-dimensional ("3D") reconstruction techniques to assist surgeons and engineers in planning and performing orthopaedic surgical procedures. [Background technology]

[0003] The goal of hip replacement surgery is to restore the natural alignment and range of motion of the patient's pre-diseased hip joint. However, this goal can be difficult to achieve in practice because the hip joint contains not only articular bones, but also various soft tissues, including cartilage, muscles, ligaments, and tendons. In all hip arthroplasty, but especially in minimally invasive hip arthroplasty, the presence of these soft tissues can severely limit the surgeon's field of vision. This problem is even more pronounced in patients with a high body mass index.

[0004] In hip arthroplasty, the pelvis itself is almost completely surrounded by soft tissue. In minimally invasive procedures, the main incision ultimately exposes the junction of the acetabulum and the proximal femoral head, but this main incision typically orients the surgeon's view across the rim (i.e., periphery) of the acetabulum. A portal incision extending through one or more quadriceps muscles of the operative leg may align with the concave surface of the acetabulum, but the proximal end of the femur must be translated and rotated away from the acetabulum to expose this view.

[0005] To further complicate matters, the view of the surgical area through the portal incision is generally much more limited than the view through the main incision. Although an endoscopic camera may be placed through the portal incision to capture an image of the concave acetabular surface, the concave surface of the acetabulum lacks a bony marker (i.e., a landmark) that can be used to reliably indicate the position of the acetabulum and the pelvis. Furthermore, any movement of the femur is likely to be transmitted to the pelvis through the connective soft tissue, thereby compromising the usefulness of any images captured by the endoscopic camera. Thus, the use of an endoscopic camera unnecessarily prolongs the procedure and has very limited effectiveness in accurately reflecting the position of the acetabulum relative to the proximal femur.

[0006] A hip prosthesis implant typically includes an acetabular shell that the surgeon places into the reamed acetabulum of the hip joint. The acetabular shell may house a liner that essentially functions as a bearing with the generally spherical head of the femoral component. The femoral component generally includes a stem, a neck, and a head. Once seated, the stem is inserted into the resected and reamed proximal end of the femur. The neck connects the proximal end of the stem to the head. The head is then placed into the prosthetic acetabular cup, generally seated against the liner of the acetabular cup.

[0007] Because the surgeon's view of the surgical field is often obstructed by soft tissue, surgeons have in the past relied on external markings to attempt to estimate the proper alignment of the acetabular cup within the acetabulum. U.S. Patent Publication No. 2013 / 0165941 (Murphy) is one such example. Other providers have provided positioning guides that include external horizontal and vertical positioning bars designed to resemble the axes of a Cartesian plane. To attempt to achieve an acetabular cup abduction angle of about 40 degrees ("°") to about 45 degrees, the surgeon positions the positioning guide approximately diagonally relative to the patient's body longitudinal axis (i.e., an imaginary centerline of the body extending from the patient's head to the groin) such that the horizontal positioning bar is positioned approximately parallel to the body longitudinal axis. To attempt to achieve an anteversion angle of about 10° to about 15°, the surgeon slightly lifts the positioning device along the vertical positioning bar relative to the body's longitudinal axis.

[0008] These external landmarks did not take into account the specific anatomy of the patient, nor did they take into account the movement of the pelvis relative to these landmarks. For example, when a given patient is lying supine, it is entirely possible that the patient's left acetabulum may be positioned slightly lower than the patient's right acetabulum. Furthermore, many hip arthroplasties require repositioning of the patient to make certain incisions or to access certain parts of the surgical field. As mentioned above, femoral movement is likely to be transmitted to the pelvis through the soft tissue. If a patient needs to be repositioned multiple times throughout a standard hip arthroplasty, it is unlikely that the pelvis will always be located within the proposed usage parameters for existing acetabular cup positioning guides that rely on external landmarks.

[0009] Properly aligning, sizing, and placing the femoral component is even more difficult, given that allowing the surgeon to have a direct line of sight to the proximal femur requires disengaging (and therefore dislocating) the proximal femur from the acetabulum (or in some cases the acetabular cup). As a result, many surgeons have relied on sound and feel to approximate acceptable femoral stem placement. Both the femoral stem and the acetabular cup are impacted into their respective bones. If the femoral stem is too large, it can easily fracture the proximal femur. If the femoral stem is too small, it can sink into the intramedullary canal of the femur over time as a result of normal use. The sinking can shorten the patient's gait and place undue pressure on the neck, head, and liner areas, thereby accelerating wear.

[0010] Furthermore, even when the acetabular cup was placed in the reamed acetabulum at the desired abduction and anteversion angles, and even when an appropriately sized femoral stem was seated in the proximal femur, the position of the femoral component relative to the acetabular cup could not previously be known using conventional techniques. Although intraoperative fluoroscopy can be used to generate two-dimensional ("2D") images of the femoral component relative to the acetabular component, the fluoroscopic images lacked sufficient 3D information to ensure accurate alignment. For example, with classical fluoroscopy, the pelvic tilt was unknown. Thus, the orientation of any bony landmarks on the pelvis was also unknown. Without the ability to accurately determine the orientation of the pelvis, it was not possible to accurately calculate the location of the natural anterior joint line using fluoroscopy alone. Furthermore, prolonged use of fluoroscopy exposes patients to excessive radiation.

[0011] Improper alignment of the head of the femoral component relative to the acetabular cup can result in shortening of the operative leg relative to the contralateral leg, dislocation of the head relative to the acetabular cup, and increased force loading on portions of the acetabular cup, liner, head, or neck (thereby increasing wear rates and decreasing implant life). Any of these shortcomings can contribute to patient discomfort.

[0012] As a result, surgeons have had to maintain a fairly large margin of error for the placement of the acetabular cup. Despite the available tools and procedures, aligning the reconstructed hip joint in a typical hip arthroplasty is based on experience, educated guesses, and chance. This problem can be particularly pronounced in minimally invasive hip arthroplasty, in part because the surgeon's field of vision is so limited. Summary of the Invention

[0013] Thus, there is a long felt and unmet need to augment pre-operative and intra-operative imaging techniques to accurately model the surgical joint anatomy and artificial endoprosthetic implants when planning and performing hip arthroplasty.

[0014] Problems of limited surgeon visualization of the surgical field in minimally invasive surgery using currently available pre-operative or intra-operative tools and techniques, and the attendant problems of misalignment that lack of such visualization can cause, can be alleviated by an exemplary system or method for verifying the position of an orthopedic element in space, which includes using a deep learning network to identify and model components of the orthopedic element and an endoprosthetic implant, and mapping the model of the orthopedic element and the endoprosthetic implant to spatial data from an input of at least two separate two-dimensional ("2D") input images of the orthopedic element of interest, where a first image of the at least two separate 2D input images is captured from a first lateral position and a second image of the at least two separate 2D input images is captured from a second lateral position that is offset from the first lateral position by an offset angle.

[0015] In an exemplary embodiment, the input image may be a radiographic image. Without wishing to be bound by theory, radiographs may be desirable because they allow for an in vivo analysis that can account for the external summation of passive soft tissue structures and dynamic forces occurring around the hip joint, including the effects of ligament constraints, load bearing forces, and muscle activity.

[0016] Without being bound by theory, it is believed that by mapping a model of the orthopedic element and a model of the endoprosthetic implant component to the spatial data, the position of the mapped and modeled orthopedic element can be calculated relative to the mapped and modeled implant component. When the system is applied to two or more orthopedic elements and two or more endoprosthetic implant components, the endoprosthetic implant components can be desirably implanted in their respective orthopedic elements at desired positions and the respective endoprosthetic implant components can be desirably aligned with one another.

[0017] It is further contemplated that certain example systems and methods described herein may be configured to accurately predict a desired size of an implant component relative to adjacent orthopedic elements.

[0018] It is further contemplated that certain exemplary systems and methods described herein can be configured to precisely orient the placement of the endoprosthetic implant component relative to the orthopedic component into which it is implanted. [Brief description of the drawings]

[0019] The foregoing will become apparent from the following more particular description of exemplary embodiments of the disclosure, as illustrated in the accompanying drawings, in which: The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the disclosed embodiments. [Figure 1] FIG. 1 is a simplified frontal x-ray view of a patient showing an exemplary internal prosthetic hip implant in a right hip joint and a natural left hip joint. [Diagram 2] 1 shows a typical view of the surgeon during minimally invasive hip arthroplasty. [Diagram 3] 2 is a perspective view of an exemplary acetabular component placed in a reamed acetabulum. FIG 3 illustrates the principles of abduction and anteversion angles. [Figure 4]Showing a misaligned acetabular cup and an incorrectly sized femoral stem. [Diagram 5] 1 is a flow chart illustrating steps of an exemplary method. [Figure 6] 4 is a flowchart illustrating further exemplary method steps. [Figure 7] FIG. 1 is a schematic diagram of a system that uses a deep learning network to identify features (e.g., anatomical landmarks) of a target orthopedic element and generate a 3D model of the target orthopedic element. [Figure 8] FIG. 1 is a schematic diagram of a pinhole camera model used to convey how the principles of epipolar geometry can be used to ascertain the location of a point in 3D space from two 2D images taken from different reference frames of a calibrated image detector. [Figure 9A] 1 is an image of a target orthopedic component taken from an anterior-posterior ("AP") position showing an exemplary calibration fixture. [Figure 9B] 9B is an image of the target orthopedic element of FIG. 9A taken approximately 45° clockwise from the reference frame of FIG. 9A using a calibration fixture. [Figure 9C] 9B is an image of the target orthopedic element of FIG. 9A taken approximately 45° counterclockwise from the reference frame of FIG. 9A using a calibration fixture. [Figure 10] FIG. 1 is a schematic diagram illustrating how a convolutional neural network ("CNN") type deep learning network can be used to identify features (e.g., anatomical landmarks) including the surface of an orthopedic element of interest. [Figure 11] FIG. 1 is an exploded view of a modeled endoprosthetic implant. [Figure 12] FIG. 1 is a schematic diagram of an exemplary system. [Figure 13] FIG. 1 is a schematic diagram of a system configured to generate a model of an orthopedic element and take two or more tissue transmission flattened input images of the same target orthopedic element at offset angles from a calibrated detector to align components of an endoprosthetic implant component. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The following detailed description of the preferred embodiments is presented for illustrative purposes only to aid in understanding, and is not intended to be exhaustive or to limit the scope and spirit of the present invention. The embodiments have been chosen and described in order to best explain the principles of the invention and its practical application. Those skilled in the art will recognize that many modifications can be made to the invention disclosed herein without departing from the scope and spirit of the invention.

[0021] Unless otherwise stated, like reference characters designate corresponding parts throughout the several views. Although the drawings depict embodiments of various features and components according to the present disclosure, the drawings are not necessarily to scale and certain features may be exaggerated to better illustrate embodiments of the present disclosure, and such illustrations should not be construed as limiting the scope of the present disclosure.

[0022] Except as otherwise expressly stated herein, the following rules of interpretation apply herein: (a) all words used herein shall be construed as to such gender or number (singular or plural) as required in such context; (b) the singular terms "a," "an," and "the" as used herein and in the appended claims include plural references unless the context clearly indicates otherwise; (c) the antecedent "about" applied to a recited range or value indicates an approximation having a deviation of the range or value known in the art or expected from measurement; (d) unless otherwise indicated, the words "herein," "hereby," "hereinbefore," and "hereinafter," and words of similar import, refer to this specification as a whole and not to any particular paragraph, claim, or other subdivision; (e) the description headings are for convenience only and do not control or affect the meaning of any portion of this specification; (f) "or" and "any" are not exclusive, and "include," "including" are not limiting. Furthermore, the words "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to").

[0023] References herein to "one embodiment," "an embodiment," "exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments may necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is implied that it is within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0024] To the extent necessary to provide narrative support, the subject matter and / or text of the appended claims are incorporated herein by reference in their entirety.

[0025] The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value within any subrange therebetween, unless otherwise expressly indicated herein. Each separate value within a recited range is incorporated into the specification or claims as if each separate value were individually recited herein. When a specific range of values ​​is provided, it is understood that each intervening value, to the nearest tenth of the unit of the lower limit between the upper and lower limits of that range, and any other recited or intervening value within the recited range of that subrange, is included herein, unless the context clearly dictates otherwise. All subranges are also included. The upper and lower limits of these smaller ranges are also included therein, subject to any specific and expressly excluded limitations in the recited range.

[0026] It should be noted that some of the terms used herein are relative terms: for example, the terms "upper" and "lower" are relative to one another in location, i.e., the upper component is located higher than the lower component in each orientation, but these terms may change if the orientation is reversed.

[0027] The terms "horizontal" and "vertical" are used to indicate orientation relative to an absolute reference, i.e., the earth's surface level. However, these terms should not be interpreted as requiring structures to be absolutely parallel or absolutely perpendicular to each other. For example, a first vertical structure and a second vertical structure are not necessarily parallel to each other. The terms "upper" and "lower" or "base" are used to refer to a place or surface where the upper is always higher than the lower or base, relative to an absolute reference, i.e., the earth's surface. The terms "upward" and "downward" are also relative to an absolute reference. An upward flow always opposes the earth's gravitational force.

[0028] Orthopedic procedures often involve surgery on a patient's joints. It will be appreciated that a joint typically includes multiple orthopedic elements. It will be further appreciated that the exemplary methods and systems described herein may be applied to a variety of orthopedic elements. The examples described with reference to Figures 1-4, 9A-9C, and 11 relate to a hip joint for illustrative purposes. It will be appreciated that the "orthopedic element" 100 referenced throughout this disclosure is not limited to the anatomy of the hip joint, but may include any skeletal structure or associated soft tissue, such as tendons, ligaments, cartilage, and muscles. A non-limiting list of examples of skeletal orthopedic elements 100 includes any partial or complete bone of the body, including but not limited to the femur, tibia, pelvis, vertebrae, humerus, ulna, radius, scapula, skull, fibula, clavicle, mandible, ribs, carpals, metacarpals, tarsals, metatarsals, phalanges, or any associated tendons, ligaments, skin, cartilage, or muscles. It will be appreciated that the exemplary surgical area 170 may include several target orthopedic elements 100. Likewise, it will be appreciated that the surgical area 170 is not limited to the hip surgical area used as the primary example herein, but rather may return to any area of ​​the body that is the target of a surgical procedure, which may include, by way of non-limiting example, the knee, ankle, spine, shoulder, wrist, hand, foot, mandible, skull, ribs, and phalanges.

[0029] 1 is a simplified representation of a frontal x-ray image of an exemplary patient's pelvis 110. Both the patient's right hip joint 101a and left hip joint 101b are shown. Both exemplary hip joints 101a, 101b include a number of orthopedic elements 100, including a femur 105, an acetabulum (see 108 and 111) of the pelvis 110, and connective tissue. The illustrated right hip joint 101a (i.e., the patient's right hip joint shown on the left side of the page) shows an exemplary internal prosthetic hip implant 102 that has been surgically installed in the patient. The illustrated left hip joint 101b shows an exemplary natural hip joint for comparison.

[0030] With reference to the illustrated right hip joint 101a, an exemplary endoprosthetic hip implant 102 generally includes an acetabular component 103 and a femoral component 104. It will be understood that an endoprosthetic implant can generally include multiple components (e.g., the acetabular component 103 and the femoral component 104), which may be comprised of multiple subcomponents. In the illustrated example, the acetabular component 103 typically includes a generally hemispherical acetabular shell 106 and an inner liner 107. The acetabular shell 106 is typically made from cobalt chrome, titanium, or other biocompatible metals. The inner liner 107 is typically made from a ceramic, metal, polymer, or other biocompatible material with a low coefficient of friction and low wear rate.

[0031] To prepare the natural acetabulum (see 108) for installation of the acetabular shell 106, the surgeon first uses a hemispherical reamer to create a generally concave surface in the patient's natural acetabulum 108 to define a "reamed acetabulum" 111. The reamed acetabulum 111 is generally complementary to the convex outer surface 109 of the acetabular shell 106. The outer surface 109 of the acetabular shell 106 is typically roughened to facilitate engagement with the reamed acetabulum 111. It is believed that the roughened surface also promotes bone formation into the space of the roughened surface, thereby increasing the strength of the bond over time.

[0032] The medial liner 107 typically lies adjacent to the medial concave surface 112 of the acetabular shell 106 when the medial liner 107 is in its assembled and installed configuration. The medial liner 107 generally functions as a bearing against which the femoral head 113 of the femoral component 104 articulates when installed.

[0033] The femoral component 104 typically includes a femoral stem 115 having a distal stem end 115a disposed distally from a proximal stem end 115b, and a neck 116 having a distal neck end 116a that engages the proximal stem end 115b. The neck 116 extends to a head end 116b. A generally spherical prosthetic femoral head 113 is disposed at the head end 116b of the neck 116 in an assembled configuration. In certain exemplary embodiments, the neck 116 may be selectively removable from the proximal stem end 115b. Such a selectively removable neck 116 may be known as a "modular neck."

[0034] It will be appreciated that the acetabular component 103 and femoral component 104, as well as sub-components including the acetabular component 103 or femoral component 104 (e.g., acetabular shell 106, medial liner 107, any fixation fasteners, femoral stem 115, femoral head prosthesis 113, etc.), are typically provided in a disassembled and unassembled configuration in one or more surgical kits. In the uninstalled and unassembled configuration, a component or sub-component does not physically engage another component or sub-component. Stated differently, forces are not directly transferred from one component or sub-component to another component or sub-component in the uninstalled and unassembled configuration. In the assembled configuration, the components or sub-components are in physical contact with each other and forces may be transferred through two or more proximally located components or sub-components. In the assembled and installed configuration, the components or sub-components are in the assembled configuration and are surgically implanted in the patient.

[0035] For comparison, the illustrated left hip joint 101b shows a natural femoral head 126 at the proximal end of the femur 105. The natural femoral head 126 is disposed within the natural acetabulum 108 of the pelvis 110. Articular cartilage 123 covers the articular surfaces of both the healthy femoral head 126 and the healthy acetabulum 108.

[0036] While there are many surgical approaches to a typical hip arthroplasty, most minimally invasive procedures begin with the surgeon making a 6-8 centimeter ("cm") incision in the operative leg radially proximal to the hip joint capsule. Various muscles and tendons are then retracted with surgical instruments to ultimately expose the joint capsule. The capsule is then perforated and the surgeon displaces the natural femoral head 126 from the natural acetabulum 108.

[0037] Preparation of the femur includes resecting and removing the natural femoral head 126 from the femur 105. After the natural femoral head 126 is removed, the surgeon may drill a canal into the intramedullary space of the newly exposed proximal end 105b of the femur 105. The surgeon may then use a femoral broach to expand the space in the intramedullary canal required to accommodate the femoral stem 115. A trial stem may be used to test the sizing and positioning of the femoral component 104. The trial component generally has the same dimensions as the actual implant component, but the trial component is designed to be more easily installed and removed.

[0038] Acetabular preparation involves reaming the natural acetabulum 108 to define a reamed acetabulum 111. The goal is to create a generally uniform hemispherical space complementary to the generally hemispherical outer surface 109 of the acetabular shell 106. A trial acetabular component can be used to attempt to test the alignment of the acetabular component 103 relative to the femoral component 104, but visibility is limited and the nature of the procedure typically does not allow for exhaustive testing of alignment. Furthermore, because visibility is limited, it is possible that the actual implant components 103, 104 will not be oriented exactly the same as the trial components.

[0039] It will be appreciated that there are various surgical approaches for a typical total hip arthroplasty (e.g., some surgeons choose to approach the hip joint posteriorly, while others choose to approach the hip joint laterally or anteriorly). FIG. 2 illustrates and illustrates a surgeon's typical view of a typical hip arthroplasty surgical area 170 through the main incision. Several retractors 14, 16 (which may include a Hoffman retractor or Cobb elevator in some procedures) are used to retract the fascia 11 located between the initial incision area and the hip capsule. An electrocautery instrument 40 may be used to resect and cauterize tissue and prevent excessive bleeding. The native femoral head 126 is also shown for reference.

[0040] FIG. 2 illustrates how the 6-8 cm main incision in the surgical field 170, the location of the hip joint relative to the incision point (see 101a, 101b), and the presence of typical surgical instruments (e.g., retractors 14, 16, mallets, broaches, reamers, pins, implant components, etc.) can significantly impede the surgeon's already limited field of vision. This problem can be exacerbated by attempting to align the acetabular component 103 and femoral component 104 of the internal hip prosthesis implant 102 to external landmarks that are separated from the orientation of the target implant anatomy (e.g., in this hip joint example, the reamed acetabulum 111 or the resected proximal femur 105), which can lead to inaccurate alignment of the implant components 103, 104 relative to the bone into which they are implanted and relative to each other. This, in turn, can contribute to the risk of implant dislocation, suboptimal force distribution, faster wear, implant failure, changes in gait, general patient discomfort, and the need for further revision surgeries subject to the same limitations.

[0041] 3 is a perspective view of an exemplary acetabular component 103 positioned within a reamed acetabulum 111. Although an abduction angle α and an anteversion angle υ are shown for the acetabular component 103, it will be understood that the femoral component 104 is also positioned at an abduction angle α and an anteversion angle υ in the proximal femur 105. The abduction angles α and anteversion angles υ of the endoprosthetic implant components (e.g., acetabular component 103 and femoral component 104) relative to the orthopedic components into which they are implanted (e.g., pelvis 110 and proximal femur 105, respectively) can be calculated and determined by one of ordinary skill in the art.

[0042] It will be understood that the "endoprosthetic implant component" may vary based on the type of endoprosthetic implant and the type of surgical area 170. For example, if the surgical area 170 is a hip joint, the "endoprosthetic implant component" may be selected from a group including the acetabular component 103, the femoral component 104, a trial construct, an instrument used in or used to facilitate the placement of the endoprosthetic implant or trial implant in place in a patient, or a combination thereof. In an embodiment in which the surgical area 170 is a knee, the "endoprosthetic implant component" may be a femoral component of an endoprosthetic knee implant, a tibial component of an endoprosthetic knee implant, a trial construct, an instrument used in or used to facilitate the placement of the endoprosthetic implant or trial implant in place in a patient, or a combination thereof.

[0043] In order to more clearly illustrate the principle of the abduction angle α of the acetabular component 103 relative to the pelvis 110, soft tissue has been omitted in FIG. 3. The abduction angle α can be measured by several methods known to those skilled in the art. One such method of visualizing the abduction angle α of the acetabular component 103 is by drawing a diameter line D extending through the diameter of the rim of the acetabular shell 106 on the coronal plane CP, relative to a generally horizontal medial-lateral reference line R that is coplanar with the coronal plane CP of the diameter line D. In FIG. 3, the reference line R is shown connecting the most distal portions 117a, 117b of the left and right ischia. However, it will be understood that other reference markers may be used, provided that the reference line R extends horizontally, medial-laterally, and coronally coplanar with the diameter line D.

[0044] The shell plane SP is also shown to extend coplanarly through the rim 2 of the acetabular shell 106. Aligning the acetabular shell 106 in three-dimensional space can be thought of as involving the selection of appropriate compound angles, which include the abduction angle α and the anteversion angle υ. The shell plane SP is shown to more clearly illustrate the concept of acetabular shell alignment in three dimensions. It will be understood that the diameter line D, the coronal plane CP, the shell plane SP, and the medial-lateral reference line R are geometric reference elements drawn to generally illustrate the abduction angle α and the concept of acetabular alignment. These geometric reference elements need not actually be visible.

[0045] Many acetabular shells 106 are designed to be placed into a reamed acetabulum 111 with an abduction angle α of about 30° to about 50°. However, this wide margin highlights the difficulty in properly aligning the acetabular shell 106 within a reamed acetabulum 111 using conventional methods. Furthermore, the general guidance of having an abduction angle α of about 30° to about 50° does not account for variability in particular patients.

[0046] FIG. 3 also illustrates the concept of anteversion angle υ. An anteversion angle υ can be calculated by several methods known to those skilled in the art. One such method of visualizing the anteversion angle υ of the acetabular shell 106 is to envision the anteversion angle υ as a rotation of the acetabular shell 106 around the central diameter line D used in the abduction angle α visualization. A typical acetabular shell 106 may have an anteversion angle υ ranging from about 10° to about 30°, or from about 10° to about 20°, or from about 15° to about 25°. In practice, it will be appreciated that the alignment of the acetabular shell 106 within the reamed acetabulum 111 is a compound angle that includes both the abduction angle α and the anteversion angle υ. Similarly, the alignment of the femoral stem 115 within the intramedullary bore 119 is a compound angle that includes both the abduction angle α and the anteversion angle υ.

[0047] Having the femoral stem 115 aligned with the acetabular shell 106 along a common anteversion angle (or anteversion plane) is one alignment parameter of a properly aligned prosthetic hip implant 102, so the anteversion angle υ of the femoral stem 115 typically has the same range of values ​​as the anteversion angle υ of the acetabular shell 106 (i.e., in the range of about 10° to about 30°, or about 10° to about 20°, or about 15° to about 25°). Positioning the femoral stem 115 in the intramedullary canal of the proximal femur 105 such that the longitudinal axis of the femoral stem is collinear with the anatomical axis of the femur 105 in which it is positioned is another alignment parameter for a properly aligned femoral component 104 relative to a properly aligned acetabular component 103 to together define a properly aligned prosthetic hip implant 102. The third alignment parameter for the femoral stem 115 is the vertical position of the prosthetic femoral head 113 relative to the natural femoral head of the surgical hip joint prior to resection (see 126).

[0048] FIG. 4 shows an acetabular component 103 that is misaligned with respect to the reamed acetabulum 111 and a femoral component 104 that is too short with respect to the reamed femoral canal (also known as the intramedullary bore 119). As shown, the abduction angle α and anteversion angle υ are excessive. When a patient moves his or her hip during normal use, the neck 116 may contact the rim 2 of the acetabular shell 106. Collectively, the rim 2 and the neck 116 may become the fulcrum of a lever that can displace the prosthetic femoral head 113 from the acetabular component 103. In addition, even if the femoral component 104 does not displace from the acetabular component 103, the force distribution of the femoral head 113 against the medial liner 107 may be overly concentrated in a relatively small area, thereby increasing wear and shortening the life of the implant 102.

[0049] FIG. 4 also shows a femoral stem 115 that is improperly sized and aligned relative to the intramedullary bore 119. Improper sizing can occur when a femoral reaming (also known as a "broaching") tool does not create a large enough intramedullary bore 119 to remove the cancellous bone around the medial cortical wall 120 of the femur 105. Over time, the femoral stem 115 compresses any intermediate cancellous bone between the lateral side of the femoral stem 115 and the medial cortical wall 120, which causes the femoral stem to sink within the intramedullary bore 119 and become misaligned relative to the proximal femur 105. Variability in broach placement can also result in varus canting, where the longitudinal axis of the femoral stem 115 is positioned at a varus angle relative to the anatomical axis (i.e., the central, or longitudinal, axis) of the distal femur 105. For example, it is common for the surgeon to contact the outer cortical wall 120 of the intramedullary canal 119 above the desired location of the femoral stem 115. The surgeon may stop reaming when the outer cortical wall 120 of the intramedullary canal 119 is contacted, believing that the patient has a narrow intramedullary canal 119 that will only accommodate a small femoral stem 115. In reality, the longitudinal axis of the femoral stem 115 is positioned at a varus angle relative to the anatomical axis of the distal femur 105. This subsidence and misalignment may ultimately change the length of one of the patient's legs relative to the other leg, thus altering the patient's gait. Stopped gait changes the force distribution through the patient's body, which may further accelerate wear of the internal prosthetic hip implant 102 as well as wear of healthy cartilage 123 on the remaining natural hip joint 101b.

[0050] Subsidence and malalignment of the femoral component 104 relative to the distal femur 105 can be particularly difficult to achieve and confirm with traditional 2D X-ray films. This is because the femoral component 104 is inserted into the proximal femur 105 through a 6-8 inch main incision. The surgeon's view of the insertion is limited by the minimally invasive nature of the procedure, and the femoral component is no longer visible to the unaided eye once it has entered the intramedullary bore 119. Traditional 2D intraoperative X-ray films (such as fluoroscopic images) do not show a third dimension and therefore cannot provide an accurate real-world depiction of the surgical field 170 in 3D space.

[0051] Patient comfort and implant longevity will likely depend in part on the placement and size of the prosthetic hip implant 102. In general, the more closely the placement of a properly sized implant replicates the natural kinematics of the pre-diseased joint, the longer the implant can be expected to last and the more comfortable the patient will be expected to feel.

[0052] In recent years, it has become possible to create 3D models of the surgical area using 2D images, such as X-rays. These models can be used preoperatively to plan the surgery much closer to the date of the actual surgery. These models can also be used intraoperatively (e.g., when projected onto a display or across the surgeon's field of view).

[0053] However, radiographs have not typically been used previously as input for 3D models due to concerns about image resolution and accuracy. Radiographs are 2D representations of 3D space. Thus, 2D radiographs necessarily distort the imaged object compared to the actual object existing in the third dimension. Furthermore, the object through which the X-rays pass can deflect the path of the X-rays as they move from the X-ray source 21 (typically the anode of the X-ray machine; see FIG. 12) to the X-ray detector 33 (as non-limiting examples, an X-ray image intensifier, phosphorus material, flat panel detector "FPD" (which may include indirect conversion FPD and direct conversion FPD), or any number of digital or analog X-ray sensors or X-ray films; see FIG. 12). Imperfections in the X-ray machine (1800, see FIG. 12) itself or its calibration can also impair the usefulness of X-ray photogrammetry and 3D model reconstruction. In addition, the emitted X-ray photons have different energies. When X-rays interact with matter disposed between the X-ray source 21 and the detector 33, noise and artifacts may be generated in part due to Compton and Rayleigh scattering, the photoelectric effect, exogenous variations in the environment, or intrinsic variations in the X-ray generating unit, the X-ray detector, and / or the processing unit or display.

[0054] Furthermore, in a single 2D image, the 3D data of the actual object is lost. Thus, there is no data that the calculator 1600 (e.g., a computer) can use from a single 2D image to reconstruct a 3D model of the actual 3D object. For this reason, CT scans, MRIs, and other imaging techniques that store three-dimensional data have often been the preferred inputs for reconstructing a model of one or more target orthopedic elements (i.e., reconstructing a 3D model from actual 3D data to generally generate a more accurate and higher resolution model). However, certain exemplary embodiments of the present disclosure discussed below overcome these problems by using deep learning networks to improve the accuracy of the reconstructed 3D model generated from an X-ray input image.

[0055] There are various methods for generating a 3D model from 2D pre- or intraoperative images. As an example, one such method may include receiving a set of 2D radiographic images of the patient's surgical field 170 using a radiographic imaging system, and calculating a first 3D model using epipolar geometry principles with the coordinate system of the radiographic imaging system and projective geometry data from each 2D image (see FIG. 8 and FIG. 9A, FIG. 9B, and FIG. 9C). Such an exemplary method may further include adjusting the initial 3D model by projecting the first 3D model on the 2D radiographic images, and then registering the first and second radiographic images 30, 50 on the first 3D model with an inter-image registration technique. Once the inter-image registration technique is applied, a modified 3D model may be generated. This process may be repeated until a desired clarity is achieved.

[0056] As another example, deep learning networks such as convolutional neural networks ("CNN"), recurrent neural networks ("RNN"), modular neural networks, or sequence to sequence models ("Sequence to Sequence"). A deep learning network (also known as a “Deep Learning Network, DNN”) may be used to generate a 3D model of a target orthopedic element (i.e., a modeled orthopedic element 100b) from a set of at least two 2D images of a patient's surgical field 170. The 2D input images 30, 50, etc. are desirably tissue-transmitted images such as radiological images (e.g., x-ray or fluoroscopic images). In such a method, the deep learning network may generate a model from projective geometry data (i.e., spatial data 43 or volumetric data 75) from the respective 2D images. The deep learning network may have the advantage of being able to generate masks of different target orthopedic elements 100 (e.g., bones, soft tissues, etc.) in the surgical field 170, as well as to calculate the volume (see 61 in FIG. 7 ) of one or more imaged orthopedic elements 100. In an exemplary embodiment, the identified orthopedic elements 100 or internal The dimensions of the components of the prosthetic implant assembly 102 may be mapped to spatial data 43 (FIG. 8) derived from the input images 30, 50 (FIG. 8) to ascertain the location of the identified orthopedic element 100 or the component of the endoprosthetic implant assembly in 3D space. In this manner, the location of the identified orthopedic element 100 and the component of the endoprosthetic implant (e.g., the acetabular component 104 or the femoral component 103) may be ascertained relative to one another. When this information is displayed to the surgeon and updated in real-time or near real-time based on the surgeon's repositioning of the implant component relative to the identified orthopedic element, the surgeon may use exemplary embodiments according to the present disclosure to accurately align the implant component relative to the identified orthopedic element in three dimensions while avoiding the limited field of view provided by the main incision.

[0057] Once the system is calibrated as described below, it is contemplated that new tissue transmission images (i.e., less than the number of input images required to calibrate the system) may be taken intraoperatively to update the reconstructed model of the surgical field (e.g., to refresh the positions of identified components of the endoprosthetic implant relative to other components of the endoprosthetic implant or identified orthopedic elements). In other exemplary embodiments, a number of new tissue transmission images equal to the number of input images selected to calibrate the system may be used to refresh the positions of components of the endoprosthetic implant relative to other components of the endoprosthetic implant or identified orthopedic elements in the system.

[0058] FIG. 5 is a flow chart outlining the steps of an exemplary method for locating an orthopedic feature in space. The method includes: L , X R8, the method includes steps 1a of calibrating a tissue penetration device, such as a radiographic imager 1800, to define spatial data 43 in order to determine a mapping relationship between the orthopedic element 100 and corresponding spatial coordinates (e.g., x and y coordinates; FIG. 8); step 2a of capturing a first image 30 (FIG. 8) of the orthopedic element 100 using a radiographic technique, the first image 30 defining a first reference frame 30a; step 3a of capturing a second image 50 (FIG. 8) of the orthopedic element 100 using a radiographic technique, the second image 50 defining a second reference frame 50a, the first reference frame 30a being offset from the second reference frame 50a by an offset angle θ; and step 3b of detecting the orthopedic element 100 using the spatial data 43. The orthopedic element 100 includes a step 4a of using a deep learning network to apply a mask to the identified orthopedic element 100a defined by the anatomical landmarks, the spatial data 43 defining anatomical landmarks on or within the orthopedic element 100, the detected orthopedic element being an identified orthopedic element 100a; a step 5a of using a deep learning network to apply a mask to the identified orthopedic element 100a defined by the anatomical landmarks; and a step of projecting the spatial data 43 from the first image 30 of the identified orthopedic element 100a and the spatial data 43 from the second image 50 of the identified orthopedic element 100a to define volumetric data 75 (FIG. 7). L , X R ) has a first value and corresponds to an image point (e.g., X L , X R) has a second value, the first value being different from the second value, step 7a applying a deep learning network to the volumetric data 75 to define a modeled orthopedic element 100b to generate a reconstructed 3D model of the orthopedic element, and step 8a mapping the three-dimensional modeled orthopedic element 100b to the spatial data 43. In another exemplary embodiment, step 4a can include using the deep learning network to detect spatial data 43 defining anatomical landmarks on or in the orthopedic element 100.

[0059] FIG. 6 is a flow chart outlining the steps of another exemplary method for locating an orthopedic feature in space. The method includes: L , X R 1b calibrating a tissue penetration imager, such as a radiographic imager, to determine a mapping relationship between the orthopedic element 100 and corresponding spatial coordinates (e.g., x and y coordinates) to define spatial data 43; capturing a first image 30 of the orthopedic element 100 using a radiographic technique, the first image 30 defining a first reference frame 30a; capturing a second image 50 of the orthopedic element 100 using a radiographic technique, the second image 50 defining a second reference frame 50a, the first reference frame 30a being offset from the second reference frame 50a by an offset angle θ; and using a deep learning network to map the identified orthopedic element 100 to the corresponding spatial coordinates (e.g., x and y coordinates). Detecting the orthopedic element 100 using spatial data 43 to define the orthopedic element 100a, step 4b, the spatial data 43 defining anatomical landmarks on or within the orthopedic element 100; using a deep learning network to apply a mask to the identified orthopedic element 100a defined by the anatomical landmarks, step 5b; projecting the spatial data 43 from the first image 30 of the identified orthopedic element 100a and the spatial data 43 from the second image 50 of the identified orthopedic element 100a to define volumetric data 75, wherein image points (e.g., XL , X R ) has a first value and corresponds to an image point (e.g., X L , X R ) has a second value, the first value being different from the second value; step 7b applying a deep learning network to the volumetric data 75 to generate a reconstructed 3D model of the orthopedic element 100b to define the orthopedic element 100b; and step 8b mapping the modeled orthopedic element 100b to the spatial data 43, the orthopedic element being a reamed acetabulum 111 of the pelvis 110. In another exemplary embodiment, step 4b can include using the deep learning network to detect spatial data 43 defining anatomical landmarks on or within the identified orthopedic element 100a.

[0060] It will be appreciated that in certain exemplary embodiments, the deep learning networks may be the same deep learning networks separately trained to perform distinct tasks (e.g., identifying the orthopedic elements 100 to define the identified orthopedic elements 100a, applying a mask to the identified orthopedic elements 100a, modeling the identified orthopedic elements 100a to define the modeled orthopedic elements 100b, etc.) In other exemplary embodiments, different deep learning networks may be used to perform one or more of the discrete tasks.

[0061] It is contemplated that the exemplary methods and systems according to the present disclosure may be used in connection with total hip arthroplasty ("THA"). In such exemplary embodiment, the orthopedic element 100 may be the femur 105, the femoral head 126, the pelvis 110, the acetabular cavity of the pelvis (e.g., the natural acetabulum 108 or the reamed acetabulum 111), and other bony anatomical landmarks present in or near the surgical field 170. However, it will be understood that nothing in this disclosure limits the application of the exemplary systems and methods to use in a THA procedure. It is contemplated that the exemplary systems and methods may be useful in any surgical procedure in which the presence of a significant amount of tissue generally obstructs the view of the orthopedic element 100 or the surgical field 170. Surgeries involving the shoulder, knee, or spine may be prime examples. Pediatric cardiothoracic procedures may be another example. Systems and methods according to the present disclosure may also be useful in wrist and ankle procedures, even though the surgeon's view is generally less obscured by surrounding tissue than in shoulder, hip, and spine procedures.

[0062] The above examples are provided for purposes of aiding understanding and are not intended to limit the scope of the present disclosure. Any method for generating a 3D model from 2D radiographic images of the same object taken from at least two lateral positions is considered to be within the scope of the present disclosure.

[0063] 7 and 8 show how a first input image 30 and a second input image 50 may be combined to create a volume 61 including volumetric data 75 (FIG. 7). In FIG. 7, the imaged surgical region 170 is of a knee joint. FIG. 7 provides an example of how a deep learning network may obtain volumetric data 75 from two calibrated input images 30, 50 offset from each other by an offset angle θ, and generate one or more modeled orthopedic elements 100b from the volumetric data 75. In FIG. 7, the surgical region 170 is of a knee joint.

[0064] 8 illustrates basic principles of epipolar geometry that can be used to transform spatial data 43 from each input image 30, 50 into volumetric data 75. The spatial data 43 is a set of image points (e.g., x, y coordinates) that are mapped to corresponding spatial coordinates (e.g., x and y coordinates) in a given input image 30, 50. L , X R It will be understood that the set of

[0065] FIG. 8 is a simplified schematic diagram of an oblique projection described by the pinhole camera model. FIG. 8 conveys basic concepts related to computer stereo vision, but is not meant to be the only way a 3D model can be reconstructed from 2D stereo images. In this simplified model, rays emanate from the optical center (i.e., the point in the lens where rays of electromagnetic radiation (e.g., visible light, x-rays, etc.) from the target object are assumed to intersect in the imager's sensor or detector array 33 (FIG. 12)). The optical center is indicated in FIG. 8 as point O L , O R In practice, the image plane (see 30a, 50a) is usually located at the optical center (e.g., O L , O R ), and the actual optical centre is projected as a point onto the detector array 33, but a virtual image plane (see 30a, 50a) is presented here to illustrate the principle more concisely.

[0066] The first input image 30 is taken from a first reference frame 30a, and the second input image 50 is taken from a second reference frame 50a that is different from the first reference frame 30a. Each image includes a matrix of pixel values. The first and second reference frames 30a, 50a are preferably offset from each other by an offset angle θ. The offset angle θ may represent the angle between the x-axis of the first reference frame 30a and the x-axis of the second reference frame 50a. In other words, the angle between the orientation of the orthopedic element in the first image and the orthopedic element in the second image may be known as the "offset angle."

[0067] point eL is the optical center O of the second input image on the first input image 30 R This is the location of point e R is the optical center O of the first input image on the second input image 50 L This is the location of point e L and e R is known as an "epipole" or epipolar point, and is the line O L -O R Above. Points X and O L , O R defines the epipolar plane.

[0068] Since the actual optical center is the point where the incident rays of electromagnetic radiation from the target object are assumed to intersect within the detector lens, in this model the rays of electromagnetic radiation are actually referred to as the optical center O for the purpose of visualizing how the location of a 3D point X in 3D space can be ascertained from two or more input images 30, 50 captured from detectors 33 of known relative positions. L , O R Each point of the first input image 30 (e.g., X L ) corresponds to a line in 3D space, then the corresponding point (e.g., X R ) can be found in the second input image, then these corresponding points (e.g., X L , X R ) must be the projection of a common 3D point X. Therefore, the corresponding image points (e.g., X L , X R ) should intersect at a 3D point X. In general, the value of X should be the sum of all corresponding image points (e.g., X L , X R), a 3D volume 61 including volume data 75 may be replicated from two or more input images 30, 50. The value of any given 3D point X may be triangulated in a variety of ways. A non-limiting list of exemplary computational methods includes midpoint method, direct linear transformation method, fundamental matrix method, intersection method, and bundle adjustment method. Furthermore, in certain exemplary embodiments, a deep learning network may be trained on a set of input images to establish a model for determining the location of a given point in 3D space based on two or more input images of the same object, where a first input image 30 is offset from a second input image 50 by an offset angle θ. It will be further understood that combinations of any of the above methods are within the scope of the present disclosure.

[0069] As used herein, an "image point" (e.g., X L , X R It will be appreciated that a 3D point X may refer to a point in space, a pixel, a portion of a pixel, or a collection of adjacent pixels. It will also be appreciated that as used herein, a 3D point X may represent a point in 3D space. In certain exemplary applications, the 3D point X may be represented as a voxel, a portion of a voxel, or a collection of adjacent voxels.

[0070] However, before the principles of epipolar geometry can be applied, the position of each image detector 33 relative to the other image detector(s) 33 must be determined (or the position of the single image detector 33 must be determined at the time the first image 30 is taken, and the adjusted position of the single image detector 33 must be known at the time the second image 50 is taken). It may also be desirable to determine the focal length and optical center of the imager 1800. To verify this in practice, the image detector 33 (or multiple image detectors) is first calibrated. Figures 9A, 9B, and 9C show calibration fixtures 973A, 973B, 973C for the orthopedic element 100 of interest. In these figures, the exemplary orthopedic element 100 includes the proximal surface of the femur 105 and the natural acetabulum 108 of the pelvis 110 including the hip joint 101.

[0071] While at least two input images 30, 50 are technically required to calibrate the exemplary system described herein, at least three input images 30, 50, 70 may be desirable when the input images are radiological input images and the target surgical region 170 involves a contralateral joint that cannot be easily isolated from radiological imaging. For example, the pelvis 110 includes a contralateral acetabulum 108. A direct medial-lateral radiograph of the pelvis 110 would show both the acetabulum proximal to the detector 33 and the acetabulum distal to the detector 33. However, due to the positioning of the pelvis 110 relative to the detector 33, and the lack of 3D data in a single 2D radiograph, the relative acetabulums appear superimposed on one another, making it difficult for a person or computer 1600 to distinguish which is the proximal acetabulum and which is the distal acetabulum.

[0072] To address this issue, at least three input images 30, 50, 70 can be used. In one exemplary embodiment, the first input image 30 can be a radiograph capturing an anterior-posterior perspective view of the surgical field 170 (i.e., an example of a first reference frame 30a). For the second input image 50, the patient or detector 33 can be rotated clockwise (which can be specified by a positive angle) or counterclockwise (which can be specified by a negative angle) relative to the patient's orientation for the first input image 30. For example, for the second input image 50, the patient can be rotated ±45° from the patient's orientation in the first input image 30. Similarly, the patient can be rotated clockwise or counterclockwise relative to the patient's orientation for the first input image 30. For example, for the third input image 70, the patient can be rotated ±45° relative to the patient's orientation in the first input image 30. It will be appreciated that if the second input image 50 has a positive offset angle (e.g., +45°) relative to the orientation of the first input image 30, the third input angle 70 preferably has a negative offset angle (e.g., -45°) relative to the orientation of the first input image 30, and vice versa.

[0073] In an exemplary embodiment, to calibrate the exemplary system, a principle or epipolar geometry may be applied to at least three input images 30, 50, 70 taken from at least three different reference frames 30a, 50a, 70a.

[0074] FIG. 9A is an anterior-posterior view of an exemplary orthopedic element 100 (e.g., proximal femur 105, natural acetabulum 108, pelvis 110, articular cartilage, other soft tissues, etc.) within an exemplary surgical field 170. That is, FIG. 9A represents a first image 30 taken from a first reference frame 30a (e.g., a first lateral position). A first calibration fixture 973A is attached to a first holding assembly 974A. The first holding assembly 974A may include a first padded support 971A engaged to a first strap 977A. The first padded support 971A is attached to the outside of the patient's thigh via the first strap 977A. The first holding assembly 974A supports the first calibration fixture 973A, which is desirably oriented parallel to the first reference frame 30a (i.e., orthogonal to the detector 33). The calibration fixture 973A is desirably positioned far enough away from the desired target orthopedic element 100 so that the calibration fixture 973A does not overlap any of the target orthopedic elements 100. Overlap would obscure the desired image data.

[0075] FIG. 9B is a view of the exemplary orthopedic elements 100 (e.g., proximal femur 105, natural acetabulum 108, pelvis 110, articular cartilage, other soft tissues, etc.) of the exemplary surgical field 170 of FIG. 9A offset by 45° in the positive direction from the first reference frame 30a. That is, FIG. 9B represents a second input image 50 taken from a second reference frame 50a (e.g., a second lateral position). A second calibration fixture 973B is attached to a second holding assembly 974B. The second holding assembly 974B may include a second padded support 971B engaged to a second strap 977B. The second padded support 971B is attached to the outside of the patient's thigh via the second strap 977B. The second holding assembly 974B supports a second calibration fixture 973B that is desirably oriented parallel to the second reference frame 50a (i.e., perpendicular to the detector 33). The calibration fixture 973B is desirably positioned far enough away from the target orthopedic elements 100 such that the calibration fixture 973B does not overlap any of the target orthopedic elements 100.

[0076] FIG. 9C is a view of the exemplary orthopedic elements 100 (e.g., proximal femur 105, natural acetabulum 108, pelvis 110, articular cartilage, other soft tissues, etc.) of the exemplary surgical field 170 of FIG. 9A offset negatively by 45° from the first reference frame 30a. That is, FIG. 9C represents a third input image 70 taken from a third reference frame 70a (e.g., a third lateral position). The third calibration fixture 973C is attached to a third holding assembly 974C. The third holding assembly 974C can include a third padded support 971C engaged with a third strap 977C. The third padded support 971C is attached to the outside of the patient's thigh via the third strap 977C. The third holding assembly 974C supports a third calibration fixture 973C that is desirably oriented parallel to the third reference frame 70a (i.e., perpendicular to the detector 33). The calibration fixture 973C is desirably positioned sufficiently far away from the target orthopedic element 100 such that the calibration fixture 973C does not overlap the target orthopedic element 100.

[0077] If the system is calibrated pre-operatively, the patient may be positioned in a standing position (i.e., legs extended) since the hip joint is stable in this orientation (see FIG. 12). If the system is calibrated intra-operatively, the patient may be lying supine on the operating table. Preferably, the distance of the patient relative to the imager should not be changed during acquisition of the input images 30, 50, 70. The first, second, and third input images 30, 50, 70 do not need to capture the entire leg, rather the images can be focused on the joint of interest in the surgical field 170.

[0078] It will be appreciated that only a single calibration fixture 973 may be used, depending on the target orthopedic features 100 being imaged and modeled. Similarly, multiple calibration fixtures 973 may be used if a particularly long collection of orthopedic features 100 is being imaged and modeled.

[0079] The calibration fixtures 973A, 973B, 973C are desirably of known size. Each calibration fixture 973A, 973B, 973C desirably has at least four or more calibration points 978 distributed throughout. The calibration points 978 are distributed in a known pattern in which the distance of one point 978 relative to another point is known. The distance from the calibration fixture 973 to the orthopedic element 100 is also desirably known. For calibration of a radiogrammetry system, the calibration points 978 may desirably be defined by metal structures on the calibration fixture 973. Metals typically absorb most of the x-ray beam that contacts the metal. Thus, metals typically appear very bright compared to materials that do not absorb x-rays much (such as air cavities or fatty tissue). Common exemplary structures that define the calibration points include, but are not limited to, a reseau cross, a circle, a triangle, a pyramid, and a sphere.

[0080] These calibration points 978 may be on the 2D surface of the calibration fixture 973, or the 3D calibration points 978 may be captured as 2D projections from a given image reference frame. In either situation, the 3D coordinate (commonly referred to as the z-coordinate) may be set equal to zero for all calibration points 978 captured in the image. The distance between each calibration point 978 is known. These known distances may be expressed as x,y coordinates in the image sensor / detector 33. To map a point in 3D space to a 2D coordinate pixel on the sensor 33, a dot product of the detector calibration matrix, the extrinsic matrix, and the homologous coordinate vector of the real 3D point may be used. This maps the real world coordinates of the point in 3D space to the calibration fixture 973. In other words, this generally allows the x,y coordinates of the real points in 3D space to be accurately transformed into the 2D coordinate plane of the image detector sensor 33 to define the spatial data 43 (see FIG. 8).

[0081] The above calibration method is provided as an example. It will be understood that any method suitable for calibrating a radiogrammetry system is considered to be within the scope of this disclosure. A non-limiting list of other radiogrammetry system calibration methods includes the use of Lithoplate, Zhang's method, bundle adjustment methods, direct linear transformation methods, maximum likelihood estimation, k-nearest neighbor regression methods ("KNN"), convolutional neural network ("CNN") based methods, other deep learning methods, or combinations thereof.

[0082] 7 illustrates the principle of how two calibrated input images 30, 50, when oriented along a known offset angle θ, can be backprojected into a 3D volume 61 that contains two channels 65 and 66. The first channel 65 projects all image points (e.g., X L etc.), and the second channel 66 contains all image points (e.g., X R3D ray 75. The backprojected 2D input images 30, 50 may then be used to generate a volume 61 of the imaged surgical region 170, including volumetric data 75. If a third input image 70 is used, there may be a third channel that includes all of the image points of the third input image 70. That is, each image point (e.g., pixel) is replicated onto its associated backprojected 3D ray. Epipolar geometry may then be used to generate a volume 61 of the imaged surgical region 170, including volumetric data 75, from these backprojected 2D input images 30, 50. If a third input image 70 is used, there may be a third channel that includes all of the image points of the third input image 70.

[0083] Referring to FIG. 7, the first input image 30 and the second input image 50 desirably have known image dimensions. The dimensions may be in pixels. For example, the first image 30 may have dimensions of 164×164 pixels. The second image 50 may have dimensions of 164×164 pixels. The dimensions of the input images 30, 50 used in certain calculations desirably have consistent dimensions. Consistent dimensions may be desirable for later defining a cubic working area of ​​a typical volume 61 (e.g., a 164×164×164 cube). In an embodiment, the offset angle θ is desirably 45° between each adjacent input image. However, in other exemplary embodiments, other offset angles θ may be used. For example, in FIG. 7, the offset angle θ is 90°.

[0084] In the illustrated example, each of the 164×164 pixel input images 30, 50 is replicated 164 times across the length of the adjacent input image to create a volume 61 having dimensions of 164×164×164 pixels. That is, the first image 30 is copied and stacked behind itself by 164 pixels, one copy per pixel, while the second image 50 is copied and stacked behind itself by 164 pixels such that the stacked images overlap, thereby creating the volume 61. In this manner, the volume 61 can be said to include two channels 65, 66, where the first channel 65 includes the first image 30 replicated n times across the length of the second image 50 (i.e., the x-axis of the second image 50) and the second channel 66 includes the second image 50 replicated m times across the length of the first image 30 (i.e., the x-axis of the first image 30), where "n" and "m" are the commanded image length expressed as the number of pixels (or other dimensions in other exemplary embodiments) that comprise the commanded image length. When the offset angle θ is known, each lateral slice (also known as an "axial slice" by some radiologists) of the volume 61 creates an epipolar plane that includes voxels backprojected from the pixels that comprise the two epipolar lines. In this manner, the spatial data 43 from the first image 30 of the target orthopedic element 100 and the spatial data 43 from the second image 50 of the target orthopedic element 100 are projected to define the volumetric data 75. Using this volumetric data 75, a 3D representation can be reconstructed using epipolar geometry principles as described above, which is geometrically consistent with the information in the input images 30,50.

[0085] In the exemplary systems and methods for identifying orthopedic elements and / or endoprosthetic implant components in space using a deep learning network and the exemplary systems and methods for locating orthopedic elements and endoprosthetic implant components in space using a deep learning network, where the deep learning network is a CNN, detailed examples are provided of how the CNN can be structured and trained. All architectures of CNNs are considered within the scope of the present disclosure. Common CNN architectures include, for example, LeNet, GoogLeNet, AlexNet, ZFNet, ResNet, and VGGNet.

[0086] Preferably, the methods disclosed herein may be implemented on a computer platform having hardware such as one or more central processing units (CPU), a random access memory (RAM), and / or input / output (I / O) interface(s) (see 1600).

[0087] FIG. 10 is a schematic diagram of a CNN illustrating how a CNN can be used to identify edges of an orthopedic element 100 of interest. Without being bound by theory, it is believed that a CNN may be desirable to reduce the size of the volumetric data 75 without losing features necessary to identify the desired orthopedic element 100 or its surface topography. The volumetric data 75 of the multiple backprojected input images 30, 50 is a multidimensional array that may be known as an "input tensor." This input tensor contains the input data for the first convolution, which in this example is the volumetric data 75. A filter (also known as a kernel 69) is shown placed on the volumetric data 75. The kernel 69 is a tensor (i.e., a multidimensional array) that defines a filter or function (this filter or function may be known as a "weight" that is applied to the kernel). In the illustrated embodiment, the kernel tensor 69 is three-dimensional. The filter or function, including the kernel 69, may be manually programmed or learned through a CNN, RNN, or other deep learning network. In the illustrated embodiment, kernel 69 is a 3x3x3 tensor, however, all tensor sizes and dimensions are considered within the scope of this disclosure as long as the kernel tensor size is smaller than the size of the input tensor.

[0088] Each cell or voxel of the kernel 69 has a numerical value. These values ​​define the filter or function of the kernel 69. A convolution or cross-correlation operation is performed between two tensors. In FIG. 10, the convolution is represented by a path 76. The path 76 that the kernel 69 follows is a visualization of the mathematical convolution operation. By following this path 76, the kernel 69 eventually traverses the entire volume 61 of the input tensor (e.g., volumetric data 75) sequentially. The goal of this operation is to extract features from the input tensor.

[0089] The convolutional layers 72 typically include one or more of a convolution stage 67, a detector stage 68, and a pooling stage 58. Although each of these operations is visually represented in the first convolutional layer 72a in FIG. 10, it will be understood that subsequent convolutional layers 72b, 72c, etc. may also include one or more, or all, of the convolutional stage 67, detector stage 68, and pooling layer 58 operations, or combinations or permutations thereof. Additionally, while FIG. 10 shows five convolutional layers 72a, 72b, 72c, 72d, 72e of various resolutions, it will be understood that in other exemplary embodiments, more or fewer convolutional layers may be used.

[0090] In the convolution stage 67, the kernel 69 is multiplied sequentially by multiple patches of pixels in the input data (i.e., in the illustrated example, volumetric data 75). The patches of pixels extracted from the data are known as receptive fields. The multiplication of the kernel 69 with the receptive field involves an element-wise multiplication between each pixel of the receptive field and the kernel 69. After multiplication, the results are summed to form one element of the convolution output. This kernel 69 then shifts to an adjacent receptive field and the element-wise multiplication operation and summation continues until all pixels of the input tensor have been subjected to this operation.

[0091] Up to this stage, the input data in the input tensor (e.g., volumetric data 75) is linear. A nonlinear activation function is then used to introduce nonlinearity into this data. The use of such a nonlinear function marks the beginning of the detector stage 68. A common nonlinear activation function is the Rectified Linear Unit function ("ReLU") given by the function:

[0092]

number

[0093] When used with a bias, the nonlinear activation function serves as a threshold to detect the presence of features extracted by the kernel 69. For example, applying a convolution or cross-correlation operation between the input tensor and the kernel 69 produces a convolution output tensor, where the kernel 69 includes a low-level edge filter in the convolution stage 67. The feature map output tensor is then returned by applying the nonlinear activation function with the bias to the convolution output tensor. The bias is added sequentially to each cell of the convolution output tensor. If for a given cell, the sum is greater than or equal to 0 (assuming that ReLU is used in this example), the sum is returned to the corresponding cell of the feature map output tensor. Similarly, if the sum is less than 0 for a given cell, the corresponding cell of the feature map output tensor is set to 0. Thus, applying the nonlinear activation function to the convolution output behaves like a threshold to determine whether and how closely the convolution output matches a given filter of the kernel 69. In this manner, the non-linear activation function detects the presence of desired features from the input data (eg, volumetric data 75 in this example).

[0094] All non-linear activation functions are considered to be within the scope of this disclosure. Other examples include Sigmoid, TanH, Leaky ReLU, parametric ReLU, Softmax, and Switch activation functions.

[0095] However, a drawback of this approach is that the feature map output of this first convolutional layer 72a records the exact location of the desired feature (edge, in the above example). Thus, small movements of the feature in the input data will generate different feature maps. To address this issue and reduce computational power, downsampling is used to reduce the resolution of the input data while still preserving important structural elements. Downsampling can be achieved by changing the stride of the convolutions along the input tensor. Downsampling is also achieved by using a pooling layer 58.

[0096] Effective padding may be applied to reduce the dimensions of the convolved tensor (see 72b) compared to the input tensor (see 72a). A pooling layer 58 is desirably applied to reduce the spatial size of the convolved data, which reduces the computational power required to process the data. Common pooling techniques may be used, including max pooling and average pooling. Max pooling returns the maximum value of the portion of the input tensor covered by the kernel 69, while average pooling returns the average of all values ​​of the portion of the input tensor covered by the kernel 69. Using max pooling, image noise may be reduced.

[0097] In certain exemplary embodiments, a fully connected layer can be added after the final convolutional layer 72e to learn a nonlinear combination of the high-level features represented by the output of the convolutional layers (e.g., the profile of the imaged natural acetabulum 108, the profile of the reamed acetabulum 109, or the surface topology of the orthopedic component).

[0098] When used on an orthopedic element 100, the above description of a CNN-type deep learning network is one example of how a deep learning network can be configured to “identify” the orthopedic element 100 to define an “identified orthopedic element” 100a.

[0099] The top half of FIG. 10 represents the compression of the input volume data 75, and the bottom half represents the decompression until the original size of the input volume data 75 is reached. The output feature map of each convolutional layer 72a, 72b, 72c, etc. is used as the input of the subsequent convolutional layer 72b, 72c, etc. to enable increasingly complex feature extraction. For example, the first kernel 69 may detect edges, a kernel in the first convolutional layer 72b may detect a set of edges of a desired orientation, a kernel in the third convolutional layer 72c may detect a longer set of edges of a desired orientation, etc. This process may continue until the entire profile of the desired orthopedic element 100 has been detected and identified by downstream convolutional layers 72.

[0100] The bottom half of FIG. 10 is an upsample (i.e., expanding the spatial support of the lower resolution feature map). A deconvolution operation is performed to increase the size of the input for the next downstream convolutional layer (see 72c, 72d, 72e). For the final convolutional layer 72e, convolution may be employed with a 1×1×1 kernel 69 to generate a multi-channel output volume 59 of the same size as the input volume 61. Each channel of the multi-channel output volume 59 may represent a desired extracted high-level feature. This may be followed by a Softmax activation function to detect the desired orthopedic element 100. For example, the illustrated embodiment may have five output channels numbered 0, 1, 2, 3, and 4, where channel 0 represents the identified background volume, channel 1 represents the identified proximal femur 105, channel 2 represents the identified reamed acetabulum 111, channel 3 represents the identified acetabular component 103, and channel 4 represents the identified femoral component 104.

[0101] It will be appreciated that in other exemplary embodiments, fewer or more output channels may be used, and it will also be appreciated that the output channels provided may represent different components of the orthopedic element 100 and endoprosthetic implant than those enumerated herein.

[0102] For example, in an exemplary embodiment where the system is configured to identify an orthopedic element 100, which is the medial cortical wall 120 of the proximal femur 105, and where the system is configured to identify an endoprosthetic implant component, which is a trial component construct, the exemplary embodiment may include three output channels numbered 0, 1, and 2, where channel 0 represents the identified background volume, channel 1 represents the medial cortical wall 120 of the proximal femur 105, and channel 2 represents the identified femoral component 104. The "trial component construct" used in the above example, when describing a construct used on the proximal femur 105, may include a broach, a trial neck, and a trial head assembly, or a trial stem.

[0103] In an exemplary embodiment in which the system is configured to identify an orthopedic element 100, which is a reamed acetabulum 111 of a pelvis 110, and in which the system is configured to identify an endoprosthetic implant component, which is an acetabular component 103 or a trial acetabular component, the exemplary embodiment may include three output channels numbered 0, 1, and 2, where channel 0 represents the identified background volume, channel 1 represents the reamed acetabulum 111 of the pelvis 110, and channel 2 represents the acetabular component 103 or the trial acetabular component.

[0104] Such an exemplary embodiment may optionally include additional output channels, such as an output channel representing the outer wall of the proximal femur 105. Other output channels may be used to output the abduction angle α and anteversion angle υ of the identified component of the endoprosthetic implant, respectively, relative to the identified orthopedic element 100a in which the component of the endoprosthetic implant is seated. Still other output channels may be used to output (as non-limiting examples) the determined size dimension of the identified orthopedic element, the recommended component type / product model of the endoprosthetic implant, the recommended component size of the endoprosthetic implant, a "best fit" output of the recommended component or recommended component size relative to the dimensions of the inner cortical wall 120, the alignment calculation of the longitudinal axis of the femoral component 104, the trial component build relative to the anatomical axis of the proximal femur 105, the calculated center of the acetabulum, or the alignment of the longitudinal axis of the neck of the femoral component 104 relative to the center of the prosthetic femoral head 113. Combinations of any of the foregoing are considered within the scope of the present disclosure.

[0105] When used with an endoprosthetic implant component or its subcomponents, the above description of a CNN-type deep learning network is an example of how a deep learning network can be "configured to identify" an endoprosthetic implant component (or its subcomponents) to define an identified endoprosthetic implant component. When used with an endoprosthetic implant, the above description of a CNN-type deep learning network is an example of how a deep learning network can be "configured to identify" an endoprosthetic implant to define an "identified endoprosthetic implant". When applied to a plurality of orthopedic elements, a plurality of components of an endoprosthetic implant, a plurality of endoprosthetic implants, or a combination thereof, it will be further understood that the above description of a CNN-type deep learning network is an example of how a deep learning network can be "configured to identify" a plurality of orthopedic elements, a plurality of components of an endoprosthetic implant, their subcomponents, a plurality of endoprosthetic implants, or a combination thereof, as the case may be. Other deep learning network architectures known or readily ascertainable by those skilled in the art are also considered to be within the scope of the present disclosure.

[0106] In an exemplary embodiment, a selected output channel containing the output volume data 59 of the desired orthopedic element 100 can be used to generate a modeled orthopedic element 100b, a modeled component of an endoprosthetic implant (e.g., an acetabular cup 106, a femoral stem 115, etc.). In certain exemplary embodiments, the modeled orthopedic element 100b is a computer model. In other exemplary embodiments, the modeled orthopedic element 100b is a physical model.

[0107] While the above example describes the use of a three-dimensional tensor kernel 69 to convolve the input volumetric data 75, it will be appreciated that the above general model may be used with 2D spatial data 43 from any of the calibrated input images 30, 50, 70. In other exemplary embodiments, a machine learning algorithm (i.e., a deep learning network (e.g., CNN, etc.)) may be used after calibration of the imager 1800, but before 2D to 3D reconstruction. That is, a CNN may be used to detect features (e.g., anatomical landmarks) of the target orthopedic element 100 from the first reference frame 30a, the second reference frame 50a, or the third reference frame 70a of the 2D input image 30, 50, 70. In an exemplary embodiment, the CNN may be used to identify high-level orthopedic features (e.g., the proximal femur 105 and part of the surface topology of the target orthopedic feature 100), components of an endoprosthetic implant (e.g., the acetabular cup 106, the femoral stem 115, etc.), or the endoprosthetic implant itself (e.g., the hip endoprosthetic implant 102) from the 2D input images 30, 50, 70. The CNN may then, optionally, apply a mask or contour to the detected orthopedic feature 100, components of the endoprosthetic implant, or the endoprosthetic implant itself. If the imager 1800 is calibrated, and if the CNN detects multiple corresponding image points (e.g., X L , X R), it is believed that a transformation matrix between the reference frames 30a, 50a of the orthopedic element 100 of interest, the components of the endoprosthetic implant, or the endoprosthetic implant itself can be used to align multiple corresponding image points in 3D space. In this way, the location of the point in the 3D space can be determined to correspond to a set of coordinates in the 3D space. In this way, the 3D point can be said to be "mapped" to the spatial data. A deep learning network that can model this relationship in this way or in other ways developed by the deep learning network can be said to be "configured" to "map" the identified orthopedic element, the identified components of the endoprosthetic implant, and / or the endoprosthetic implant itself (as the case may be) to the spatial data confirmed by the first input image 30 and the second input image 50 (and possibly the third input image 70 or further input images or "new" refresh images), thereby determining the location of the identified orthopedic element 100a, the identified components of the endoprosthetic implant, and / or the endoprosthetic implant itself (as the case may be) in three-dimensional space.

[0108] In embodiments where any of the first input image 30, the second input image 50, or the third input image 70 are radiological x-ray images (including, but not limited to, fluoroscopic radiological images), training a CNN may present some challenges. In comparison, a CT scan typically produces a series of images of a desired volume. Each CT image, including a typical CT scan, may be thought of as a segment of the imaged volume. From these segments, a 3D model may be relatively easily created by adding the area of ​​the desired element as it is shown in each successive CT image. The modeled element may then be compared to the data from the CT scan to ensure accuracy. One drawback of CT scanning is that it exposes the patient to excessive amounts of radiation (approximately 70 times the radiation dose of one conventional radiograph).

[0109] In contrast, radiographic imaging systems typically do not generate sequential images that capture different segments of the imaged volume. Rather, all of the information of the image is flattened in a 2D plane. In addition, because a single radiographic image 30 inherently lacks 3D data, it is difficult to check the model generated by the epipolar geometric reconstruction technique described above with the actual geometry of the target orthopedic element 100. To address this issue, the CNN may be trained with CT images, such as digitally reconstructed radiograph ("DRR") images. By training the deep learning network in this manner, the deep learning network may develop unique weights (e.g., filters) for the kernel 69 to identify the surface topography of the desired orthopedic element 100 or the target orthopedic element 100. Because radiographs have a different appearance than DRRs, an image-to-image transformation may be performed to render the input x-ray image to have a DRR-style appearance. An exemplary image-to-image transformation method is the CycleGAN image transformation technique. In embodiments where an image-to-image style transfer method is used, the style transfer method is desirably used before inputting the data into a deep learning network for feature detection.

[0110] The above examples are provided for purposes of aiding understanding and are not intended to limit the scope of the present disclosure. All methods for generating a 3D model 100 of a target orthopedic element from 2D radiographic images of the same target orthopedic element 100 taken from at least two lateral positions (e.g., 30a, 50a) are considered to be within the scope of the present disclosure.

[0111] Determining the thickness and boundaries of a particular identified orthopedic element 100a, endoprosthetic implant component, and / or endoprosthetic implant itself, as well as their exact coordinates in 3D space, allows the location of the identified orthopedic element 100a, endoprosthetic implant component, and / or endoprosthetic implant, endoprosthetic implant component, and / or endoprosthetic implant to be known while bypassing the limited view provided to the surgeon through the main incision. When the location of the identified orthopedic element 100a is known and the location of the identified components of the endoprosthetic implant are known, this information can be used to check against the desired alignment parameters of the implant components relative to the identified orthopedic element 100a on which they are to be placed (e.g., acetabular shell 106 placed in the reamed acetabulum 111, femoral stem 115 placed in the intramedullary canal of the proximal femur 105, etc.).

[0112] Similarly, if the position of a first component (e.g., the acetabular component) of an endoprosthetic implant is known in three dimensions relative to a second component (e.g., the femoral component) of the endoprosthetic implant, the surgeon can use the exemplary systems and methods described herein to assess the placement, and thus the alignment, of the first component relative to the second component. The surgeon can re-image the surgical area and update the position of the first component relative to the second component at subsequent time intervals until the surgeon is satisfied with the alignment. It is contemplated that such alignment may be performed intraoperatively to mitigate issues of misaligned components of a multi-component endoprosthetic implant.

[0113] In certain exemplary embodiments involving using a deep learning network to add masks or contours to the detected 2D orthopedic elements 100 from the respective input images 30, 50, 70, only the 2D masks or contours of the identified orthopedic elements 100 components of the endoprosthetic implant and / or the endoprosthetic implant may be sequentially backprojected in the manner described with reference to Figures 7 and 8 above to define the identified orthopedic elements 100, components of the endoprosthetic implant and / or volumes 61 of the endoprosthetic implant. In this exemplary method, a modeled orthopedic element 100b, a modeled component of the endoprosthetic implant and / or a modeled endoprosthetic implant may be generated.

[0114] 11 is an expanded view of a modeled endoprosthetic implant 102b including several modeled endoprosthetic implant components, namely a modeled acetabular component 103b and a modeled femoral component 104b. The modeled acetabular component 103b includes a modeled acetabular shell 106b and a modeled acetabular liner 107b. The modeled femoral component 104b includes a modeled femoral stem 115b, a modeled femoral stem neck 116b, and a modeled prosthetic femoral head 113b.

[0115] The exemplary system or method may further include calculating the center of the acetabulum. Such exemplary system or method may further include aligning the longitudinal rotation axis of the femoral stem implant with the center of the acetabulum. In yet another exemplary system or method, the longitudinal axis of the femoral stem 115 may be aligned (i.e., collinear) with the longitudinal axis of the femur 105. In yet another exemplary system and method, the rotation axis of the neck of the femoral stem 115 may be aligned with the center of the prosthetic femoral head 113. In yet another exemplary system and method, the position of the prosthetic head 113 may be aligned (e.g., perpendicular) with the pre-diseased natural femoral head based on an input image of the natural femoral head. In such an exemplary embodiment, the longitudinal axis of the femoral stem 115 is desirably collinear with the anatomical axis of the femur 105, and the femoral stem 115 is desirably positioned at an anteversion angle υ in the range of about 10° to about 30°, desirably about 15° to about 25°.

[0116] A computer platform having hardware such as one or more central processing units ("CPU"), random access memory ("RAM"), and input / output ("I / O") interface(s) may receive at least two 2D radiographic images taken at different orientations along the lateral plane. The orientations may be orthogonal to one another (i.e., a first frame of reference has an offset angle θ of 90° relative to a second frame of reference). However, in embodiments in which the orthopaedic element 100 includes a hip joint 101, at least three 2D radiographic input images may be desirable to avoid interference from the contralateral acetabulum. In such an exemplary embodiment, the offset angle θ may be desirably 45° between adjacent frames of reference. In other exemplary embodiments, other obtuse or acute offset angles θ may be used.

[0117] Referring to FIG. 12, an exemplary system for verifying the position of the orthopedic element 100 and components of the endoprosthetic implant in space may include a tissue transmission imager 1800 (e.g., a radiographer, a fluoroscopy imager, etc.) including an emitter 21 and a detector 33, where the detector 33 of the radiographer 1800 captures a first input image 30 (FIGS. 8 and 9A) at a first lateral position 30a (FIGS. 8 and 9A) and a second input image 50 (FIGS. 8 and 9B) at a second lateral position 50a (FIGS. 8 and 9B), where the first lateral position 30a is offset from the second lateral position 50a by an offset angle θ (FIG. 8). In an exemplary embodiment including at least three input images, the detector 33 of the radiation imaging device 1800 captures a third input image 70 at a third lateral position 70a (Figures 8 and 9C), which is offset by two separate offset angles θ1, θ2 from the second lateral position 50a and the first lateral position 30a.

[0118] The exemplary system may further include a transmitter 29 (FIG. 12) and a computer 1600 (see FIG. 13 for further details), which transmits the first input image 30 and the second input image 50 (and optionally the third input image 70, if present) from the detector 33 to the computer 1600, which is configured to identify the orthopedic element 100a, a component of the endoprosthetic implant, a sub-component of a component of the endoprosthetic implant, or the endoprosthetic implant itself using one of the deep learning methods discussed herein. It will be understood that the exemplary system disclosed herein may be used pre-operatively, intra-operatively, and / or post-operatively.

[0119] In certain exemplary embodiments, the exemplary system may further comprise a display 19 .

[0120] FIG. 12 is a schematic diagram of an exemplary system including a radiographer 1800 including an X-ray source 21, such as an X-ray tube, a filter 26, a collimator 27, and a detector 33. In FIG. 12, the radiographer 1800 is shown from top to bottom. The illustrated radiographer 1800 is a type of tissue transmission imager. A patient 1 is placed between the X-ray source 21 and the detector 33. The radiographer 1800 may be mounted on a rotatable gantry 28. The radiographer 1800 may acquire a first radiograph input image 30 of the patient 1 from a first reference frame 30a. The gantry 28 may then rotate the radiographer 1800 by an offset angle. The radiographer 1800 may then acquire a second radiograph input image 50 from a second reference frame 50a. It will be understood that other exemplary embodiments may include using multiple input images taken at multiple offset angles θ. For example, in a hip arthroplasty, the radiographer 1800 may be further rotated (or the patient may be rotated) to capture a third radiological input image 70 from a third reference frame 70a. In such embodiments, the offset angle may be less than or greater than 90° between adjacent input images.

[0121] It will be understood that the offset angle need not be exactly 90 degrees in all embodiments. An offset angle having a value within the range of ±45 degrees is contemplated to be sufficient. In other exemplary embodiments, the operator may take three or more images of the orthopedic element using radiographic imaging techniques. It is contemplated that each subsequent image after the second image may define a subsequent image frame of reference. For example, a third image may define a third frame of reference, a fourth image may define a fourth frame of reference, an nth image may define an nth frame of reference, and so on.

[0122] In other exemplary embodiments including three input images and three separate reference frames, each of the three input images may have an offset angle θ of about 60 degrees relative to one another. In some exemplary embodiments including four input images and four separate reference frames, the offset angle θ may be 45 degrees from the adjacent reference frame. In an exemplary embodiment including five input images and five separate reference frames, the offset angle θ may be about 36 degrees from the adjacent reference frame. In an exemplary embodiment including n images and n separate reference frames, the offset angle θ may be 180 / n degrees.

[0123] It is further contemplated that embodiments involving multiple images, particularly three or more images, do not necessarily have to have regular and consistent offset angles. For example, an exemplary embodiment involving four images and four separate reference frames may have a first offset angle of 85 degrees, a second offset angle of 75 degrees, a third offset angle of 93 degrees, and a fourth offset angle of 107 degrees.

[0124] The transmitter 29 then transmits the first input image 30 and the second input image 50 to the computer 1600. The computer 1600 may use the deep learning network to identify the orthopedic element 100a, a component of the endoprosthetic implant, a sub-component of a component of the endoprosthetic implant, or the endoprosthetic implant itself in any manner consistent with this disclosure.

[0125] 12 also illustrates another embodiment in which output data from the calculator 1600 is transmitted to a display 19. The display 19 can depict a modeled endoprosthetic implant 102b. The display can optionally display any of the items identified by the exemplary systems and methods described herein, including, but not limited to, an identified endoprosthetic implant, a component of the endoprosthetic implant or a subcomponent thereof, or one or more orthopedic elements. In an exemplary embodiment, it is contemplated that the identified component of the endoprosthetic implant, or a representative model of a component of the endoprosthetic implant, can be superimposed on the identified orthopedic element on which the component of the endoprosthetic implant sits (e.g., the femoral component and the proximal femur, respectively). The superimposition can be calculated and displayed using the mapped spatial data of each identified element (e.g., the component of the endoprosthetic implant and the orthopedic element on which the component of the endoprosthetic implant sits).

[0126] In this way, surgeons and others in the operating room can visualize the endoprosthetic implant components and the target orthopedic features in three dimensions in near real time and align them relative to one another.

[0127] Furthermore, since the spatial data of the identified components of the endoprosthetic implant and the spatial data of the identified orthopedic elements can be obtained from the exemplary system described herein, in an embodiment of the exemplary system, the degree of alignment can be calculated and further displayed on the display 19. For example, the calculated abduction angle α of the identified components of the endoprosthetic implant can be displayed on the display. As another example, the calculated anteversion angle υ of the identified components of the endoprosthetic implant is displayed on the display 19. As yet another example, the vertical position of the prosthetic head 113 can be displayed and superimposed (see 126) on a reconstructed 3D image of the natural femoral head of the surgical hip joint based on the preoperative planning input images 30, 50, 70. As yet another example, the display 19 may optionally display a "best fit" percentage, where a percentage that reaches or approaches 100% reflects the alignment of the identified components of the endoprosthetic implant (e.g., the femoral component 104) with respect to the reference orthopedic elements.

[0128] For example, in an embodiment where the identified component of the endoprosthetic implant is the femoral component 104, the reference orthopedic component may be the natural proximal femur 105 of the surgical joint identified and reconstructed according to any of the embodiments of the present disclosure from the pre-operative planning input image. In such an exemplary embodiment, the registration best fit percentage may take into account the anteversion angle υ of the identified femoral component 104, the varus-valgus position of the femoral stem 115 of the femoral component 104 relative to the anatomical axis of the femur 105, the anteversion angle of the femoral stem 115 in the intramedullary canal of the femur 105 of the surgical hip joint 101, and the vertical, horizontal, and anterior-posterior position of the prosthetic femoral head 113 relative to the natural femoral head (see 126) of the surgical hip joint 101 prior to resection. Any combination of the foregoing embodiments of what may be displayed on the display 19 is considered to be within the scope of the present disclosure.

[0129] In an embodiment in which the identified component of the endoprosthetic implant is the femoral component of a hip implant and the identified orthopedic component is the proximal femur into which the femoral component is inserted and seated, the exemplary system can display the varus or valgus angle of the longitudinal axis of the femoral component relative to the anatomical axis of the femur (i.e., the central axis of the femur extending through the intramedullary canal of the femur).

[0130] The exemplary system may further include one or more databases. The one or more databases may include a list of endoprosthetic implant component types and associated component size dimensions for the component types in the list of endoprosthetic implant components (e.g., different product models of a particular component). In an exemplary embodiment, the database may include a list of component sizes for one particular type of endoprosthetic implant component.

[0131] The calculator can compare the dimensions of the identified components of the endoprosthetic implant to the values ​​stored in the database. The calculator can then select or display a recommended type of component and / or a recommended size of a particular component from the values ​​stored in the database based on how closely the dimensions of the identified components of the endoprosthetic implant match the dimensions of the values ​​stored in the database. In this manner, the calculator 1600 can be said to be "configured to select" a recommended type of component of the endoprosthetic implant based on the determined size dimensions of the identified orthopedic elements in three-dimensional space. Similarly, in this manner, the calculator 1600 can be said to be "configured to recommend a size of a component of the endoprosthetic implant" based on the determined size dimensions of the identified orthopedic elements in three-dimensional space.

[0132] The display 19 may take the form of a screen. In other exemplary embodiments, the display 19 may include a glass or plastic surface worn or held by a surgeon or other people in an operating room. Such a display 19 may include a portion of an augmented reality device, such that the display shows a 3D model in addition to the wearer's field of view. In certain embodiments, such a 3D model may be superimposed on the actual surgical joint. In yet other exemplary embodiments, the 3D model may be "locked" to one or more features of the surgical orthopedic element 100, thereby maintaining the virtual position of the 3D model relative to one or more features of the surgical orthopedic element 100 regardless of the movement of the display 19. It is still further contemplated that the display 19 may include a portion of a virtual reality system in which the entire field of view is simulated.

[0133] Although x-rays from x-ray imaging systems may be desirable because x-rays are relatively inexpensive compared to CT scans and because some x-ray imaging system equipment, such as fluoroscopy systems, is generally compact enough to be used intraoperatively, nothing in this disclosure limits the use of 2D images to x-rays, and nothing in this disclosure limits the type of imaging system to x-ray imaging systems, unless expressly claimed otherwise. Other 2D images may include, for example, CT images, CT fluoroscopy images, fluoroscopy images, ultrasound images, positron emission tomography ("PET") images, and MRI images. Other imaging systems may include, for example, CT, CT fluoroscopy, fluoroscopy, ultrasound, PET, and MRI systems.

[0134] Preferably, the exemplary method may be implemented on a computer platform (e.g., computer 1600) having hardware such as one or more central processing units (CPUs), a random access memory (RAM), and / or input / output (I / O) interface(s). An example of the architecture of the exemplary computer 1600 is provided below with reference to FIG.

[0135] FIG. 13 generally illustrates a block diagram of an exemplary computing device 1600 on which one or more of the methods discussed herein may be performed according to some exemplary embodiments. In certain exemplary embodiments, computing device 1600 may operate on a single machine. In other exemplary embodiments, computing device 1600 may include connected (e.g., networked) machines. Examples of networked machines that may include exemplary computing device 1600 include, for example, cloud computing configurations, distributed host configurations, and other computer cluster configurations. In a network configuration, one or more of computing devices 1600 may operate as a client machine, a server machine, or both a server-client machine. In an exemplary embodiment, computing device 1600 may reside on a personal computer ("PC"), a mobile phone, a tablet PC, a web appliance, a personal digital assistant ("PDA"), a network router, a bridge, a switch, or any machine capable of executing instructions that specify actions to be taken by the machine or by a second machine controlled by the machine.

[0136] Exemplary machines that may include the exemplary computing device 1600 may include, by way of example, components, modules, or similar mechanisms capable of performing logical functions. Such machines may include tangible entities (e.g., hardware) capable of performing specified operations during operation. As an example, the hardware may be hardwired (e.g., specifically configured) to perform certain operations. As an example, such hardware may include a configurable execution medium (e.g., circuits, transistors, logic gates, etc.) and a computer-readable medium having instructions, which, when operated, configure the execution medium to perform certain operations. The configuration may be via a loading mechanism or under the direction of the execution medium. The execution medium selectively communicates to the computer-readable medium when the machine is operating. As an example, when the machine is in operation, the execution medium may be configured by a first set of instructions to perform a first action or set of actions at a first time, and then reconfigured by a second set of instructions to perform a second action or set of actions at a second time.

[0137] The exemplary computing device 1600 may include a hardware processor 1697 (e.g., a CPU, a graphics processing unit (“GPU”), a hardware processor core, or any combination thereof), a main memory 1696, and a static memory 1695, some or all of which may communicate with each other via an interlink (e.g., a bus) 1694. The computing device 1600 may further include a display unit 1698, an input device 1691 (preferably an alphanumeric or alphanumeric input device such as a keyboard), and a user interface (“UI”) navigation device 1699 (e.g., a mouse or stylus). In an exemplary embodiment, the input device 1691, the display unit 1698, and the UI navigation device 1699 may be touch screen displays. In an exemplary embodiment, the display unit 1698 may include holographic lenses, glasses, goggles, other eyewear, or other AR or VR display components. For example, the display unit 1698 may be worn on the user's head and provide a head-up display to the user. The input device 1691 may include a virtual keyboard (e.g., a keyboard displayed virtually in a virtual reality ("VR") or augmented reality ("AR") setting) or other virtual input interface.

[0138] The computing device 1600 may further include a storage device (e.g., a drive unit) 1692, a signal generator 1689 (e.g., a speaker), a network interface device 1688, and one or more sensors 1687, such as a global positioning system ("GPS") sensor, an accelerometer, a compass, or other sensor. The computing device 1600 may include an output controller 1684, such as a serial (e.g., universal serial bus ("USB")), parallel, or other wired or wireless (e.g., infrared ("IR"), near field communication ("NFC"), radio, etc.) connection, for communicating with or controlling one or more auxiliary devices.

[0139] Storage 1692 may include a non-transitory, machine-readable medium 1683 on which is stored one or more sets of data structures or instructions 1682 (e.g., software) embodied or utilized by any one or more of the functions or methods described herein. The instructions 1682 may reside, completely or at least partially, within main memory 1696, within static memory 1695, or within hardware processor 1697 during their execution by computer 1600. By way of example, one or any combination of hardware processor 1697, main memory 1696, static memory 1695, or storage 1692 may constitute a machine-readable medium.

[0140] Although the machine-readable medium 1683 is shown as a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a distributed or centralized database, or associated caches and servers) configured to store one or more instructions 1682.

[0141] The term "machine-readable medium" may include any medium capable of storing, encoding, or transmitting instructions for execution by the computer 1600 and causing the computer 1600 to perform any one or more of the methods of this disclosure, or capable of storing, encoding, or transmitting data structures used by or associated with such instructions. A non-limiting, exemplary list of machine-readable media may include magnetic media, optical media, solid-state memory, non-volatile memory, such as semiconductor memory devices (e.g., electronically erasable programmable read-only memory ("EEPROM"), electronically programmable read-only memory ("EPROM"), and magnetic disks, such as internal hard disks and removable disks, flash storage devices, magneto-optical disks, CD-ROM disks, and DVD-ROM disks.

[0142] The instructions 1682 may further be transmitted or received over a communications network 1681 using a transmission medium via a network interface device 1688 utilizing any one of a number of transport protocols (e.g., Internet Protocol ("IP"), User Datagram Protocol ("UDP"), Frame Relay, Transmission Control Protocol ("TCP"), Hypertext Transfer Protocol ("HTTP"), etc.). Exemplary communications networks may include a wide area network ("WAN"), a plain old telephone ("POTS") network, a local area network ("LAN"), a packet data network, a mobile telephone network, a wireless data network, and a peer-to-peer ("P2P") network. By way of example, the network interface device 1688 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communications network 1681.

[0143] By way of example, network interface device 1688 may include multiple antennas for communicating wirelessly using at least one of a single-input multiple-output ("SIMO") or multiple-input single output ("MISO") methodology. The phrase "transmission medium" includes any intangible medium capable of storing, encoding, or conveying instructions for execution by computer 1600, including analog or digital communications signals or other intangible media to facilitate communication of such software.

[0144] Exemplary methods according to the present disclosure may be at least partially machine or computer-implemented. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform the exemplary methods described herein. Exemplary implementations of such exemplary methods may include code, such as assembly language code, microcode, high-level language code, or other code. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. A computer 1600 capable of executing computer-readable instructions for performing the methods and calculations of a deep learning network may be said to be "configured to execute" the deep learning network. Furthermore, in one example, the code may be tangibly stored on or in a volatile, non-transitory, or non-volatile tangible computer-readable medium, such as during execution or other times. Examples of these tangible computer readable media may include, but are not limited to, removable optical disks (e.g., compact discs and digital video discs), hard drives, removable magnetic disks, memory cards or sticks, removable flash storage drives, magnetic cassettes, random access memories (RAM), read only memories (ROMS), and other media.

[0145] It is further contemplated that the exemplary methods disclosed herein may be used for pre-operative planning, intra-operative planning or execution, or post-operative evaluation of implant placement and function.

[0146] An exemplary method for locating an orthopedic element in space includes calibrating a radiographic imager to determine a mapping relationship between image points and corresponding spatial coordinates to define spatial data; capturing a first image of the orthopedic element using a radiographic technique, the first image defining a first frame of reference; capturing a second image of the orthopedic element using a radiographic technique, the second image defining a second frame of reference, the first frame of reference being offset from the second frame of reference at an offset angle; and detecting the orthopedic element using the spatial data using a deep learning network, the spatial data defining anatomical landmarks on or within the orthopedic element. the first image having a first mask value and the second image having a second mask value, the first value being different from the second value; applying the deep learning network to the volumetric data to generate a reconstructed three-dimensional model of the orthopedic element; and mapping the three-dimensional model of the orthopedic element to the spatial data.

[0147] In an exemplary embodiment, the exemplary method may further include performing style transfer on the first image and the second image using a deep learning network. In an exemplary embodiment, the style transfer converts spatial data from a radiographic technique into dynamic digital radiography data.

[0148] In an exemplary embodiment, the first value is a positive value.

[0149] In an exemplary embodiment, the second value is a negative value.

[0150] In an exemplary embodiment, the exemplary method further includes projecting the reconstructed three-dimensional model onto a display.

[0151] In an exemplary embodiment, the deep learning network includes a convolutional neural network.

[0152] In an exemplary embodiment, the radiological imaging technique is fluoroscopy.

[0153] In an exemplary embodiment, the method is performed intraoperatively.

[0154] In an exemplary embodiment, the orthopedic element is the acetabulum of the pelvis.

[0155] In an exemplary embodiment, the exemplary method further includes calculating a center of the acetabulum.

[0156] In an exemplary embodiment, the exemplary method further includes aligning a longitudinal axis of rotation of a neck of a femoral stem (eg, a component of an endoprosthetic implant) with a center of the acetabulum.

[0157] In an exemplary embodiment, the exemplary method further includes aligning an acetabular shell (eg, a component of an endoprosthetic implant) with the patient's reamed acetabulum.

[0158] In an exemplary embodiment, the exemplary method further includes aligning a femoral stem (eg, a component of an endoprosthetic implant) with the intramedullary canal of the patient's reamed proximal femur.

[0159] In an exemplary embodiment, the method further includes aligning a longitudinal axis of a femoral stem (eg, a component of an endoprosthetic implant) with the anatomical (ie, central) axis of the femur.

[0160] An exemplary method for locating an orthopedic element in space includes calibrating a radiographic imager to determine a mapping relationship between image points and corresponding spatial coordinates to define spatial data; capturing a first image of the orthopedic element using a radiographic technique, the first image defining a first frame of reference; capturing a second image of the orthopedic element using a radiographic technique, the second image defining a second frame of reference, the first frame of reference being offset from the second frame of reference at an offset angle; and detecting the orthopedic element using the spatial data using a neural network, the spatial data defining anatomical landmarks on or within the orthopedic element. applying a mask to the orthopedic element defined by the anatomical landmarks using a deep learning network; projecting spatial data from a first image of the desired orthopedic element and spatial data from a second image of the desired orthopedic element to define volumetric data, wherein spatial data including image points located within the mask region of either the first image or the second image has positive values, and spatial data including image points located outside the mask region of either the first image or the second image has negative values; applying the deep learning network to the volumetric data to generate a three-dimensional model of the orthopedic element; and mapping the three-dimensional model of the orthopedic element to the spatial data, wherein the orthopedic element is an acetabulum of a pelvis.

[0161] In an exemplary embodiment, the orthopedic element is resected or reamed during the surgical procedure.

[0162] In an example embodiment, the method further includes performing style transfer on the first image and the second image using a deep learning network.

[0163] In an exemplary embodiment, the style transformation converts spatial data from a radiographic imaging technique into dynamic digital radiography data.

[0164] In an exemplary embodiment, the first value is a positive value.

[0165] In an exemplary embodiment, the second value is a negative value.

[0166] In an exemplary embodiment, the method further includes projecting the reconstructed three-dimensional model onto a display.

[0167] In an exemplary embodiment, the deep learning network includes a convolutional neural network.

[0168] In an exemplary embodiment, the radiological imaging technique is fluoroscopy.

[0169] In an exemplary embodiment, the method is performed intraoperatively.

[0170] In an exemplary embodiment, the method further includes calculating a center of the acetabulum.

[0171] In an exemplary embodiment, the method further includes rotationally aligning a longitudinal axis of the neck of the femoral stem with a center of the prosthetic femoral head and aligning the longitudinal axis of the femoral stem with an anatomical axis of the intramedullary canal of the femur in which the femoral stem is to be placed.

[0172] In an exemplary embodiment, the exemplary method further includes aligning an acetabular shell (eg, a component of an endoprosthetic implant) with the patient's reamed acetabulum.

[0173] In an exemplary embodiment, the exemplary method further includes aligning a femoral stem (eg, a component of an endoprosthetic implant) with the intramedullary canal of the patient's reamed proximal femur.

[0174] In an exemplary embodiment, the method further includes aligning a longitudinal axis of a femoral stem (eg, a component of an endoprosthetic implant) with an anatomical axis of the femur in which the femoral stem is to be placed.

[0175] An exemplary system for identifying positions of an orthopedic element and a component of an endoprosthetic implant in space includes a tissue transmission imager; a first input image taken by the tissue transmission imager from a first frame of reference, the first image depicting a calibration fixture; a second input image taken by the tissue transmission imager from a second frame of reference that is offset from the first frame of reference, the second input image showing the calibration fixture; and a computer configured to execute a deep learning network, the deep learning network configured to identify the orthopedic element and the component of the endoprosthetic implant to define identified orthopedic elements and identified components of the endoprosthetic implant, and to map the identified orthopedic elements and identified components of the endoprosthetic implant to spatial data identified by the first input image and the second input image, thereby determining positions of the identified orthopedic elements and identified components of the endoprosthetic implant in three-dimensional space.

[0176] In an exemplary embodiment, the system further includes a third input image, the third input image being taken by the tissue transmission imager from a third frame of reference, the third image depicting the calibration fixture.

[0177] In an exemplary embodiment of the system, the deep learning network is further configured to identify the plurality of orthopedic elements and the plurality of components of the endoprosthetic implant to define the plurality of identified orthopedic elements and the plurality of identified components of the endoprosthetic implant.

[0178] In still a further exemplary embodiment of the system, a first identified component of the plurality of identified components of the endoprosthetic implant is an acetabular component of the hip joint endoprosthetic implant, and a second identified component of the plurality of identified components of the endoprosthetic implant is a femoral component of the hip joint endoprosthetic implant.

[0179] In an exemplary embodiment of the system, the identified component of the endoprosthetic implant is an acetabular shell and the identified orthopedic element is a reamed acetabulum adjacent to the acetabular shell.

[0180] In an exemplary embodiment of the system, the identified component of the endoprosthetic implant is a femoral stem and the identified orthopedic feature is the intramedullary canal of the femur adjacent to the femoral stem.

[0181] In an exemplary embodiment of the system, the identified orthopedic elements are modeled in three dimensions to define a modeled orthopedic element.

[0182] In an exemplary embodiment, the modeled orthopedic element is displayed on a display.

[0183] In an exemplary embodiment, the identified components of the endoprosthetic implant are modeled in three dimensions to define modeled components of the endoprosthetic implant.

[0184] In an exemplary embodiment, the modeled components of the endoprosthetic implant are displayed on a display.

[0185] In an exemplary embodiment, the calculated abduction angles of the identified components of the endoprosthetic implant are displayed on a display.

[0186] In an exemplary embodiment, the calculated anteversion angles of the identified components of the endoprosthetic implant are displayed on a display.

[0187] An exemplary system for recommending a type of endoprosthetic implant component to be surgically implanted in a patient, the system comprising: a tissue transmission imager; a first input image, the first input image taken by the tissue transmission imager from a first frame of reference, the first image depicting a calibration fixture; a second input image, the second input image taken by the tissue transmission imager from a second frame of reference, the second frame of reference being offset from the first frame of reference, the second image depicting the calibration fixture; and a computer configured to execute a deep learning network, the deep learning network identifying an orthopedic component. Separately, the system includes a calculator configured to define an identified orthopedic element and map the identified orthopedic element to spatial data identified by the first input image and the second input image, thereby defining a determined size dimension of the identified orthopedic element in three-dimensional space; a list of endoprosthetic implant component types and associated component size dimensions for the component types in the list of endoprosthetic implant components, wherein the calculator is further configured to select a recommended type of endoprosthetic implant component based on the determined size dimension of the identified orthopedic element in three-dimensional space.

[0188] In an exemplary embodiment, the system further includes a third input image, the third input image being taken by the tissue transmission imager from a third frame of reference, the third image depicting the calibration fixture.

[0189] In an exemplary embodiment, the identified orthopedic feature is the internal geometry of the bone before or after reaming, or before or after broaching.

[0190] In an exemplary embodiment, the calculator is configured to execute a best fit algorithm to select recommended components of the endoprosthetic implant based on the determined size dimensions of the identified orthopedic elements.

[0191] 1. An exemplary system for determining a size of an endoprosthetic implant component to be surgically implanted in a patient, the system comprising: a tissue transmission imager; a first input image, the first input image taken by the tissue transmission imager from a first frame of reference, the first image depicting a calibration fixture; a second input image, the second input image taken by the tissue transmission imager from a second frame of reference, the second frame of reference being offset from the first frame of reference, the second image depicting the calibration fixture; and a computing device configured to execute a deep learning network, the deep learning network the system includes a calculator configured to identify an orthopedic element, define an identified orthopedic element, and map the identified orthopedic element to spatial data identified by the first input image and the second input image, thereby defining a determined size dimension of the identified orthopedic element in three-dimensional space; and a list of components of the endoprosthetic implant and associated component size dimensions for the components in the list of components of the endoprosthetic implant, the calculator being further configured to recommend a size of the component of the endoprosthetic implant based on the determined size dimension of the identified orthopedic element in three-dimensional space.

[0192] In an exemplary embodiment, the system further includes a third input image, the third input image being taken by the tissue transmission imager from a third frame of reference, the third image depicting the calibration fixture.

[0193] In an exemplary embodiment, the identified orthopedic feature is the internal geometry of the bone before or after reaming, or before or after broaching.

[0194] In an exemplary embodiment, the calculator is configured to execute a best fit algorithm to select recommended components of the endoprosthetic implant based on the determined size dimensions of the identified orthopedic elements.

[0195] It is to be understood that the invention is not limited to the particular configurations and method steps disclosed herein or illustrated in the drawings, but includes any modifications or equivalents known in the art that are within the scope of the claims. Those skilled in the art will appreciate that the apparatus and methods disclosed herein will find utility.

Claims

1. A system for verifying the position of orthopedic elements and components of prosthetic joint implants in space, comprising: a tissue transmission imaging device; a first two-dimensional input image taken by the tissue transmission imager from a first frame of reference, the first input image depicting the calibration fixture; a second two-dimensional input image taken by the tissue transmission imager from a second frame of reference displaced from the first frame of reference at a first displacement angle, the second input image depicting the calibration fixture; the third two-dimensional input image, captured by the tissue transmission imager from a third frame of reference displaced from the first frame of reference and the second frame of reference at a second displacement angle, depicting the calibration fixture; and a computer configured to project spatial data from the first two-dimensional input image, the second two-dimensional input image, and the third two-dimensional input image at the first misalignment angle and the second misalignment angle to define volumetric data, and further configured to execute a deep learning network, wherein the deep learning network is configured to identify an orthopedic element and a component of a prosthetic implant from the volumetric data to define an identified orthopedic element and an identified component of the prosthetic implant, map the identified orthopedic element and the identified component of the prosthetic implant within the spatial data confirmed by the first input image, the second input image, and the third input image, thereby determining positions of the identified orthopedic element and the identified component of the prosthetic implant in three-dimensional space, and further to calculate an abduction angle of the identified component of the prosthetic implant or an anteversion angle of the identified component of the prosthetic implant; Equipped with the first two-dimensional input image, the second two-dimensional input image, and the third two-dimensional input image include the spatial data; system.

2. The system described in claim 1, wherein the deep learning network is further configured to identify a plurality of orthopedic elements and a plurality of components of the artificial joint implant to define a plurality of identified orthopedic elements and a plurality of identified components of the artificial joint implant.

3. The system described in claim 2, wherein a first identified component of the plurality of identified components of the artificial joint implant is an acetabular component of a hip joint implant, and a second identified component of the plurality of identified components of the artificial joint implant is a femoral component of a hip joint implant.

4. The system described in claim 1, wherein the identified component of the artificial joint implant is an acetabular shell and the identified orthopedic element is a reamed acetabulum adjacent to the acetabular shell.

5. The system described in claim 1, wherein the identified component of the artificial joint implant is a femoral stem and the identified orthopedic element is an intramedullary canal of the femur adjacent to the femoral stem.

6. The system described in claim 1, wherein the identified orthopedic elements are modeled in three dimensions to define modeled orthopedic elements.

7. The system described in claim 6, wherein the modeled orthopedic element is displayed on a display.

8. The system described in claim 1, wherein the identified components of the artificial joint implant are modeled in three dimensions to define modeled components of the artificial joint implant.

9. The system described in claim 8, wherein the modeled components of the artificial joint implant are displayed on a display.

10. The system described in claim 1, wherein the calculated abduction angle of the identified component of the artificial joint implant is displayed on a display.

11. The system described in claim 1, wherein the calculated forward tilt angle of the identified component of the artificial joint implant is displayed on a display.

Images