Near real-time continuous 3D positioning of objects in 2D X-ray images

Abstract

A system and method are provided for providing continuous automatic alignment of bone(-pieces), instruments, and implants based on only one additional X-ray image without the need to change the position of the X-ray source. According to the invention, a 2D X-ray image is received, which shows the target surgical site. Artificial intelligence-assisted automatic detection and localization of objects such as bone(-pieces), tools (such as drills), and / or implants, combined with prior information about the surgical procedure, is used to calculate their relative 3D positions and orientations at any desired time. This alignment of objects can be repeated multiple times, thus providing guidance in quasi-real time.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer-assisted surgery. Further, the present invention relates to a system and method for providing information related to an object based on an X-ray image. In particular, the present invention relates to a system and method for automatically aligning potentially moving objects in near real-time, i.e., for determining relative 3D positions and orientations. The method can be implemented as a computer program executable on a processing device of the system.

Background Art

[0002] In cases where a long bone is fractured, the bone fragments can be fixed by an implant such as an intramedullary nail, which can be inserted into the medullary canal of the bone or within the bone plate and can be fixed to the surface of the bone as support for healing the fracture. The surgical procedure for implanting such an implant can be minimally invasive and may require repeatedly acquiring X-ray images to enable the surgeon to correctly place the implant. The implant can also be coupled to one or more sub-implants, such as screws or blades.

[0003] There are various important and difficult steps in the intramedullary nailing procedure of long bones, including proper reduction of the fracture (ensuring correct positioning of the bone fragments), determination of the entry point for inserting the implant into the bone, and fixation of the implant by inserting screws into the holes of the implant.

[0004] An important step in implanting a nail into a long bone is the determination of the entry point. A sub-optimal choice for the entry point can result in non-optimal positioning of the nail and thus can also result in improper positioning of the coupled sub-implants, such as neck screws or blades. Moreover, for a given entry point, if the surgeon has already performed reaming, the canal in which the nail is to be positioned is defined and there may be no possibility of correction.

[0005] There are two main methods for determining the entry point: by palpation or based on X-ray images. When palpation is performed, after making an initial cut, the surgeon feels the location of the entry point (for example, in cases where a cephalomedullary nail is implanted in the femur, this is the tip of the greater trochanter) with their fingers and determines the location of the entry point based on the estimated bone surface and rules of thumb (e.g., the so-called 1 / 3-2 / 3 rule). The drawback of this procedure is that the bone surface can only be determined incompletely by touch, which can result in a substantial deviation from the optimal entry point. Moreover, rules of thumb (e.g., the 1 / 3-2 / 3 rule) may be considerably suboptimal depending on the patient's specific anatomical structure.

[0006] The entry point can also be determined based on X-ray images. In cases where a cephalic nail is implanted in the femur, an anteroposterior (AP) X-ray image may be taken first, in which the mouth opener is placed on the tip of the trochanter. A lateral X-ray image is then taken so that the femoral shaft and neck are parallel. The tip of the mouth opener is moved dorsally or ventrally until it is positioned midway between the two axes (which is checked by the X-ray image). The drawbacks of this procedure are, firstly, that it is difficult to obtain a lateral X-ray image from the correct direction, and secondly, that determining the two axes based on the X-ray images may only be done inaccurately.

[0007] Determining the entry point for inserting a nail into the humerus in a minimally invasive manner is all the more difficult because the correct reduction of fractures near the neck and head is typically performed while the implant is being inserted. The correct entry point is approximately on the nearest point of contact of the anatomical neck (the anatomical neck of the humerus), or at a defined distance medially from this point. When any fracture is correctly reduced, the entry point is visible in the anterior-posterior (AP) radiographic image, because in such images the nearest point of contact of the joint contour can be identified. However, ensuring the correct AP imaging direction is difficult, and may even be impossible, depending on the patient's physical setup and the radiographic equipment in the operating room (OR). Moreover, even in the perfect AP imaging direction, there is substantial uncertainty in determining the location of the entry point with respect to the depth of imaging. Since the entry point may be indistinguishable in such radiographic images, taking images from different line-of-sight directions (e.g., axial) may not solve this problem.

[0008] Even a combination of two methods for determining the entry point (palpation and X-ray imaging) may not adequately improve accuracy. To date, there are no established computer-assisted surgical (CAS) techniques to address this problem.

[0009] Another challenge in any osteosynthesis is that properly reduced fractures is crucial for a satisfactory clinical outcome. Typically, fractures heal satisfactorily only if the reduction is performed correctly. Reduction of long bones, in particular, is often difficult to assess during minimally invasive surgery, especially with regard to the correct anteversion angle (for the femur) or torsion angle (for the humerus or tibia). Inaccurate anteversion or torsion angles are often only noticed after the surgery is complete. At this stage, even if the fracture itself has healed, an inaccurate anteversion or torsion angle causes significant discomfort and limitations for the patient. Therefore, properly correct anteversion or torsion angles are crucial for a satisfactory clinical outcome, especially for osteosynthesis of the femur, tibia, or humerus. Similar comments apply to the head-neck-diaphysis (CCD) angle and leg length, which are also important for a satisfactory clinical outcome.

[0010] Abnormal rotation of bone fragments is one of the most common reasons for revision surgery when treating tibial and femoral fractures. A non-pathological AV angle (AV angle) relative to the femoral neck is typically between 10 and 20 degrees. Abnormal rotation up to 10 degrees relative to the optimal angle (e.g., the other, healthy leg) may be compensated for by the patient, but larger abnormal rotations can cause discomfort and problems during walking. Determining the AV angle during surgery is difficult and is often done inaccurately or not at all. Studies have shown that 10% to 25% of osteosynthesis procedures on the leg result in deviations of more than 10 degrees from the ideal value. A reliable intraoperative procedure for determining the AV angle is therefore crucial.

[0011] The difficulty in determining the anteversion or torsion angle lies in the fact that long bones are too long to fit within a single X-ray image. Furthermore, the geometry necessary to determine the anteversion or torsion angle is located at the proximal and distal parts of the bone, for example, relative to the femur, cervical axis, and condyle. Therefore, the geometry shown in separate proximal and distal X-ray images must be related to one another.

[0012] Conventional techniques propose different approaches to determining the anteversion angle. In the case of femoral and head nails, one approach is to subjectively determine whether the screw (or head element) should make an angle of approximately 10 degrees with the operating room floor, by manually determining whether the knee is pointing upward towards the operating room ceiling, and whether the nail axis should intersect with the center of the femoral head. The CAS approach, proposed by Blau et al. (US 2015 / 0265361 A1 and WO 2019 / 077388 A1), uses two reference bodies with metal labels, one in the distal region of the femur and one in the proximal region, as well as two proximal and one distal X-ray images, all showing their respective reference bodies.

[0013] Another challenging step in intramedullary nailing is fixation itself. The main difficulty with fixation using long nails is the bending and twisting of the nail, as it conforms to the bone's medullary canal to some extent. This prevents the simple, static mechanical fixation procedure employed in cases of short nails. Freehand fixation is difficult, time-consuming, and may require the acquisition of many X-ray images. For this reason, some manufacturers offer flexible mechanical solutions (referred to here as "long aiming devices") that adapt to nail bending. While long aiming devices simplify the procedure, their application is still not easy, as the X-ray images showing the long aiming device must be correctly interpreted and the C-arm position adjusted accordingly. Only after the correct adjustment of the C-arm can the long aiming device be properly adjusted.

[0014] Typical, unassisted, freehand fixation procedures can have a high mis-drill rate. This is especially true when the distance to be drilled is long, for example, when fixing an antegrade femoral nail close to the condyle. The conventional approach consists of the surgeon estimating the correct position of the drill tip and the appropriate drill angle. Mis-drills may not always be a problem and can sometimes be corrected simply by drilling a new hole. However, it may be prudent to choose a new starting position for drilling to avoid the new hole ultimately being the same drill tube as the previous one. Furthermore, in some cases (fixation holes with internal threads, or implants with combined plates and nails), this may be almost impossible. In these cases, it is crucial that the drilling is correct on the first attempt, which requires not only an accurate determination of the initial drill tip position but also maintaining this drill angle throughout the drilling process.

[0015] In such cases, and especially in scenarios where drilling is performed in close proximity to critical structures (such as when installing sacroiliac joint or pedicle screws), support in the form of continuous verification of the drilling angle and trajectory during drilling would be desirable. In conventional procedures, this would require iterative adjustment of the C-arm and acquisition of X-ray images from different line-of-sight directions to estimate the 3D position and orientation of the drill relative to the target object (e.g., a nail).

[0016] The object of this invention is to address these challenges in dynamically changing situations (e.g., drilling holes in bone, reducing bone fragments, inserting implants into bone). Existing technologies such as computer-assisted surgical systems and / or surgical robots typically require time-consuming, often invasive, pre-positioning procedures. Since 2002, non-invasive procedures have existed for aligning anatomical structures to reference bodies / trackers based on X-rays during surgery (fluorescence-CT matching by Brainlab AG) as a basis for real-time tracking of motion, which are further combined with tracking systems that have attached trackers / reference bodies. [Overview of the Initiative] [Problems that the invention aims to solve]

[0017] This invention proposes a system and method for aligning multiple objects that can move relative to one another at a desired time point, without requiring a reference body or tracker. An objective of this invention may be to provide, in near real-time, perhaps instantaneously, the determination of such alignment, i.e., relative 3D position and orientation, of objects based solely on prior information extracted from current and previous X-ray images, thereby enabling the movement of those objects relative to each other between the acquisition of current and previous X-ray images. Determining a specific point or curve of an object on or within an object, perhaps relative to another object, may also be an objective of this invention. Providing relative 3D orientation and 3D position between multiple objects at least partially shown in an X-ray image may also be an objective of this invention.

[0018] At least one or more of the objectives mentioned are addressed by the apparatus described in claim 1 and the method described in claim 13. Further embodiments according to the present invention are described in their respective dependent claims. [Means for solving the problem]

[0019] Generally, the apparatus includes a processing unit configured to process X-ray images. Software programs are provided for execution by this processing unit. The following steps can be performed by the processing unit of this apparatus using the computer software program.

[0020] First, a first X-ray image is received, which is a projection image of an object that is at least partially visible. Next, a model of that object is received, and this model is applied to determine the position of the object in the first X-ray image, so that a point is identified in the first X-ray image and its 3D position relative to the object is known. In other words, the position of the point is known in the patient in 3D space, and that point is visible in the 2D projection image.

[0021] Similar to the first X-ray image, a second X-ray image is received, which is a projection image of the tool, at least partially visible, along with the object, at least partially visible. A model of the object is again applied to pinpoint the object's location in the second X-ray image. With respect to the tool, a coordinate system is defined, and axes are determined and identified within that coordinate system.

[0022] Finally, the 3D position and orientation of the tool relative to the object are determined based on the second X-ray image, the 3D model of the tool, the object whose position is identified, the knowledge that the 3D position of a point relative to that object is the same when the first and second X-ray images are generated, and the knowledge of the distance between that point and its axis.

[0023] In this specification, “object” can be any object that is at least partially visible in an X-ray image, such as an anatomical structure or an implant. If “object” is considered to be an implant, it will be understood that the implant is already placed within an anatomical structure. “Tool” can also be something that is at least partially visible in an X-ray image, such as a drill, wire, screw, or similar. In a more specific example, if “object” is bone, “tool” could be an implant, such as a bone pin, that is intended to be inserted into the bone but has not yet been inserted. It can be said that “tool” is the object to be inserted, and “object” is an anatomical structure or an object such as an implant that is already placed within an anatomical structure. It should be noted again that the present invention does not require the use of any reference body or tracker.

[0024] The term "3D representation" may refer to a complete or partial description of a 3D volume or 3D surface, which may also refer to selected geometric aspects such as radii, curves, planes, angles, or similar. While the present invention may enable the determination of complete 3D information relating to the 3D surface or volume of an object, methods for determining only selected geometric aspects are also considered in the present invention.

[0025] Since X-ray imaging is a 2D imaging diagnostic method, it is generally not possible to uniquely determine the 3D orientation (i.e., 3D position and 3D orientation) of individual objects shown in an X-ray image, nor is it generally possible to uniquely determine the relative 3D positions and 3D orientations between objects shown in an X-ray image.

[0026] The X-ray beam originates from an X-ray source (focal point) and is detected by an X-ray detector within the image plane. Thus, the physical dimensions of an object are related to the dimensions of its projection within the X-ray image through the slice theorem. There is generally ambiguity in determining the depth of imaging, which is the distance from the image plane and is later also referred to as the "z coordinate". Throughout the present invention, the terms "localize" and "localization" mean the determination of the 3D orientation of an object with respect to a selected coordinate system and the determination of the 2D spatial position of the projection of that object onto the image plane, but not the determination of the z coordinate.

[0027] If a 3D model of an object shown in an X-ray image is available, this may enable the localization of the object. Under the condition that the object is large enough and has sufficient structure, it may even be possible to roughly determine (or estimate) the z coordinate of the object. However, even if a deterministic 3D model of a known object shown in an X-ray image is available, there are cases where neither localization nor determination of the z coordinate is possible. As an example, this particularly applies to thin objects such as a drill or a k-wire. Without knowing the depth of imaging of the tip of the drill, there are multiple 3D poses of the drill that result in the same or substantially the same projection in a 2D X-ray image. Thus, it may generally not be possible to determine the relative 3D position and 3D orientation of the drill, for example, with respect to an implant also shown in the X-ray image. On the other hand, if the depth of imaging of such an object can be determined through other means or is known in advance, this may enable the determination of the 3D position and 3D orientation of that object.

[0028] It is an object of the present invention to enable the localization of an object whose geometry may not be locatable without further information regarding its depth of imaging, and to determine the 3D position and 3D orientation of such an object with respect to another object.

[0029] For example, the object may include an implant with a hole. In that case, the 3D position of a point with respect to the object can be determined according to an embodiment of the present invention based on the axis of the hole in the implant. It should be noted that the implant can be an intramedullary nail having a transverse axis extending through a hole for fixing the bone structure with a nail. Such a hole may have a thread. The axis of the hole cuts the outer surface of the bone to define an entry point for the fixing screw. In another example, a combination of a nail that can be installed inside a long bone and a plate that can be installed outside the long bone can be fixed together by at least one screw extending through both the hole in the plate and the hole in the nail. Also here, the entry point for the screw can be defined by an axis extending through those holes.

[0030] In a further example, the object can be considered as a nail already implanted in the bone, and the X-ray image also shows at least a part of a tool such as a drill. In this case, the tool is at least partially visible in the first X-ray image, and the identified point is a point on the tool, for example, the tip of the tool. Based on the second X-ray image, the 3D position and orientation of the tool with respect to the object can be determined, but the tool has been moved with respect to the object between the generation of the first X-ray image and the generation of the second X-ray image. Determining the 3D position and orientation of the drill with respect to the implant in the bone when shown in the second X-ray image can assist in evaluating whether the drilling is in the direction aiming at the hole in the implant that should extend when, for example, a screw is later implanted along the hole drilled by the drill.

[0031] The 3D position of the point identified in the X-ray image can be determined in different ways. On the one hand, the 3D position of a point with respect to the object can be determined based on knowledge of the position of the bone surface and the knowledge that the point is located on the bone surface. For example, the tip of the drill can be positioned on the outer surface of the bone when the first X-ray image is generated. That point can still be the same when the second X-ray image is generated, even if the drill drills a hole in the bone. Therefore, the point in both X-ray images can be the entry point, even if it is defined only by the tip of the drill in the first X-ray image.

[0032] On the other hand, the 3D position of a point relative to an object can be determined based on further X-ray images from a different line of sight. For example, a C-arm-based X-ray system can be rotated before generating further X-ray images.

[0033] Furthermore, the 3D position of a point relative to an object can be determined based on the determination of the 3D position and orientation of the tool relative to the object, based on the first X-ray image. That is, if the 3D position and orientation at the time of generating the first X-ray image are already known, that knowledge can be used later, and after the movement of the tool relative to the object, to determine the 3D position and orientation. In fact, this procedure can be repeated many times in a series of X-ray images.

[0034] If the tip of a tool is visible in the X-ray image, the determination of the tool's 3D position and orientation relative to the object may be based further on the tool's tip defining additional points. It should be noted that these additional points may simply be points in the projected image, i.e., 2D points. However, along with the known 3D position of a point, e.g., an entry point, the progression of motion between X-ray images may be determined by taking these additional points into consideration.

[0035] For several reasons, determining the 3D position and orientation of a tool relative to an object can be more difficult. For example, at least a portion of the tool, where that part is visible in the X-ray image, may be rotationally symmetric, such as a drill that rotates during the generation of the X-ray image. According to one embodiment of the present invention, the 3D position and orientation of a tool relative to an object can nevertheless be determined with at least sufficient accuracy. For example, when considering a thin and long tool such as a drill or K-wire, or a thin and long implant, a single projection may not show enough detail to allow for the distinction of the tool's orientation in 3D space, and those orientations may be similar or identical projections. However, when comparing two or more projection images, there is a likelihood for a particular orientation that can be assumed as a result. Furthermore, additional aspects, such as the visible tool tip, can be taken into consideration.

[0036] In another example, when generating an X-ray image showing an object along with a tool, the tool may be partially shielded. This could occur if the tip of the tool is shielded by an implant, or if the handle of a drill is largely shielded by a tube that prevents damage to surrounding soft tissue while drilling into bone. In these cases, a third X-ray image may be received, generated from a different line of sight than the previous image. Such a third X-ray image may provide additional information beyond what can be obtained from the image generated in the primary line of sight. For example, the tip of the tool may be visible in the third X-ray image. The 3D position of the tip is not visible in the second X-ray image, but the axis of the tool is visible in the second X-ray image, thus it can be determined by defining a plane in the direction of the X-ray imaging device's focal point when generating the second X-ray image, and consequently, the tip of the tool must lie on that plane. Furthermore, the tip of the tool can be considered to define a line in the direction of the X-ray imaging device's focal point when generating the third X-ray image. A line defined by a tip, that is, defined by a visible point in the third X-ray image, cuts a plane in 3D space defined based on the second X-ray image. It will be understood that the second and third X-ray images are aligned, for example, by identifying the positions of objects in both images.

[0037] Based on the processed X-ray images, the device may be configured to provide instructions to the user. Specifically, the device may be configured to compare the determined 3D position and orientation of the tool relative to the object with the expected or intended 3D position and orientation. The automatically generated instructions may provide guidance to the user. Monitoring during drilling is possible, as well as proper orientation at the start of drilling. For example, during drilling, the device may evaluate whether the drilling direction will eventually reach the target structure and, if necessary, provide instructions for correcting the drilling direction. When providing instructions, the device may take into account at least one of the depth of drilling already performed, the density of the object, the diameter of the drill, and the stiffness of the drill. It will be understood that the inclination of the drill during drilling may result in drill bending or shifting of the drill axis, depending on the properties of the surrounding material, e.g., bone. These aspects, which can be anticipated to some extent, may be taken into account by the device when providing instructions.

[0038] One possible solution proposed by EP19217245 is to utilize prior information regarding imaging depth. For example, from previous X-ray images acquired from different imaging directions (which describe the direction in which the X-ray beam passes through the object), it may be determined that the tip of the k-wire is on the trochanter, thereby limiting the imaging depth of the k-wire tip relative to another object. This may be sufficient to resolve ambiguity regarding the 3D position and 3D orientation of the k-wire relative to another object in the current imaging direction.

[0039] 3D alignment of two or more X-ray images Another possible solution is to utilize two or more X-ray images acquired from different imaging directions and to align these images. The more different the imaging directions (e.g., AP and ML images), the more useful the additional images may be in determining 3D information. Image alignment can be performed based on uniquely identifiable objects shown in the images whose 3D models are known and which must not move between images. As mentioned above, the most common approach in the art is to use a reference body or tracker. However, not using any reference body is generally preferred to simplify both product development and the use of the system. If the movement of the C-arm is precisely known (e.g., if the C-arm is electronically controlled), image alignment may be possible based solely on these known C-arm movements.

[0040] However, the movement of the C-arm is typically not precisely known. As described in LU101009B1, rigid objects of known geometric shapes, such as implants, may allow for the determination of at least the imaging direction, even if the implant may only be able to be located and not determine the imaging depth.

[0041] However, there are many scenarios in which no rigid object is present in the X-ray image. For example, when determining an entry point for implanting a nail, there is no implant in the X-ray image. The present invention teaches a system and method for enabling 3D registration of multiple X-ray images in the absence of a single rigid object of known geometry that would enable unique and sufficiently accurate 3D registration. The approach proposed herein involves using a combination of features of two or more objects or at least two or more parts of one object that, individually, would not enable unique and sufficiently accurate 3D registration, but together enable such registration, and / or limiting the C-arm movement allowed between image acquisitions (e.g., only rotation around a specific axis of the X-ray imaging device, such as the C-arm axis, or translation along a specific axis may be permitted). The object used for registration may be an artifact of known geometry (e.g., a drill or k-wire) or it may be a part of an anatomical structure. Objects or parts of objects may also be approximated using simple geometric models (for example, the femoral head may be approximated by a ball), or only their specific features may be used (which may be a single point, e.g., the tip of a k-wire or drill). The object features used for alignment must not move between image acquisitions, and if such features are single points, only the point must not move. For example, if a k-wire tip is used, its tip must not move between images, while the inclination of the k-wire may change between images.

[0042] According to one embodiment, each of the X-ray images showing at least a portion of the object can be aligned. A first X-ray image may be generated in a first imaging direction and at a first position of the X-ray source relative to the object. A second image may be generated in a second imaging direction and at a second position of the X-ray source relative to the object. Such two X-ray images can be aligned based on a model of the object, together with at least one of the following conditions: - A point with a fixed 3D position relative to the object is definable and / or detectable in both X-ray images, i.e., identifiable and / or locatable in both X-ray images. Note that a single point may suffice. Further note that this point may have a known distance to the object's structure, such as its surface. - Two identifiable points with fixed 3D positions relative to the object are present in both X-ray images. - A portion of a further object with a fixed 3D position is visible in both X-ray images. In such cases, a model of the further object may be used when aligning the X-ray images. Even a single point is intended to be considered as part of the further object. - While acquiring the first and second X-ray images, the only movement of the X-ray source relative to the object is translation. - During the generation of the first and second X-ray images, the only rotation of the X-ray source is around an axis perpendicular to the imaging direction. For example, the X-ray source can rotate around the C-axis of a C-arm-based X-ray imaging apparatus.

[0043] It will be understood that the alignment of X-ray images based on an object model can be more accurate if two or more of the aforementioned conditions are met.

[0044] According to one embodiment, a point having a fixed 3D position relative to an object can be a point on a further object, and as long as that point is fixed, it allows for the movement of further objects. It will be understood that a fixed 3D position relative to an object can be on the surface of that object, i.e., a point of contact, but can also be a point at a defined distance (greater than zero) from the object. This distance can be from the surface of the object (allowing for a position outside or inside the object) or from a specific point on the object (for example, if the object is a ball, the center of the ball).

[0045] According to one embodiment, a further object having a fixed 3D position relative to an object may be in contact with the object or at a defined distance from it. Note that the orientation of the further object relative to the object may be fixed or variable, and the orientation of the further object may change due to rotation and / or translation of the further object relative to the object.

[0046] It will be understood that X-ray image alignment can be performed with three or more objects.

[0047] According to various embodiments, the following is an example that enables image alignment (without a reference body): 1. Approximation of the femoral head or artificial femoral head (as part of a hip joint implant) with a ball (object 1) and use of a k-wire or drill tip (object 2), while also limiting the allowable C-arm movement between images. 2. Approximation using a cylindrical diaphysis or vertebral body (object 1) and the use of a k-wire or drill tip (object 2). The permissible C-arm movement may or may not be limited between images. 3. When using a ball (object 1) for the femoral head or artificial femoral head (as part of a hip joint implant) and a cylinder (object 2) for the femoral shaft, the C-arm movement allowed between images does not need to be restricted. 4. Use of a guide rod fixed within the bone (the guide rod has a stopper to prevent it from being inserted too far) or a k-wire, during which the allowable C-arm movement between images is also restricted. In this case, only one object is used, and this method is embodied by the restricted C-arm movement between images. 5. Use of an approximation with a guide rod or k-wire (object 1) fixed within the bone and a ball of the femoral head (object 2).

[0048] It should be noted that this method can also be used to improve alignment accuracy or to verify other results. Specifically, in the case of image alignment using multiple objects or at least multiple parts of one object, one or more of which even enable 3D alignment on their own, and which also restricts the possible C-arm movement, this overlap determination may improve alignment accuracy compared to not using the proposed method. Alternatively, images may be aligned based on a subset of available objects or features. Such alignment may be used to verify the detection of the remaining objects or features (not used for alignment), or it may allow for the detection of movement between images (e.g., whether the tip of a mouthpiece is moving).

[0049] A further embodiment of this approach may involve aligning two or more X-ray images showing different (but possibly overlapping) parts of an object (e.g., one X-ray image showing the proximal part of a femur and another X-ray image showing the distal part of the same femur) by fitting a single model together to all available X-ray projections, while restricting the C-arm movement allowed between X-ray images (e.g., only translation is allowed). The fitted model may be a complete or partial 3D model (e.g., a statistical shape or appearance model), or it may be a reduced model that describes only a certain geometric aspect of the object (e.g., the position of an axis, plane, or selected point).

[0050] As will be explained in detail below, the 3D reconstruction of an object may be determined based on aligned X-ray images. It will be understood that X-ray image alignment may be performed and / or improved based on the 3D reconstruction of an object (or at least one of several objects). The 3D reconstruction determined based on aligned X-ray images may be used for further X-ray image alignment. Alternatively, the 3D reconstruction of an object may be determined based on a single or first X-ray image along with a 3D model of the object, and then used when aligning a second X-ray image with the first X-ray image.

[0051] In general, X-ray image registration and / or 3D reconstruction may be advantageous in the following situations: The focus is on determining the femoral anteversion angle. The focus is on determining the torsion angle of the tibia or humerus. The subject is the determination of the CCD angle between the head and shaft of the femur. The focus is on determining the anterior curvature (antecurvation) of long bones. The subject is determining the length of bones. This involves determining the entry point for implants in the femur, tibia, or humerus.

[0052] Examples of object combinations are listed below for illustrative purposes. Object 1 is the head of the arm, and the point is the tip of an opening instrument or drill. Object 1 is a vertebra, and the point is the tip of an opening instrument or drill placed on the surface of the vertebra. Object 1 is the tibia, and the point is the tip of the mouth opener. Object 1 is the tibia, and object 2 is the fibula, femur, talus, or another bone of the foot. Object 1 is the proximal portion of the femur, and object 2 is an opening device for the surface of the femur. Object 1 is the distal portion of the femur, and object 2 is an opening device for the surface of the femur. Object 1 is the distal part of the femur, object 2 is the proximal part of the femur, at least one X-ray image shows the distal part of the femur, at least one X-ray image shows the proximal part of the femur, and a further object is an opening device placed on the proximal part of the femur. Object 1 is the ilium, object 2 is the sacrum, and the dot is a mouth opener or the tip of a drill. Object 1 is an intramedullary nail implanted in the bone, and Object 2 is the bone itself. Object 1 is an intramedullary nail implanted in the bone, object 2 is the bone, and the dot is the tip of a sub-implant such as an opening instrument, drill, or fixing screw.

[0053] Calculation of 3D representation / reconstruction Once two or more radiographs are aligned, they can be used to compute a 3D representation or reconstruction of the anatomical structure shown at least partially within the radiographs. According to one embodiment, this can proceed along the lines proposed by P. Gamage et al., “3D reconstruction of patient specific bone models from 2D radiographs for IMAGE-guided orthopedic surgery” DOI:10.1109 / DICTA.2009.42. In the first step, the features of the bone structure of interest (typically characteristic bone margins, which may include the outer bone contour and some characteristic medial margins) are determined within each radiograph, perhaps using a neural network trained for segmentation. In the second step, a 3D model of the bone structure of interest is transformed so that its 2D projection matches the features determined in the first step (e.g., characteristic bone margins) within all available radiographs. While Gamage et al. use a general-purpose 3D model for the anatomical structure of interest, other 3D models, such as statistical shape models, may also be used. It should be noted that this procedure requires not only the relative field of view between images (provided by image alignment) but also the acquisition direction for one of the images. This direction may be known (for example, the surgeon may have been instructed to acquire images from a specific line of sight direction, e.g., anterior-posterior (AP) or medial-lateral (ML)) or it may be estimated based on various approaches (e.g., by using LU100907B1). While a more accurate relative field of view between images may improve the accuracy of the 3D reconstruction, the accuracy of determining the acquisition direction for one of the images may not be a critical factor.

[0054] The accuracy of the determined 3D representation can be improved by incorporating prior information about the 3D location of one or more points, or even a partial surface, on the bone structure in question. For example, in a 3D reconstruction of a femur with an implanted nail, a k-wire may be used to indicate a specific point on the surface of the femur in an X-ray image. From a previous procedural step, the 3D location of this indicated point in the coordinate system given by the implanted nail may be known. This knowledge can then be used to more accurately reconstruct the 3D surface of the femur. When such prior information about the 3D location of a specific point is available, this can even enable 3D reconstruction based on a single X-ray image. Furthermore, if the implant (such as a plate) is aligned with the shape of a portion of the bone and placed on this aligned portion of the bone, this information can also be used for 3D reconstruction.

[0055] As an alternative approach, 3D reconstruction of an object (e.g., bone) can also be performed without prior image alignment; that is, image alignment and 3D reconstruction can be performed together, as proposed by LU101009B1. This disclosure teaches that ambiguity resolution can be improved by limiting the allowable C-arm movement and / or by utilizing easily detectable features of another object (e.g., a drill or k-wire) present in at least two of the images on which simultaneous alignment and reconstruction are based. Such easily detectable features may be, for example, the tip of a k-wire or drill located either on the surface of the object being reconstructed or at a known distance from it. This feature must not move between image acquisitions. In the case of a k-wire or drill, this means that the instrument itself may change its inclination, as long as its tip is in a fixed position. Reconstruction without prior image alignment may work well if three or more images are used for such reconstruction. It should be noted that simultaneous image registration and 3D reconstruction generally perform better than approaches where registration is performed first, because it allows for simultaneous optimization of all parameters (i.e., for both registration and reconstruction). This is especially true in redundantly determined cases, such as reconstructing the 3D surface of bone using prior information about implanted nails or plates and the 3D positions of points on their surface.

[0056] For simultaneous image alignment and 3D reconstruction, a first X-ray image showing a first portion of a first object may be received, the first X-ray image being generated in a first imaging direction and at a first position of the X-ray source relative to the first object, and at least one second image showing a second portion of the first object may be received, the second X-ray image being generated in a second imaging direction and at a second position of the X-ray source relative to the first object. By using a model of the first object, the model can be deformed and fitted to match its appearance in the X-ray images, so that the projections of the first object in the two X-ray images can be matched together so that the spatial relationships of the images can be determined. The result of such simultaneous alignment and 3D reconstruction may be enhanced by at least one point having a fixed 3D position relative to the first object, which is identifiable and detectable in at least two of the X-ray images (it will be understood that three or more images may also be aligned while improving the 3D reconstruction). Furthermore, at least a portion of a second object having a fixed 3D position relative to the first object may be taken into consideration, and based on a model of the second object, at least a portion of the second object may be identified and detected in the X-ray image.

[0057] It should be noted that the first and second parts of the first object may overlap, which improves the accuracy of the results. For example, both the so-called first and second parts of the first object could be the proximal part of the femur, and the imaging directions could be different so that at least the appearance of the femur differs in the image.

[0058] Determination of planting curves and / or entrance points An objective of the present invention may be to determine the implantation curve or path along which an implant, such as a nail or screw, is inserted and planted within the bone, and / or to determine the entry point, which is the point at which the surgeon incises the bone to insert the implant. The entry point is therefore the intersection of the implantation curve and the bone surface. The implantation curve may be a straight line (or axis), or it may be curved, since the implant (e.g., a nail) has a curve. It should be noted that the optimal position of the entry point may also be determined by the location of the implant and the fracture within the bone, i.e., how far apart the fracture is distally or proximal.

[0059] There are various situations in which the implantation curve and / or entry point may need to be determined. In some cases, especially when a complete anatomical reduction has not yet been performed, only the entry point may be determined. In other cases, the implantation curve is obtained first, and the entry point is then obtained by determining the intersection of the implantation curve and the bone surface. In yet another case, the implantation curve and entry point are determined simultaneously. Examples for all of these cases are described in this invention.

[0060] Generally, a 2D X-ray image is received according to one embodiment, and the X-ray image shows the surgical site of the subject. Within the X-ray image, a first point associated with the structure of the subject and an implantation path within the bone for the implant intended to be implanted may be determined, and the implantation curve or path has a predetermined relationship with respect to the first point. The entry point for inserting the implant into the bone is located on the implantation path. It will be understood that the first point may not be the entry point.

[0061] Based on 3D reconstruction of the bone, the system can also assist in selecting the optimal implant and calculating the optimal intraosseous position (implantation curve) (i.e., entry point, insertion depth, rotation, etc.) so that the implant is sufficiently far from narrow areas of bone. Once an entry point is selected, the system can calculate a new ideal intraosseous position based on the actual entry point (if the implant is already visible in the bone). The system can then update the 3D reconstruction to take into account the actual position of the bone fragments. The system can also calculate and display the predicted positions of sub-implants that have not yet been implanted. For example, in the case of a pulpectomy nail, the predicted position of the neck screw / blade can be calculated based on a complete 3D reconstruction of the proximal femur.

[0062] Freehand fixing procedure Based on the aforementioned general determination of points and implantation paths in 2D X-ray images, for example, when considering the implantation of a bone nail fixing screw, the following conditions may be satisfied for a predetermined relationship between the implantation path and the point: if the structure in question is a hole in the implant, the hole may have a predetermined axis, the point may be associated with the center of the hole, and the implantation path may be a point in the direction of the axis of the hole.

[0063] One possible application described is a workflow example for a freehand fixation procedure in which an implant is secured by inserting a screw through a hole in the implant. According to one embodiment, the position of an already implanted nail is located in an X-ray image, which determines the implantation curve, where the implantation curve is a straight line (axis) along which the screw is implanted. A 3D reconstruction of the bone surface (at least near the implantation curve) can be performed for an already implanted nail (i.e., within the coordinate system given by the nail). This can proceed as follows: At least two X-ray images are acquired from different line-of-sight directions (e.g., one AP or ML image and one image taken from an oblique angle). The X-ray images can be classified and aligned using a neural network, for example with respect to the implanted nail, and the bone contour is segmented in all images, possibly by a neural network. The 3D reconstruction of the bone surface can possibly follow the 3D reconstruction procedure outlined above. The intersection of the implantation curve and the bone surface determines the 3D position of the entry point for the nail. Since the line of sight direction in the X-ray image can be determined based on the location of the nail, this also makes it possible to indicate the location of the entry point in a given X-ray image.

[0064] The accuracy of this procedure may be improved by incorporating the known 3D position of at least one point on the bone surface relative to the nail. Such knowledge can be obtained by combining the procedure in this invention with the freehand fixation procedure taught by EP19217245. A possible approach may be to use EP19217245 to obtain an entry point for the first fixation hole, which then becomes a known point on the bone surface. This known point can be used in this invention for 3D reconstruction of the bone and subsequent determination of entry points for second and further fixation holes. A point on the bone surface may also be identified, for example, by the tip of a drill touching that bone surface. If a point is identified in two or more X-ray images acquired from different imaging directions, this can improve accuracy.

[0065] Determining the entry point for implanting a nail into the femur. When considering the implantation of a nail into the femur based on the aforementioned general determination of the first point and implantation route in the 2D X-ray image, at least one of the following conditions may be satisfied for the predetermined relationship between the implantation route and the first point: If the target structure is the femoral head, the first point can be associated with the center of the femoral head and, as a result, can be positioned on the proximal extension of the implantation pathway, i.e., proximal to the entry point in the X-ray image. If the target structure is a narrow portion of the femoral neck, the first point may be associated with the center of the cross-section of the narrow portion of the femoral neck, and the proximal extension of the implantation route may be closer to the first point within the narrow portion than to the outer surface of the femoral neck. If the target structure is a narrow portion of the femoral shaft, the first point may be associated with the center of the cross-section of the narrow portion at the proximal end of the femoral shaft, and the implantation route may be closer to the first point within the narrow portion than to the outer surface of the femoral shaft. If the target structure is the isthmus of the femoral shaft, the first point may be associated with the center of the cross-section of the isthmus, and the first point may be located on the implantation path.

[0066] In the embodiments, the structure of interest does not need to be fully visible in the X-ray image. It may be sufficient for only 20 to 80 percent of the structure of interest to be visible in the X-ray image. Depending on the specific structure of interest, i.e., whether the structure of interest is the femoral head, femoral neck, femoral shaft, or another anatomical structure, at least 30 to 40 percent of that structure needs to be visible. As a result, for example, the center of the femoral head may be identifiable even if only 20 to 30 percent of the femoral head is visible, even if the center itself is not visible in the X-ray image, i.e., outside the imaging area. The same is possible for the isthmus of the femoral shaft, even if the isthmus is outside the imaging area and only 30 to 50 percent of the femoral shaft is visible.

[0067] To detect a target point within an image, a neural segmentation network can be used to classify each pixel as a potential keypoint. A neural segmentation network can be trained on a 2D Gaussian heatmap with centers located at true keypoints. Gaussian heatmaps can be rotationally invariant, or they can be directional if uncertainty in a particular direction is acceptable. To detect a target point outside the image itself, one possible approach is to segment additional pixels outside the original image, using all the information contained within the image itself to allow extrapolation.

[0068] An example workflow for determining the entry point for implanting an intramedullary or capillary nail into the femur is presented. According to one embodiment, firstly, the projection of the implantation curve is determined relative to the X-ray image. In this embodiment, the implantation curve is approximated by a straight line (i.e., the implantation axis). As a first step, it may be checked whether the current X-ray image satisfies the necessary requirements for determining the implantation axis. These requirements may include image quality, sufficient visibility of a part of the anatomical structure, and at least a generally appropriate field of view (ML) over the anatomical structure. Furthermore, the requirements may include whether the aforementioned conditions are satisfied. These requirements may be checked by an image processing algorithm, possibly utilizing a neural network. In addition, where applicable, the relative positions of bone fragments may be determined and compared to their desired positions, based on which it may be determined whether these bone fragments are positioned sufficiently well (i.e., the anatomical reduction has been performed sufficiently well).

[0069] More specifically, the aforementioned conditions can be described as follows: The implantation axis is determined by a single point and direction associated with at least two anatomical landmarks (for example, these may be the center of the femoral head and the isthmus of the femoral shaft). As previously stated, landmarks may be determined by a neural network, even if they are not visible in the X-ray image. Whether a proposed implantation axis is acceptable can be checked by determining the distance from the proposed axis to various landmarks on the bone contour as visible in the X-ray. For example, the proposed implantation axis should pass near the center of the femoral neck isthmus, i.e., it should not be too close to the bone surface. If such conditions are violated, the X-ray image has not been acquired from the appropriate imaging direction, and another X-ray image from a different imaging direction needs to be acquired. Determining the implantation curve in another X-ray image from a different line of sight direction may result in a different implantation axis and, therefore, a different entry point. The present invention also teaches a method for adjusting the imaging apparatus to acquire X-ray images from the appropriate direction.

[0070] It should be noted that implants may have curvature, meaning that a straight implantation axis can only approximate the projection of the inserted implant. The present invention, instead, can determine an implantation curve that more closely follows the 2D projection of the implant, based on a 3D model of the implant. Such an approach may use multiple points associated with two or more anatomical landmarks to determine the implantation curve.

[0071] The projection of the implantation axis determines the implantation plane in 3D space (or, more generally, the projection of the implantation curve determines the 2D manifold in 3D space). The entry point can be obtained by intersecting this implantation plane with another bone structure known to contain the entry point, which can be approximated by a line. In the case of the femur, such a bone structure may be the trochanteric rim, which is narrow and straight enough to be well approximated by a line, and on which the entry point can be considered to be located. Note that, depending on the implant, other locations for the entry point may be possible, for example, on the piriformis fossa.

[0072] The trochanteric margin may be detectable in lateral radiographic images. Alternatively, or additionally, another point identifiable in the image (e.g., the tip of the indicated k-wire or some other mouth opener) may be used, for which some prior information regarding its position relative to the entry point is known. In the case of the femur, an example of this would be when palpation and / or previously acquired radiographic images from a different field of view (e.g., AP) have determined that the tip of the k-wire is located on the trochanteric margin, thereby restricting the position of the k-wire tip in at least one dimension or degree of freedom.

[0073] There are at least three ways to utilize such prior information regarding the tip of the k-wire (or any other opening instrument) relative to the entry point. The easiest possibility might be to use an orthogonal projection of the k-wire tip onto the projection of the implantation axis. In this case, it may be required to reposition the k-wire tip in subsequent X-ray images taken from a different angle (e.g., AP) based on the information in the ML image, and perhaps, after acquiring a new ML image after the repositioning, check whether the k-wire tip is still on the desired structure (trochanteric margin). Another possibility is to estimate the angle between the projection of the structure (which may not be identifiable in the ML image) and the projection of the implantation axis based on anatomical prior information, and project the k-wire tip obliquely onto the projection of the implantation axis at this estimated angle. Finally, a third possibility might be to use aligned pairs of AP and ML images to calculate the intersection of the projected epipolar line with the projected implantation axis, defined by joining the k-wire tip and the focus of the AP image in the ML image. Once the entry point is acquired, it determines the planting axis in 3D space.

[0074] Alternatively, the bone structure (here, the trochanteric rim), whose intersection with the implantation plane determines the entry point, can also be found by performing a partial 3D reconstruction of the proximal femur. According to one embodiment, this 3D reconstruction may proceed as follows, based on two or more X-ray images from different line-of-sight directions, at least two of which include the k-wire: The characteristic bone rim (including at least the bone contour) of the femur is detected in all the X-ray images. Furthermore, the femoral head is found in all the X-ray images, approximated by a circle, and the tip of the k-wire is detected. The images can now be aligned using the approach presented above, based on the characteristic bone rim, the approximated femoral head and the tip of the k-wire, and the restricted movement of the C-arm. After image alignment, a 3D surface including at least the trochanteric region can be reconstructed. The accuracy of the 3D reconstruction can be improved by utilizing prior information regarding the distance of the k-wire tip from the bone surface (e.g., which can be determined from the AP image). Various alternatives to this procedure are possible and will be described in the detailed description of the embodiments.

[0075] In the aforementioned approach, the implantation curve is determined within 2D X-ray images, and then various alternative means for obtaining the entry point are described. Alternatively, the entire procedure (i.e., determination of the implantation curve and entry point) may be based on a 3D reconstruction of the proximal femur (or, if retrograde nails are used, the distal femur) including a sufficient portion of the diaphysis. Such a 3D reconstruction may again be based on multiple X-ray images that have been aligned using the methods presented above. For example, alignment may use ball-based approximation of the femoral head and cylindrical or mean diaphysis shape approximation of the diaphysis. Alternatively, simultaneous optimization and determination of alignment and bone reconstruction (including the surface and possibly internal structures such as the pulposus and endocortex) may be performed. Once the 3D reconstruction of the relevant portion of the femur is obtained, the 3D implantation curve can be fitted by optimizing the distance between the implant surface and the bone surface. The intersection of the 3D implantation curve and the already determined 3D bone surface yields the entry point.

[0076] With respect to a 2D X-ray image, the position and orientation of the implantation curve are determined based on a first point, the implantation curve includes a first portion within the bone at a first distance from the bone surface and a second portion within the bone at a second distance from the bone surface, where the first distance is smaller than the second distance, and the first point is located on a first identifiable structure of the bone and slightly away from the first portion of the implantation axis. A second point may be used, which may be located on an identifiable structure of the bone and slightly away from the second portion of the implantation curve. Furthermore, the position and orientation of the implantation curve may be further determined based on at least one additional point, at least one additional point located on a second identifiable structure of the bone and positioned on the implantation curve.

[0077] Determining the entry point for implanting a nail into the tibia. Based on the simultaneous alignment and 3D reconstruction described in the "3D Representation / Reconstruction Calculation" section above, the entry point for implanting the intramedullary nail into the tibia can be determined.

[0078] According to one embodiment, accuracy is improved and ambiguity resolved by requiring the user to place an opening instrument (e.g., a drill or k-wire) on the surface of the tibia at any point on its proximal portion, but ideally near the presumed entry point. The user acquires a lateral image of the proximal portion of the tibia and at least one AP image. A 3D reconstruction of the tibia can be calculated by fitting a statistical model of the tibia together to its projection in all the X-ray images, taking into account the fact that the tip of the opening instrument does not move between images. Accuracy can be further improved by requiring the user to acquire two or more images from different (e.g., nearly AP) imaging directions, and possibly another (e.g., lateral) image as well. Any overlapping determinations can enable the detection of possible movement of the tip of the opening instrument and / or verify the detection of the tip of the opening instrument.

[0079] Based on a 3D reconstruction of the tibia, this system can determine the entry point, for example, by identifying it on the average shape of a fitted statistical model. Such guidance for finding the entry point for an antegrade tibial nail based solely on images (i.e., without palpation) may allow surgeons to perform a suprapatellar approach, which may generally be preferable, but it should be noted that it conventionally has the disadvantage of not being able to palpate the bone at the entry point.

[0080] Determining the entry point for implanting a nail into the humerus. A further application of the proposed image registration and reconstruction techniques presented above could be the determination of entry points for implanting intramedullary nails into the humerus.

[0081] In general, a system including a processing unit for processing X-ray images may be used to assist in X-ray image-based humeral surgery to achieve the aforementioned objectives. When a software program product is run on the processing unit, it causes the system to perform a method including the following steps: First, a first X-ray image is received, generated in a first imaging direction and showing the proximal portion of the humerus; and second X-ray images are received, generated in a second imaging direction and showing the proximal portion of the humerus. These images may include the proximal portion of the humeral shaft as well as the articular surface and further the glenoid fossa, i.e., the humeral head with articular structures complementary to the shoulder. Note that the second imaging direction is typically different from the first imaging direction. Next, (i) the first and second X-ray images are aligned, (ii) an approximation of at least a portion of the 2D contour of the humeral head is determined in both images, (iii) a 3D approximation of the humeral head is determined based on the approximated 2D contour and the alignment of the first and second images, and (iv) the 2D image coordinates of the sum of at least three different points in the first and second X-ray images are determined. Finally, an approximation of the anatomical neck is determined as a curve on the 3D approximation of the humeral head based on at least three determined points. Note that the at least three points do not need to lie on the determined curve. If it is possible to determine additional points of the anatomical neck that are not on the same plane as the first three points, a more accurate approximation of the anatomical neck can be determined. This may allow for the determination of the rotational position of the anatomical neck and, therefore, the humeral head around the shoulder joint axis. Another method for determining the rotational position around the joint axis may be to detect the positions of the greater tubercle and / or lesser tubercle, provided that at least one of the two is in a fixed position relative to the proximal bone fragment. Another alternative may involve using 3D information acquired before surgery (e.g., CT scans) to generate a 3D reconstruction of the proximal bone fragment based on X-ray images taken during surgery. This method may be combined with the methods described above.

[0082] According to one embodiment, at least a portion of the 2D contour of the humeral head may be an approximation of a 2D circle or a 2D ellipse. Furthermore, the 3D approximation of the humeral head may be a 3D ball or a 3D ellipsoid. The approximation of the anatomical neck may be a circle or an ellipse in 3D space.

[0083] According to one embodiment, further X-ray images may be received, and an approximation of the humeral shaft axis may be determined in at least two X-ray images from the group consisting of a first X-ray image, a second X-ray image, and further X-ray images. Based on the approximated humeral shaft axis in at least two X-ray images, along with the alignment of the first and second X-ray images, an approximation of the 3D shaft axis of the humerus may be determined.

[0084] According to one embodiment of the disclosed method, the entry point and / or the dislocation of the proximal fragment of the fractured humerus may then be determined based on an approximated anatomical neck and an approximated 3D trunk axis and / or an approximated glenoid fossa of the humeral joint. As a result, the implantation curve may be determined within the proximal fragment based on the entry point and head dislocation. Furthermore, information for reducing the proximal fragment may be provided.

[0085] According to one embodiment, at least two X-ray images can be aligned, and these two X-ray images may be two from a first X-ray image, a second X-ray image, and a further X-ray image. The X-ray images can be aligned based on a model of the humeral head and based on one additional point having a fixed 3D position relative to the humeral head, which is identified and detected in at least two X-ray images. The one additional point may be the tip of an instrument and may be positioned on the articular surface of the humeral head. In this case, the fact that the distance between this point and the center of the humeral head is equal to the radius of the humeral head approximated by a ball can be used to improve the accuracy of the X-ray image alignment.

[0086] The embodiments of the method according to this disclosure are described in further detail below. The humeral head, located within the shoulder joint, can be approximated by a ball (sphere). Hereinafter, unless otherwise specified, the humerus is approximated by such a ball, which is understood to mean the approximation of the projection of the humerus in an X-ray image by a circle. Thus, “center” and “radius” always refer to such an approximating ball or circle. It should be noted that it may also be possible to use other simpler geometric approximations of the humeral head, for example, by an ellipsoid. In that case, the anatomical neck would be approximated by an ellipse.

[0087] The following describes an example workflow for determining the entry point. A complex issue in determining the entry point within the humerus is that fractures treated with humeral nails often occur along the surgical neck, thus altering the position of the humeral head. In a correct reduction, the center of the humeral head should be close to the humeral shaft axis. According to one embodiment, this can be verified in an axial X-ray image showing the proximal humerus. If the center of the humeral head is not close enough to the shaft axis, the user is advised to apply a traction force distal to the arm to compensate for rotation of the humeral head around the articular axis (which may not be detectable). An approximate entry point is then proposed on the shaft axis approximately 20% medial to the center of the head (meaning in the typical axial X-ray image described above). The user is then asked to place an opening instrument (e.g., a k-wire) on this proposed entry point. Alternatively, as mentioned above, to improve alignment accuracy, the system requires the user to intentionally position the mouth opener inside the predicted entry point (meaning 30–80 percent above the indicated center of the humeral head in the axial X-ray image) to ensure that the tip of the instrument is positioned over the spherical portion of the humeral head. The system can detect the humeral head and the tip of the instrument in the new axial X-ray image (for example, by using a neural network).

[0088] The user is then instructed to acquire an AP image, allowing only certain C-arm movements (e.g., rotation around the C-axis and additional translation) while keeping the tip of the instrument in place (the inclination of the instrument is permitted to be changed). The humeral head and the tip of the instrument are detected again. The axial and AP images can then be aligned based on the ball and the tip of the instrument approximating the humeral head, as described above in the section "3D alignment of two or more X-ray images".

[0089] The curve that defines the limits of the articular surface of the shoulder joint is called the anatomical neck (anatomical neck of the humerus). The anatomical neck defines the limits of the spherical portion of the humerus, but it is typically impossible for a surgeon to identify it in an X-ray image. It can be approximated by a 2D circle in 3D space, obtained by intersecting a plane with a ball that approximates the humeral head, and this plane is inclined with respect to the trunk axis of the humerus. The spherical articular surface is oriented dorsally in an upward (valgus) direction (with the patient's arm suspended relaxed downward from the shoulder, parallel to the chest). Three points are sufficient to define this intersecting plane. Axial and AP X-ray images can each allow for the determination of two points on the anatomical neck, namely the start and end points of the arc of the circle that defines the limits of the spherical portion of the humerus. This is therefore a matter of overlapping determination: based on two X-ray images, four points can be determined, while only three points are needed to define the intersecting plane. If additional X-ray images are used, the problem may become even more redundantly determined. This redundant determination may allow for a more accurate calculation of intersecting planes, or it may allow for addressing situations where a point might not be determined, for example, because it is blocked.

[0090] When determining the approximation of the anatomical neck by intersecting a determined plane with a ball approximating the humeral head, it should be noted that various modifications may be possible. For example, the intersecting plane may be shifted laterally to constitute a more accurate position of the anatomical neck on the humeral head. Alternatively, or additionally, the radius of the circle approximating the anatomical neck may be adjusted. It may also be possible to use a geometric model with more degrees of freedom to approximate the humeral head and / or the anatomical neck.

[0091] The entry point can be taken so as to be the point on the anatomical neck closest to the intersection of the trunk axis and the bone surface in 3D space, or it can be positioned at a user-defined distance medially from that point. The thus determined anatomical neck and entry point can be displayed as an overlay in the current X-ray image. If this entry point is very close to the circle approximating the head in the X-ray image, this will result in a potentially large inaccuracy in the z-coordinate. To mitigate this situation, instructions may be given to rotate the C-arm so that the proposed entry point moves further inward toward the head in the X-ray image. This can be advantageous because, in either case, obtaining an X-ray image in which the entry point is placed close to the approximating circle can be difficult due to mechanical constraints. In other words, the rotation of the C-arm between the axial direction and the AP image may be, for example, 60 degrees, which may be easier to achieve in the surgical workflow than a 90-degree rotation.

[0092] Further details of this workflow, optional embodiments, and extensions are described in the detailed descriptions of the embodiments below.

[0093] Further positioning methods enabling near real-time continuous 3D alignment of objects

[0094] This disclosure teaches two further methods for locating an object (e.g., a drill or a small-diameter implant) whose geometry may be unlocatable without further information about the depth of its imaging, and for determining the 3D position and 3D orientation of such object relative to another object, such as a nail, bone, or a combination thereof (i.e., for providing 3D alignment of these objects). The first method does not require a 2D-3D match of the object (e.g., a drill), and it may suffice to detect a point of this object (e.g., the tip of the drill) in two X-ray images. For example, if a soft tissue protection sleeve is used during drilling, a 2D-3D match of the drill may require stopping the drill and retracting the sleeve before acquiring the X-ray image, which can be tedious and error-prone, so this may be advantageous. For accurate 2D-3D matching, this retraction may be necessary because, otherwise, the tip of the drill may not be sufficiently visible in the X-ray image, even if the drill is already in the bone. The proposed method may be advantageous because the drill tip may be rotating, and retracting the sleeve may not be necessary for acquiring X-ray images.

[0095] The second method presented here does not require rotation or readjustment of the C-arm (even if changing the C-arm position is not prohibited). For example, in a drilling scenario, this could allow for continuous verification of the actual drilling trajectory at any point during the drilling process and comparison with a requested trajectory based on X-ray images, along with near real-time (NRT) feedback to the surgeon.

[0096] In the first method, the 3D position of an identifiable point of an object (e.g., the tip of a drill) relative to another object (e.g., a nail) can be determined, for example, by taking two X-ray images from different line-of-sight directions (without moving the tip of the drill between the acquisition of these two images), detecting the tip of the drill in both X-ray images, aligning them based on known nail coincidences, and then calculating the point of the best approximation of the epipolar line passing through each tip of the drill in the 3D nail coordinate system. The relative 3D orientation of an object (e.g., a drill) can be determined if the axis of the object contains a specific point whose 3D coordinates relative to another object (e.g., a nail) are known (e.g., it is known that the drill axis passes through the entry point on the bone surface, i.e., the position of the tip of the drill at the start of drilling). When calculating the relative 3D orientation of an object, potential bending of the drill and distortion of the X-ray image at each part can be taken into consideration.

[0097] The second method eliminates ambiguity regarding the z-coordinate of an object (e.g., a drill) by incorporating prior information that a known axis within the object's coordinate system (e.g., the drill shaft) passes through a point whose 3D coordinate relative to another object (e.g., a nail) is known (e.g., the entry point, i.e., the starting point of drilling). Again, in such trajectory calculations, potential bending of the drill and distortion of the X-ray image at each part can be taken into account.

[0098] If the actual drilling trajectory does not match the required drilling trajectory (i.e., in the case of distal fixation, the drill may miss the nail fixation hole if it continues along its current path), the system may instruct the user to tilt the power tool, using the rotary drill bit, by a specified angle. Doing so widens the side passage through the cancellous bone, and as a result returns to the correct trajectory. Since this widens the entry hole into the bone and therefore may shift the position of the original entry point, such correction may need to take this additional uncertainty into account.

[0099] This method may also be able to address implants consisting of a plate-nail combination using a screw connection between the hole in the plate and the hole in the nail. NRT guidance for such implant types may proceed as follows: Based on a 3D reconstruction of the relevant anatomical structure, the ideal position for the combined implant may be calculated, and a trade-off may be made between the goodness of the plate position (e.g., surface matching) and the goodness of the nail position (e.g., sufficient distance from the bone surface in narrow areas). Based on the calculated position, the entry point for the nail into the bone may be calculated. After nail insertion, the ideal position of the combined implant may be recalculated based on the current position of the nail axis. This system may provide the surgeon with guidance to rotate and translate the nail so that the final position of the nail and, where applicable, the projected final positions of the sub-implants (e.g., screws) and, at the same time, the plate (which is more or less firmly bonded to the nail) are optimized. After reaching the final position of the nails, the system can provide support for optimally positioning the plate by determining its position in the X-ray image (where it has not yet reached its final destination) and taking into account the constraints imposed by the already inserted nails. Next, drilling can be performed through the holes in the plate. This drilling is a critical step, as the drilling must also reach the nail holes, and mis-drilling may not be possible from a different starting point, making correction difficult. If the plate has already been secured previously (using screws that do not go through nails), the starting point and therefore the entry point for drilling are also determined. In such cases, verification and correction of the drill angle may be possible multiple times if necessary.

[0100] If the plate holes only allow drilling at specific angles, the positioning of the plate based on the actual position of the nails may be decisive. In such cases, with no further room for adjustment, the system can provide guidance for positioning the plate based on the current position of the nails. This may allow deriving a drilling trajectory during drilling based solely on the alignment of the plate with the nails, which may then allow determining the position of the drill even if only a small portion of the drill is visible in the X-ray image (the drill tip may still be required).

[0101] The proposed system can provide the surgeon with continuous guidance in near real-time. If the alignment is sufficiently rapid, even a continuous video stream from the C-arm can be evaluated, providing pseudo-continuous navigation guidance to the surgeon. By calculating the relative 3D position and orientation of objects in the current X-ray image and comparing them to the desired conformation, the surgeon can be given instructions on how to achieve the desired conformation. Necessary adjustments or movements can be performed freehand by the surgeon, or the surgeon can be supported mechanically and / or with sensors. For example, an accelerometer could be attached to a power instrument to support the adjustment of the drill angle. Another possibility is the use of a robot capable of positioning one or more objects according to the calculated necessary adjustments. Based on the NRT feedback of this system, adjustments can be recalculated at any time and corrected if necessary.

[0102] Reduction support Another object of the present invention may be to support the anatomically correct reduction of bone fragments. Typically, surgeons attempt to reduce fractured bone fragments in the most natural relative position possible. For further improved results, it may be considered to check whether such reduction is anatomically correct before or after inserting any implants for fixation.

[0103] Reduction can be supported by calculating a 3D reconstruction of the bone in question. Such a 3D reconstruction does not need to be a complete reconstruction of the entire bone and may not need to be accurate in all aspects. In cases where only specific measurements are extracted, the 3D reconstruction only needs to be accurate enough to allow for a sufficiently accurate determination of these measurements. For example, if the femoral anteversion angle (AV) is to be determined, it may be sufficient to have a sufficiently accurate 3D reconstruction of the femur in the condyloid process and neck. Other examples of measurements of the subject may include leg length, degree of leg deformity, curvature (such as femoral anteversion), or craniocervical-stomatal (CCD) angle, as there is often varus rotation of the proximal femoral fragment that occurs before or after intramedullary nail insertion. Once the measurements of the subject are determined, they can be used to select an appropriate implant, or they can be compared to a desired value, which may be derived from a database or be patient-specific, for example, by comparing the leg undergoing surgery to the other healthy leg. The surgeon may be given instructions on how to achieve a desired value, such as a desired anteversion angle.

[0104] This could include monitoring a certain measurement throughout surgery by automatically calculating it from available X-ray images, and perhaps also alerting the surgeon if the measurement deviates excessively from the desired value.

[0105] In some cases, 3D reconstruction may be possible even from a single X-ray image, especially if the line of sight can be determined (e.g., based on LU100907B1) and only specific measurements (e.g., CCD angle) are relevant. However, generally, two or more X-ray images taken from different line of sight and / or showing different parts of the bone can improve the accuracy of 3D reconstruction (see the "Calculation of 3D Representation / Reconstruction" section above). 3D reconstruction can be calculated even for parts of the bone that are not visible or only partially visible in the X-ray image, provided that the invisible parts are not displaced relative to the visible parts due to the fracture, or if the displacement parameters are already known or can be determined by other means. For example, based on a statistical 3D model of the femur, the femoral head can be reconstructed fairly accurately from a pair of ML and AP images if most of the femoral head is not visible. As another example, the distal part of the femur can be reconstructed based on two proximal X-ray images if the femoral shaft is not fractured. Needless to say, the accuracy of distal reconstruction can be improved if additional X-ray images showing that distal region are also available.

[0106] In 3D reconstruction of bone based on two or more X-ray images, accuracy can be further improved if these X-ray images can be aligned before the 3D reconstruction is calculated, following one of the approaches described in the "3D Alignment of Two or More X-ray Images" section above. In cases where the 3D reconstruction of bone is calculated based on two or more X-ray images showing different parts of the bone (e.g., two X-ray images showing the proximal part of the femur and one X-ray image showing the distal part of this femur), the 3D alignment of the X-ray images showing different parts may be possible by basing it on an object that has a visible, known 3D model (e.g., a nail already implanted) in at least one X-ray image for each part of the bone, and / or by limiting the acceptable C-arm movement between the acquisition of those X-ray images (see LU101009B1).

[0107] The AV angle may need to be determined either before or after incising the patient (for example, to detect the dorsal gap in the reduction of a trochanteric fracture) when the implant has not yet been inserted. In such cases, the alignment of two or more images of the proximal femur (e.g., AP and ML) may be carried out in accordance with the guidelines in the "3D Alignment of Two or More X-ray Images" section above, as follows: When determining the entry point for nail insertion, an opening instrument (whose diameter is known), such as a k-wire, may be placed on the suspected entry point and consequently detected in the X-ray image. Based on the position of its tip, the image may be aligned together with the detected femoral head. In cases where further objects, such as a k-wire, are not visible in the X-ray image, image alignment may still be performed by requiring a specific movement of the C-arm between images. For example, the system may require a 75-degree rotation of the C-arm around the C-axis. If this rotation is performed with sufficient precision, image alignment can also be performed with sufficient precision. Non-overlapping bone portions (e.g., the distal and proximal portions of the femur) can be aligned by restricting the allowed C-arm movement to translation only, as described in one embodiment.

[0108] It should be noted that 3D reconstruction does not necessarily require determining the AV angle. Determining one additional point, for example near the cervical axis, may provide sufficient information to determine the AV angle based on the 2D approach. Alignment of 2D structures detected in the X-ray image (e.g., structures within the proximal and distal parts of the femur) can be performed by employing the aforementioned method.

[0109] In other cases, for example, when determining the correct rotation angle of a bone, it may be beneficial to consider adjacent bones or bone structures. For instance, in the case of a fractured tibia, the evaluation of its proximal orientation may take into account the femoral condyle, patella, and / or fibula. Similar comments apply to the evaluation of its distal rotation position. The relative position of the tibia to the fibula or other bone structures (e.g., overlapping edges of joints in the foot) may clearly indicate the line of sight to the distal tibia. All these evaluations are obtained based on a neural network, which may perform simultaneous optimization based on the confidence value (of correct detection) for each structure considered. The results of such evaluations can be combined with knowledge of patient or extremity positioning to assess the current reduction of the bone. For example, in the case of a humerus, the system may instruct the surgeon to position the patient's radius parallel to the patient's body. For reduction evaluation, detecting these structures in X-ray images can consequently guide the user to achieve a central position relative to the glenoid fossa of the humeral articular surface, which may be sufficient.

[0110] X-ray dose reduction It should be kept in mind that the overall objective may be to reduce X-ray exposure to patients and operating room staff. During fracture treatment according to the embodiments disclosed herein, as few X-ray images as possible should be generated. For example, images acquired to check the positioning of the proximal fragment relative to the distal fragment may also be used for determining the entry point. As another example, images generated in the entry point determination process may also be used to measure the AV angle or CCD angle.

[0111] According to one embodiment, since it is not necessary to have a complete anatomical structure visible in the X-ray image, X-ray exposure can also be reduced. 3D reconstruction or localization of objects such as anatomical structures, implants, surgical instruments, and / or components of implant systems can be provided even if they are not visible or are only partially visible in the X-ray image. For example, even if the projected image does not fully show the femoral head, it can still be fully reconstructed. As another example, the distal portion of the femur may be reconstructed based on one or more proximal images in which the distal portion is not fully shown.

[0112] In some cases, it may be necessary to determine a point of interest associated with an anatomical structure, such as the center of the femoral head or a specific point on the femoral shaft. In such cases, it may not be necessary for the point of interest to be shown in the X-ray image. This is especially true if uncertainty or inaccuracy in determining such a point of interest affects dimensions or degrees of freedom that are ultimately less important. For example, the center point of the femoral head and / or a specific point on the axis of the femoral shaft may be located outside the X-ray image, but based on a deep neural network approach, for example, the system may still be able to determine and utilize those points in order to calculate the implantation curve with sufficient accuracy, since inaccuracies in the direction of the implantation curve may not significantly affect the calculated implantation curve.

[0113] According to one embodiment, the processing unit of the system may be configured to determine points of anatomical structures and / or objects associated with those anatomical structures based on an X-ray projection image showing a certain minimum required percentage (e.g., 20%) of the anatomical structure. If less than the minimum required portion of the anatomical structure is visible (e.g., less than 20%), the system may guide the user to obtain the desired display. For example, if the femoral head is not visible at all, the system may provide instructions to move the C-arm in a direction calculated based on the appearance of the femoral shaft in the current X-ray projection image.

[0114] Matching 3D models with 2D projected images It should be noted that processed X-ray image data may be received from the imaging device, for example, directly from a C-arm or G-arm based 2D X-ray system, or alternatively from a database. X-ray projection images may represent the anatomical structure of the subject, particularly bone. Bones may be, for example, bones of the hand or foot, but more specifically, long bones of the lower limbs, such as the femur and tibia, and long bones of the upper limbs, such as the humerus. Images may also include bone implants or surgical instruments, such as artificial objects like drills or k-wires.

[0115] This disclosure distinguishes between “object” and “model.” The term “object” is used for actual objects, such as bone or a portion of bone, or another anatomical structure, or for implants such as intramedullary nails, bone plates or bone screws, or for surgical instruments such as sleeves or k-wires. “Object” may describe only a portion of an actual object (e.g., a portion of bone), or it may be an assembly of an actual object and therefore consist of sub-objects (e.g., the object “bone” may be fractured and therefore consist of sub-objects “a fractured portion of bone”).

[0116] On the other hand, the term “model” is used for a virtual representation of an object. For example, a dataset defining the shape and dimensions of an implant may constitute a model of the implant. As another example, a 3D representation of an anatomical structure, such as one generated as an example during a diagnostic procedure, may be taken as a model of an actual anatomical object. It should be noted that a “model” may describe a specific object, for example, a specific nail, or it may describe a class of objects, such as a femur, which may have some variability. In the latter case, such an object may be described, for example, by a statistical shape or appearance model. It may then be an object of the present invention to find a 3D representation of a specific instance from a class of objects shown in an acquired X-ray image. For example, it may be an object to find a 3D representation of a femur shown in an acquired X-ray image based on a general statistical shape model of a femur. It may also be possible to use a model that includes a discrete set of deterministic possibilities, and the system would then select which of these best describes the object in the image. For example, there may be several nails in the database, and the algorithm would then identify which nail is shown in the image.

[0117] It should be noted that a model may be a complete or partial 3D model of an actual object, or it may only describe certain geometric aspects of an object (which may be less than 3 in dimensions), such as the fact that the femoral or humeral head can be approximated by a ball in 3D and a circle in a 2D projection image, or that the shaft of a bone has a direction described by its trunk axis.

[0118] Since a 3D representation is essentially a set of computer data, it is readily possible to extract specific information from that data, such as the geometric characteristics and / or dimensions of a virtually represented object (e.g., axes, contours, curvature, center point, angles, distances, or radii). For example, if the width of a nail is known from the model data, and the scale is determined based on one object, this may also make it possible to measure the geometric characteristics or dimensions of another shown and potentially unknown object, provided that such an object is located at a similar imaging depth. If the imaging depth of one object is known (e.g., the object is large enough, or the size of the X-ray detector and the distance between the image plane and the focal point are known), and there is information about the difference in imaging depth between two objects (e.g., based on anatomical knowledge), it may even be possible to calculate the size of different objects at different imaging depths based on the intercept theorem.

[0119] According to one embodiment, objects in an X-ray image are automatically classified and identified within the X-ray projection image. However, objects can also be manually classified and / or identified within the X-ray projection image. Such classification or identification can be supported by the device by automatically referencing structures recognized by the device.

[0120] Matching a model of an object to its projection shown in an X-ray image may consider only selected features of the projection (e.g., contours or characteristic edges), or it may consider the overall appearance. Contours or characteristic edges may be determined using a neural segmentation network. The appearance of an object in an X-ray image is determined, among other things, by the attenuation, absorption, and deflection of X-ray irradiation, which are also determined by the material of the object. For example, a steel nail generally absorbs more X-ray irradiation than a titanium nail, which can affect not only the appearance of the projected image of the nail within its contour, but also the shape of the contour itself, for example, the contour of the nail hole. The strength of this effect is also determined by the X-ray intensity and the amount of tissue surrounding the object through which the X-ray beam must pass. As another example, the transition between soft and hard tissue may be discernible in an X-ray image because such a transition creates a border between dark and bright areas in the image. For example, the transition between muscle tissue and bone tissue may be a recognizable structure, but the transition between the endocortex, the medial spongy bone tissue and the lateral hard cortex may also be recognizable as a feature in an X-ray image. Whenever a bone contour is determined in this disclosure, it should be noted that such a contour may also be the endocortex or any other recognizable feature of the bone shape.

[0121] In one embodiment, 2D-3D matching can be performed on an object described by a deterministic model, following the approach outlined by Lavallee S., Szeliski R., Brunie L. (1993) Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps, in Laugier C. (eds): Geometric Reasoning for Perception and Action. GRPA 1991, Lecture Notes in Computer Science, vol.708. Springer, Berlin, Heidelberg. In this approach, additional effects such as image distortion (e.g., the pillow effect captured by an image intensifier tube) or nail bending can be absorbed by introducing additional degrees of freedom into the parameter vector or by using a appropriately tuned model.

[0122] In one embodiment, matching a virtual projection of an object described by a statistical shape or appearance model to its actual projection can be carried out in accordance with the approach outlined in V. Blanz, T. Vetter (2003), Face Recognition Based on Fitting a 3D Morphable Model, IEEE Transactions on Pattern Analysis and Machine Intelligence. In this paper, a statistical, morphable 3D model is fitted to a 2D image. For this purpose, statistical model parameters for contours and appearances, as well as camera and pose parameters for their respective projections, are determined. Another approach may follow X. Dong and G. Zheng, Automatic Extraction of Proximal Femur Contours from Calibrated X-Ray Images Using 3D Statistical Models, in T. Dohi et al. (Eds.), Lecture Notes in Computer Science, 2008. Deforming the 3D model in such a way that its virtual projection matches the actual projection of the object in the X-ray image also enables the calculation of the imaging direction (which describes the direction in which the X-ray beam passes through the object).

[0123] When displaying X-ray images, geometric features and / or dimensions may be shown as overlays within the projected image. Alternatively, or additionally, at least a portion of the model may be shown within the projected image, for example, as a transparent visualization or 3D rendering, which may facilitate the identification of the model and, therefore, the structural features of the object imaged by the user. [Effects of the Invention]

[0124] Overall review For definitions of the rotation and translation axes of the C-arm, refer to Figure 25. In this figure, the X-ray source is indicated by XR, the rotation axis indicated by letter B is called the vertical axis, the rotation axis indicated by letter D is called the propeller axis, and the rotation axis indicated by letter E is called the C axis. Note that for some C-arm models, axis E may be closer to axis B. The intersection point between axis D and the central X-ray beam (labeled XB) is called the center of the "C" in the C-arm. The C-arm can be moved up and down along the direction indicated by letter A. The C-arm can also be moved along the direction indicated by letter C. The distance of the vertical axis from the center of the "C" in the C-arm may vary depending on the C-arm. Note that it may also be possible to use a G-arm instead of a C-arm.

[0125] A neural network can be trained on a large amount of data comparable to the data to which it is applied. In the case of evaluating bone structures in an image, the neural network should be trained on a large number of X-ray images of the bone in question. It will be understood that a neural network can also be trained on simulated X-ray images.

[0126] According to one embodiment, two or more neural networks may be used, each of which may be specifically trained for the substeps necessary to achieve the desired solution. For example, a first neural network may be trained to evaluate X-ray image data to classify anatomical structures in a 2D projection image, while a second neural network may be trained to detect characteristic edges of that structure in the 2D projection image. A third network may be trained to determine specific keypoints, such as the center of the femoral head. The neural networks may also be combined with other algorithms, including but not limited to model-based algorithms such as Active Shape Model. It should also be noted that the neural networks may directly solve one of the tasks within the present invention, for example, the determination of implantation curves.

[0127] It should be noted that a processing unit can be implemented by a single processor that executes all the steps of a process, or by a group or multiple processors that do not need to be located in the same place. Cloud computing, for example, allows processors to be located anywhere. For example, a processing unit could be divided into a first subprocessor that controls user interaction, including a monitor for visualizing results, and a second subprocessor that performs all the calculations (perhaps located elsewhere). The first or another subprocessor could also control the movement of the C-arm or G-arm of an X-ray imaging device, for example.

[0128] According to one embodiment, the apparatus may further include, for example, storage means for providing a database for storing X-ray images. It will be understood that such storage means may also be provided within a network to which the system may be connected, and that data related to a neural network may be received via that network. Furthermore, the apparatus may include an imaging unit for generating at least one 2D X-ray image, the imaging unit may be capable of generating images from different directions.

[0129] According to one embodiment, the system may include a device for providing information to a user, the information including at least one piece of information from a group consisting of X-ray images and instructions relating to procedural steps. It will be understood that such a device may be a monitor or augmented reality device for visualizing the information, or it may be a speaker for providing the information acoustically. The device may further include input means for manually determining or selecting the location or part of an object in an X-ray image, such as a bone contour, for measuring distance in the image. Such input means may be, for example, a computer keyboard, computer mouse, or touchscreen for controlling a pointing device, such as a cursor on a monitor screen, which may be included in the device. The device may also include a camera or scanner for reading a package label or otherwise identifying an implant or surgical instrument. The camera may also allow the user to communicate with the device visually by gesture or imitation, for example, by virtually touching a device displayed in virtual reality. The device may also include a microphone and / or speaker that can communicate acoustically with the user.

[0130] It should be noted that all references to C-arm movement within this disclosure always refer to relative repositioning between the C-arm and the patient. Therefore, any C-arm translation or rotation can generally be replaced by the corresponding translation or rotation of the patient / OR table, or a combination of C-arm translation / rotation and patient / table translation / rotation. This is particularly relevant when dealing with limbs, as moving the patient's limbs may actually be easier than moving the C-arm. It should be noted that required patient movement generally differs from C-arm movement, and in particular, patient translation is typically unnecessary when the structure of interest is already in the desired position in the X-ray image. This system can calculate C-arm adjustments and / or patient adjustments. It should be further noted that all references to the C-arm can similarly apply to the G-arm.

[0131] The methods and techniques disclosed herein may be used in systems supporting human users or surgeons, or they may be used in systems in which some or all of the steps are performed by a robot. Accordingly, all references to “user” or “surgeon” in this patent application may refer to human users and robotic surgeons, mechanical support devices, or similar devices. Similarly, whenever it is mentioned that instructions are given regarding how to adjust a C-arm, it should be understood that such adjustments may also be performed without human intervention, i.e., automatically, by the robotic C-arm, by the robotic table, or they may be performed by OR staff with some automated support. It should be noted that because robotic surgeons and / or robotic C-arms may operate with greater precision than humans, repetitive procedures may require fewer repetitions, and more complex instructions (e.g., combinations of multiple repetitive steps) may be performed.

[0132] The computer program may preferably be loaded into the random access memory of a data processor. A data processor or processing unit of a system according to one embodiment may therefore be equipped to perform at least a portion of the described process. Furthermore, the present invention relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program may be stored. However, the computer program may also be presented via a network such as the World Wide Web and downloaded from such a network into the random access memory of a data processor. Furthermore, the computer program may also be executed on a cloud-based processor, and the results may be presented via a network.

[0133] Please note that preliminary information regarding the implant (e.g., nail size and type) can be obtained before or during surgery by simply scanning any characters on the implant packaging (e.g., barcode) or the implant itself.

[0134] As should be clear from the above description, the main aspect of the present invention is the processing of X-ray image data, enabling the automatic interpretation of visible objects. The method described herein should be understood as a method to assist in the surgical treatment of patients. Consequently, according to one embodiment, the method does not involve any steps of treatment of an animal or human body by a surgeon.

[0135] It will be understood that the steps of the methods described herein, and in particular the steps of the methods described in relation to the workflow according to embodiments, some of which are visualized in the drawings, are major steps, and these major steps may be differentiated or divided into several substeps. Furthermore, additional substeps may be between these major steps. It will also be understood that only a portion of the entire method may constitute the present invention, i.e., steps may be omitted or aggregated.

[0136] It should be noted that the embodiments described relate to different subject matter. In particular, some embodiments describe method-type claims (computer programs), while others describe apparatus-type claims (systems / apparatus). However, those skilled in the art will infer from the foregoing and subsequent descriptions that, unless otherwise specified, combinations of features belonging to one type of subject matter, and any combinations of features relating to different subject matter, are disclosed in this application.

[0137] The embodiments, as well as further embodiments, features, and advantages of the present invention as defined above, can also be derived from the examples of embodiments described below and will be explained with reference to the examples of embodiments shown in the drawings, but the present invention is not limited thereto. [Brief explanation of the drawing]

[0138] [Figure 1] A lateral X-ray image of the femur is shown to determine the entry point for the intramedullary nail. [Figure 2] ML X-ray images of the proximal tibia and the mouth opening device are shown. [Figure 3]AP X-ray images of the proximal tibia and the mouth opening device are shown. [Figure 4] AP X-ray images of the proximal tibia and the mouth opening device are shown. [Figure 5] AP X-ray images of the proximal tibia and the mouth opening device are shown. [Figure 6] This shows image alignment of the tibia based on two AP X-ray images and one ML X-ray image. [Figure 7] This image shows an axial X-ray of the proximal portion of the humerus. [Figure 8] The images show axial X-ray images of the proximal humerus and the guide rod. [Figure 9] AP X-ray images of the proximal humerus and the guide rod are shown. [Figure 10] This shows the image alignment relative to the humerus based on AP X-ray images and axial X-ray images. [Figure 11] The images show 2D X-ray images of the proximal humerus, the anatomical neck, and the axial direction of the guide rod. [Figure 12] AP X-ray images of the proximal humerus, a 2D point on the anatomical neck, and the guide rod are shown. [Figure 13] AP radiographs of the proximal humerus, 2D projected anatomical neck, entry point, and guide rod are shown. [Figure 14] The images show AP X-ray images of the proximal humerus, the 2D projected anatomical neck, the entry point, and a guide rod with its tip positioned on the entry point. [Figure 15] The fractured 3D humerus and guide rod are shown from the AP line of sight. [Figure 16] The fractured 3D humerus and guide rod are shown from the axial line of sight. [Figure 17] The fractured 3D humerus and the inserted guide rod are shown from the AP line of sight. [Figure 18] The images show axial X-ray images of the proximal humerus, a 2D point on the anatomical neck, and the inserted guide rod. [Figure 19]The images show AP X-ray images of the proximal humerus, a 2D point on the anatomical neck, and the inserted guide rod. [Figure 20] The proximal portion of the femur, its contour, and AP X-ray images of the mouth opener are shown. [Figure 21] The proximal portion of the femur, its contour, and ML X-ray images of the mouth opening device are shown. [Figure 22] This shows an ML X-ray image of the distal femur. [Figure 23] ML X-ray images of the distal portion of the femur and its contour are shown. [Figure 24] This shows a 3D model of the femur and the definition of the femoral anteversion angle. [Figure 25] The C-arm is shown along with its axis of rotation and translation. [Figure 26] This illustrates a potential workflow for determining the entry point for the tibia. [Figure 27] This illustrates a potential workflow for determining the entry point for the humerus. [Figure 28] AP X-ray images of the distal femur, the inserted implant, and the drill placed on the surface of the femur are shown. [Figure 29] ML X-ray images of the distal femur, the inserted implant, and the drill placed on the surface of the femur are shown. [Figure 30] This shows image alignment of the distal femur based on AP and ML X-ray images. It includes the femur, the inserted implant, and the drill. [Figure 31] The same three-dimensional configuration as in Figure 30 is shown from a different viewing angle. [Figure 32] ML X-ray images of the distal femur are shown, along with calculated entry points for multiple nail holes. [Figure 33] This illustrates a potential workflow for determining the entry point for intramedullary femoral implants. [Figure 34] This illustrates a potential workflow for determining the femoral anteversion angle. [Figure 35]This demonstrates a potential workflow for a freehand, fixed procedure (quick version). [Figure 36] This demonstrates a potential workflow for a freehand fixing procedure (enhanced version). [Figure 37] This outlines a potential workflow for verifying and correcting drill trajectories. [Figure 38] This shows three different drill positions in 3D space. [Figure 39] Figure 38 shows the 2D projection of the scenario. [Modes for carrying out the invention]

[0139] Throughout the drawings, the same reference numerals and letters are used to indicate similar features, elements, components, or parts of the illustrated embodiments, unless otherwise specified. Furthermore, this disclosure is described in detail with reference to the drawings, but so in relation to the illustrated embodiments and is not limited by any particular embodiment illustrated in the drawings.

[0140] Determining the entry point for implanting an intramedullary nail into the femur. A first objective of the present invention may be the determination of the implantation curve and entry point for implanting an intramedullary nail into the femur. To determine the entry point, the X-ray image needs to be taken from a specific line of sight direction. In a true lateral view, the trunk axis and cervical axis are parallel with a constant offset. However, this line of sight is not the desired line of sight of the present invention. The desired line of sight is a lateral view having rotation around the C-axis of the C-arm such that the implantation axis passes through the center of the femoral head. The center of the femoral head can be determined with sufficiently high accuracy, for example, by a neural network. The uncertainty in determining the center of the femoral head may mainly relate to deviations in the direction of the implantation axis, which does not significantly affect the accuracy of ensuring the desired line of sight direction. The system can assist the user in obtaining the desired line of sight direction by estimating the required rotation angle around the C-axis based on an anatomical structure database or based on LU100907B1.

[0141] This system can also help the user obtain the correct line of sight. For example, consider a scenario where the 2D distance between the center of the femoral head and the tip of the mouth opener is too small compared to the 2D distance between the tip of the mouth opener and the lowest visible part of the femoral shaft. This effect occurs when the focal axis of the C-arm is almost perpendicular to the implantation axis. In this case, the center of the shaft at the isthmus would be almost invisible in the current X-ray projection image. Therefore, the system can instruct the C-arm to rotate around axis B, as shown in Figure 25. Following this instruction would result in an X-ray projection image in which the first distance increases and the second distance decreases (i.e., the neck becomes larger and the isthmus of the shaft becomes visible).

[0142] One way to determine the angle at which the C-arm needs to rotate to obtain the desired line of sight as described above is to consider the anatomical appearance in the AP radiograph. The following points may be identified in the image: the center of the femoral head, the tip of the mouth opener, and the center of the diaphysis in translation to the greater trochanter. Two lines may be drawn between the first two points and the latter two points, respectively. Since these three points can also be identified with sufficient accuracy in the ML radiograph, it may be possible to estimate the angle between the focal line in the ML radiograph and the anatomical structure (e.g., the implantation axis and / or cervical axis). If this angle is too small or too large, the system may provide instructions to increase or decrease that angle, respectively.

[0143] According to one embodiment, the implantation axis may be determined as follows. Figure 1 shows a transverse (ML) X-ray image of the femur. The system can detect the center of the diaphysis (labeled ISC) and the center of the femoral head (labeled CF) in the isthmus. The line defined by these two points may be considered the implantation axis (labeled IA). Furthermore, the system may detect the projected lateral boundary (labeled OB) of the cervical and diaphysis, or alternatively, multiple points on the boundary. Boundary segmentation may be performed, for example, by a neural network. Alternatively, the neural network may directly estimate specific points instead of the complete boundary. For example, instead of the diaphysis boundary, the neural network may estimate the center of the diaphysis, and the diaphysis diameter may be estimated based on the size of the femoral head. Based on this information, it may be possible to estimate the location of the diaphysis boundary without finding the boundary itself. The implantation axis should be at a certain distance from the cervical boundary and the diaphysis boundary. If either distance is too small, the system can calculate the necessary rotation of the C-arm around the C-axis so that the desired line of sight is achieved in the subsequently acquired X-ray projection image. The direction of rotation of the C-arm may be determined based on a weighted assessment of the distance within the neck and the distance within the diaphysis. The angle of rotation may be calculated based on an anatomical model of the femur.

[0144] Once the desired line of sight is achieved, the intersection of the implantation axis and the trochanteric margin axis can be defined as the entry point. The trochanteric margin axis can be detected directly in the image. If this is neither desirable nor feasible, the trochanteric margin axis can also be approximated in the X-ray image by a line connecting the tip of the aperture device to the implantation axis. This line can be assumed to be perpendicular to the implantation axis, or, if available prior information proposes otherwise, it extends obliquely to the implantation axis.

[0145] The implant may consist of a nail and a head element. If the distance between the projected tip of the mouth opener and the projected entry point is not within the desired distance (e.g., the distance is greater than 1 mm), the system may instruct the user on how to move the mouth opener to reach the entry point. For example, if the tip of the mouth opener on the femur is positioned too far forward compared to the determined entry point, the system may instruct the user to move the tip of the mouth opener backward.

[0146] According to one embodiment, the system can detect the isthmus of the femoral shaft, the center of the femoral head (labeled CF), and the tip of the mouth opener (labeled KW) in an X-ray image. The implantation axis (labeled IA) can be assumed to be a line passing through the center of the femoral head (labeled CF) and the center of the isthmus of the shaft (labeled ISC). The entry point can be assumed to be a point on the implantation axis (labeled EP) adjacent to the tip of the mouth opener KW. The system can provide instructions to move the mouth opener so that it is positioned on the EP. After moving the instrument to the projected point, it may be useful to acquire an AP image to verify that the tip of the mouth opener is still on the projected tip of the greater trochanter within the AP line of sight. If we have knowledge of the epipolar rays projected from the detected K-wire tip in the AP image, likely based on alignment of the AP image with the ML image, and if there is no movement of the k-wire tip between the acquisition of the AP image and the acquisition of the ML image, this would result in a more accurate determination of the entry point, without the need for additional verification in another AP image to see if the tip is still positioned on the projected tip of the greater trochanter.

[0147] A potential workflow example for determining the entry point for an intramedullary implant within the femur (see Figure 33): 1. The user obtains an AP X-ray image in which the tip of the mouth retractor is positioned on the projected tip of the greater trochanter. 2. The user obtains ML X-ray projection images without moving the tip of the mouth opener. 3. This system detects the center of the femoral head, the center point of the isthmus of the diaphysis, and the tip of the mouth opener within the X-ray image. a. If both the femoral head and the diaphysis are not clearly visible, the system will instruct the C-arm to move laterally to increase the field of view. b. If only the femoral head is not fully visible, while the isthmus is fully visible, the system instructs the C-arm to move proximal along the leg. c. The system calculates a first distance between the center of the femoral head and the tip of the mouth opener, and a second distance between the tip of the mouth opener and a point on the diaphysis. This point may be the center of the diaphysis at the isthmus (if visible), or, if the isthmus is not visible, the most distal visible point of the diaphysis, or alternatively, the estimated center of the diaphysis at the isthmus (based on the visible portion of the diaphysis). d. If only the diaphysis is not fully visible, while the femoral head is fully visible, the system instructs the C-arm to move distally along the leg. One way to determine whether the diaphysis is fully visible may be to compare the second distance from step 3c to a threshold. Another method may be to assess the curvature of the diaphysis to determine whether the isthmus is visible in the current X-ray image. e. If the first distance from step 3c is too small compared to the second distance, the C-arm must rotate clockwise (right femur) or counterclockwise (left femur) around the C-arm axis B (see Figure 25), and vice versa. The angle by which the C-arm must rotate can be calculated based on the two distances and possibly additional information from the AP image from step 1. The latter may include, for example, the CCD angle of the femur. Curvature of the diaphysis, as shown in the ML X-ray image, may also be taken into consideration. 4. Steps 2 and 3 are repeated until all important parts of the femur are clearly visible and the two distances from step 3c are in the desired ratio. 5. In addition to the points from step 3, this system detects the left and right contours of the femoral neck and the left and right contours of the femoral shaft. 6. A line is drawn from the center of the femoral neck to the center of the isthmus of the femoral shaft. Four distances are calculated between this line and the four contours of the femoral neck and femoral shaft. 7. For each of the cervical and diaphyseal regions, a measurement criterion is defined to evaluate how closely the line passes through the center of each region. For example, if the line touches the left contour of the neck, the measurement criterion for the neck is 0; if the line touches the right contour of the femur, it is 1; and if the line is located in the center of the neck, it is 0.5. 8. The new measurement criterion is defined based on the weighted average of the cervical and diaphysis measurement criterions. If the new measurement criterion is lower than the first threshold, the C-arm must be rotated around its C-axis so that the focal point of the C-arm moves forward. If the new measurement criterion is higher than the second threshold, which is higher than the first threshold, the C-arm must be rotated around its C-axis in the opposite direction. The angle by which the C-arm must be rotated can be calculated based on the distance between the measurement criterion and the corresponding threshold. 9. If the measurement criteria defined in step 8 are outside the two thresholds from step 8, a new ML X-ray projection image should be acquired. 10. Steps 5-9 are repeated until the metric defined in step 8 falls between the two thresholds from step 8. The drawn line represents the final projected implantation axis. 11. The distance between the projected tip of the opening device and the line from step 10 is calculated. 12. Optional: The position of the tip of the mouth opener is determined. Based on the appearance of the tip of the mouth opener (i.e., its size in the X-ray projection image), the system provides instructions to move the tip of the mouth opener either backward or forward. 13. If the tip of the mouth retractor is far from the line from step 10, its position is optimized and a new ML X-ray projection image is obtained. 14. Steps 11-13 are repeated until the tip of the opening tool is within a certain distance from the line from step 10. 15. An AP X-ray projection image is taken to ensure that the tip of the mouth opener is still over the tip of the greater trochanter. If not, return to step 2.

[0148] Procedure for implanting a nail with a sub-implant into the tibia

[0149] Examples of potential workflows (see Figure 26): 0. For the following workflow, it is assumed that the proximal part of the neck is intact (or properly reduced). 1. The user positions the mouth opener on the surface of the neck (at any point proximal to the neck, but ideally near the entry point estimated by the surgeon). 2. The user obtains (approximately) lateral images of the proximal part of the neck (labeled TIB) as shown in Figure 2. 3. The user obtains at least one AP image (ideally multiple images from slightly different directions) of the proximal tibia as shown in Figures 3, 4, and 5. 4. This system detects the size (or diameter, etc.) of the mouth opening instrument (labeled as OI) in all images in order to estimate the size (scaling) of the tibia. 5. This system fits a statistical model of the tibia to all images, for example, by matching the statistical model to the bone contour (or, more generally, the appearance of the bone). The result of this step is a 3D reconstruction of the tibia. a. This includes six parameters per image for rotation and translation, one parameter for scaling (already estimated in step 4), and a fixed number of modes (determining the modes is equivalent to a 3D reconstruction of the tibia). Thus, if there are n images and m modes, the total number of parameters is (6·n+1+m). b. Based on all estimated rotations and translations of the tibia (within each image), the system performs image alignment for all images, as shown in Figure 6. Thus, the spatial relationships between the AP images (labeled I.AP1 and I.AP2), ML images (labeled I.ML), the tip of the mouth opener (labeled OI), and the tibia (labeled TIB) can be determined. c. Optional: For potentially more accurate results, the system may use information from the femoral condyle or fibula, for example, by using statistical information for these bones. 6. Based on the 3D reconstruction of the tibia, the system determines the entry point. This can be done, for example, by defining the entry point on the average shape of a statistical model. This point can then be identified on the 3D reconstruction. 7. Optional: Based on a 3D reconstruction of the tibia, the system (virtually) positions the implant within the bone and calculates the length of the proximal fixation screw. This step also takes the actual implant into account, which may improve the estimation of the entry point. 8. This system displays the entry point as an overlay within the current X-ray image. 9. If the tip of the opening device is not close enough to the estimated entry point, the system will instruct the system to correct the tip's position. a. The user corrects the position of the tip of the mouth opener and obtains a new X-ray image. b. The system calculates the entry point in the new image (for example, by image difference analysis or by fitting the 3D reconstruction of the tibia into the new image). c. Return to Step 8. 10. The user inserts the implant into the tibia and obtains a new image. 11. This system determines the position of the implant. Based on a 3D reconstruction of the tibia, this system provides the necessary 3D information (e.g., the length of the proximal fixation screw). 12. This system provides support for proximal fixation. 13. This system calculates the torsion angle by comparing the proximal portion of the tibia (which may include the femoral condyle) and the distal portion of the tibia (which may include the foot). For a more accurate calculation of the torsion angle, this system may also use information about the fibula (for example, by determining the position of the fibula and calculating its spatial relationship to the tibia).

[0150] Procedure for implanting a nail with a sub-implant into the humerus

[0151] Example of a potential workflow (see Figure 27): 0. The user provides the desired distance between the entry point and the anatomical neck (e.g., 0 mm or 5 mm inward). 1. The user obtains an axial X-ray image of the proximal portion of the humerus, as shown in Figure 7. 2. The system detects the contour of the humeral head (e.g., using a neural network). Based on the detected contour, the system approximates the humeral head with a circle (labeled HH), i.e., it estimates the 2D center and radius. This may include multiple candidates for the humeral head (2D center and radius), which are ranked based on their plausibility (e.g., based on a statistical model, mean square approximation error, confidence level, etc.). Based on the detected trunk axis (labeled IC), the system rotates the image so that the trunk axis is a vertical line. The system evaluates whether the center of the head is close enough to the trunk axis. If the distance between the center of the head and the trunk axis is too large, the system advises the user to apply distal traction force to the arm to compensate for translational reduction (i.e., head vs. diaphysis, the force from the soft tissue will result in reduction perpendicular to the traction force). 3. This system estimates the initial entry point (labeled EP), which is located somewhere between the intersection of the humeral head and the trunk axis (for example, 20% above the center of the intersection). 4. The user places the guide rod on the initial guess of the entry point from step 3. 5. The user obtains further axial X-ray images in which the guide rod (labeled OI) is visible, as shown in Figure 8. 6. The system detects the humeral head (labeled HH) (2D center and radius) and the 2D trunk axis (labeled IC) to determine the position of the guide rod (labeled OI) and obtain the 2D coordinates and 2D scaling of its tip (based on the known diameter of the guide rod). 7. The system advises the user to rotate the C-arm around its C-axis (furthermore, permitted C-arm movements are translations in the distal-proximal or anterior-posterior directions; prohibited movements are other rotations and translations in the inward-outward direction). 8. The user obtains an AP X-ray image of the proximal humerus (it does not need to be a true AP image) as shown in Figure 9, while keeping the tip of the guide rod still (angular movement of the guide rod is permitted as long as the tip remains in place). 9. The system detects the humeral head (labeled HH) (2D center and radius) and the 2D trunk axis (labeled IC) to determine the position of the guide rod (labeled OI) and obtain the 2D coordinates and 2D scaling of its tip (based on the known diameter of the guide rod). 10. Based on the information from steps 6-9, the system performs image alignment as shown in Figure 10 to calculate the spherical approximation of the humeral head (labeled HH) and the 3D trunk axis which is in the same coordinate system as the sphere. 11. There are four points (i.e., two per image, axial and AP) that define the start and end of the circular portion of the projected humeral head (labeled CA in Figures 11 and 12). The system detects at least three of these four points. Based on these at least three points, the system determines the anatomical neck in 3D (for example, by defining a plane based on the three points that intersects the spherical approximation of the humeral head). 12. The system may also use the fourth point from step 11 to improve the determination of the anatomical neck (for example, using the weighted least squares method, in which case the weights are based on the individual confidence level of each of the four points). 13. When the anatomical structure is virtually rotated in space such that the 3D trunk axis is a vertical line and the humeral head is above its shaft, the entry point is defined as the highest point in space on the anatomical neck (labeled CA3D in Figure 13). Based on the settings from Step 0 and the results from Steps 10-12, the system calculates the final entry point (labeled EP). 14. The user positions the guide rod over the calculated entry point to obtain a new AP X-ray image as shown in Figure 14. 15. This system detects the tip of the guide rod (labeled OI) and evaluates whether the tip of the guide rod is positioned sufficiently close to the calculated entry point (labeled EP). 16. Steps 14 and 15 are repeated until the tip of the guide rod is close enough to the entry point. 17. Optional instruction for the angular movement of the guide rod. a. Based on the most recent image alignment (which includes the humeral head in 3D), the system determines the spatial relationship between the humeral head and the guide rod, as shown in Figures 15 and 16. If the orientation of the guide rod deviates too far from the optimal insertion direction, the system provides instructions for angular movement of the guide rod. The optimal insertion direction can be estimated, for example, using a statistical model or by comparing the axis of the guide rod (labeled OIA) with the axis of the humeral head (labeled HA). b. If instructions were given in step a, the user takes a new X-ray image from the same direction according to those instructions. Image difference analysis detects changes in the image and updates the image alignment. c. Steps a and b are repeated until no further angular movement of the guide rod is required. 18. Improvement of the ability to arbitrarily select image alignment and validation of the humeral head contour. a. The user inserts the guide rod as shown in Figure 17. b. The user acquires an X-ray image (for example, in the axial direction as shown in Figure 18). c. This system determines the position of the guide rod (labeled OI) and detects the humeral head (labeled HH) (2D center and radius). d. The system advises the user to rotate the C-arm around its C-axis (see step 7 for additional possible C-arm movements). e. The user acquires X-ray images from another direction (e.g., AP as shown in Figure 19) without moving the guide rod. f. This system determines the position of the guide rod (labeled OI) and detects the humeral head (labeled HH) (2D center and radius). g. Based on the information from both images, the system performs image alignment. Since the 3D model of the guide rod is known, the image alignment is more accurate than in step 10. Based on image alignment, this system can verify the detection of the humeral head in both images. i. Based on the validation results, the system optimizes the contour of the humeral head in both images (for example, by selecting a different candidate for the humeral head). 19. Optional correction of rotational displacement of the humeral head. a. The user acquires an X-ray image (axial or AP). The system determines the position of the guide rod and detects the 2D trunk axis and the 2D humeral head axis (defined by the visible circular portion of the humeral head). b. If previous images had significantly different imaging directions (e.g., axial in previous images and AP in current images), the system performs image alignment based on the most recent pair of images. Based on the image alignment, the system determines the ideal 2D angle between the stem axis and the head axis with respect to the current image. c. If previous images had very similar imaging orientations (e.g., identified by difference analysis), the ideal 2D angle between the stem axis and the head axis remains unchanged (compared to the previous images). d. This system calculates the current 2D angle between the trunk axis and the head axis. e. If the angle between the trunk axis and the head axis is not close enough to the ideal angle from step 19b or 19c (e.g., 20° for the axial image or 130° for the AP image), the system will provide instructions to correct rotational displacement in the dorsoventricular (axial image) or medial-lateral (AP image) direction. f. If previous images had very similar imaging orientations, but the visible circular portion of the humeral head is smaller or larger compared to the previous images (for example, due to previous correction of displacement), the rotational displacement may have changed for other imaging orientations as well. Therefore, the system provides additional instructions to rotate the C-arm around its C-axis to change the imaging orientation for the next image (i.e., to update the image alignment). g. If instructed, the user corrects the rotational displacement (and rotates the C-arm as necessary) and returns to step 19a. 20. Check for torsion of arbitrary choice. a. The user positions their forearm so that it is parallel to their body (or upper leg). b. The user acquires an axial X-ray image. c. This system detects the 2D centers of the humeral head and glenoid fossa. This system calculates the distance between the center of the glenoid fossa and the head axis. Based on this result, this system indicates the direction and angle in which torsion correction is necessary. d. The user corrects the twist by rotating their head in that direction from step c by that angle. e. Steps 20b to 20d are repeated until the center of the glenoid fossa is sufficiently close to the axis of the humeral head.

[0152] Potential Correction: Instead of estimating the entry point 20% (inward) from the center of the intersection in step 3, the system may use a higher value (e.g., 70%) to ensure that the tip of the guide rod is positioned over the spherical portion of the humeral head. In step 10, the system may use the information that the tip of the guide rod is positioned over the spherical approximation of the humeral head to improve image registration. Due to the aforementioned 70% method, the current position of the tip of the guide rod is further from the entry point (compared to the 20% method). As the user guides the tip of the guide rod to reach the entry point (steps 14-16), the system determines (e.g., by image difference analysis) whether the line of sight has changed. If the line of sight has not changed, the calculated entry point is used from the previous X-ray image, and the guidance information is updated based on the updated detected position of the tip. If the line of sight changes only slightly, the entry point is shifted accordingly (for example, by a technique called object tracking; see, for example, "A survey on moving object tracking using IMAGE processing" by SRBalaji et al. (2017)). If the line of sight changes significantly, the system instructs the user to rotate the C-arm around its C-axis and acquire X-ray images from a different line of sight (for example, the axial direction if the current image was AP) while keeping the tip of the guide rod still. Based on the updated image, the system performs image alignment based on information acquired by the previous alignment (for example, the radius of the spherical approximation of the humeral head) to display the entry point in the current image and navigate the user to reach the entry point with the tip of the guide rod.

[0153] Determination of femoral anteversion angle

[0154] Below, an example workflow for determining the AV angle, either before or after implant insertion, is presented, which may be more robust and / or accurate than the latest technology. According to one embodiment, the entire procedure for determining the femoral anteversion angle may proceed as follows (see Figure 34). 1. The user places the tip of the mouth opener approximately on the tip of the greater trochanter. 2. The user obtains an AP X-ray image of the proximal femur as shown in Figure 20. 3. This system detects the 2D contours of the femur (labeled FEM) and femoral head, which are approximated by a circle (labeled FH) (i.e., it is determined by its 2D center and 2D radius) to determine the position of the tip of the mouth opener (labeled OI). 4. If several important parts of the femur or the tip of the mouth opener are not sufficiently visible, the system will instruct the user to rotate and / or move the C-arm and return to step 2. 5. The user rotates the C-arm around its C-axis to acquire ML X-ray images. The user may also use inward-outward and / or anterior-posterior shifts of the C-arm. The tip of the mouthpiece should not be moved while the C-arm is being moved. 6. The user obtains an ML X-ray image of the proximal femur as shown in Figure 21. 7. The system detects the 2D contours (i.e., 2D center and 2D radius) of the femur (labeled FEM) and femoral head (labeled FH) to determine the position of the tip of the mouth opener (labeled OI). 8. If several important parts of the femur or the tip of the mouth opener are not sufficiently visible, the system will instruct the user to move the C-arm (translation only) or rotate the C-arm around its C-axis, and the user will return to step 6. 9. Based on the proximal AP and ML image pairs, the system performs image alignment. If image alignment is unsuccessful, the system instructs the user to rotate and / or move the C-arm, and the user returns to step 2. 10. The user moves the C-arm distally along the patient's leg. Rotational movement of the C-arm is not permitted in this step, but three parallel movements are allowed. 11. The user obtains ML X-ray projection images of the distal femur as shown in Figures 22 and 23. 12. This system detects the 2D contour of the femur (labeled as FEM). 13. No specific orientation or alignment of the femoral condyle is required. However, if some important parts of the femur are not sufficiently visible, the system will instruct the user to move the C-arm (only translation is permitted) and return to step 11. 14. Based on image registration, the system fits the statistical model (trained on fractured and unfractured femurs) to all images together so that the projected contour of the statistical model matches the detected 2D contour of the femur in all images. This step directly results in a 3D reconstruction of the femur. To improve the accuracy of the 3D reconstruction, the system calculates the 3D position of the tip of the mouth opener (based on proximal image registration) and may use this point as a reference point, taking into account the fact that the tip of the mouth opener is placed on the surface of the femur. 15. This system determines the femoral anteversion angle based on a 3D reconstruction of the femur as shown in Figure 24. According to "3D femoral neck anteversion measurements based on the posterior femoral plane in ORTHODOC® system" by Yeon Soo Lee et al. (2006), the femoral anteversion angle can be calculated based on the center of the femoral head (labeled FHC), the center of the femoral neck (labeled FNC), the posterior apex of the trochanter (labeled TRO), and the lateral and medial apex of the posterior femoral condyle (labeled LC and MC). This system identifies these five points on the 3D reconstruction of the femur from step 10 and thus calculates the femoral anteversion angle.

[0155] Freehand fixing procedure There may be different embodiments of the distal fixation procedure for femoral nailing. Two examples of potential workflows are presented below: one "quick" and one with "enhanced" precision. In both workflows, the user can verify the drilling trajectory at any time during drilling, based on radiographic images with near real-time (NRT) feedback, and correct the drilling angle if necessary. This verification does not require rotation or readjustment of the C-arm. An example workflow for such verification is provided below.

[0156] For an example of a potential workflow (quick version), see Figure 35: 1. The user obtains an X-ray image of the distal femur (e.g., AP or ML as shown in Figure 28). 2. The system locates the implant and detects the contour of the femur. If the location of either the implant or the femur contour cannot be determined, the system instructs the user to increase visibility (for example, by moving the C-arm). The user follows the instructions and returns to step 1. 3. The user places the drill on the surface of the femur (e.g., along the nail hole trajectory). The user acquires an X-ray image from a different line of sight (e.g., 25°-ML as shown in Figure 29). 4. This system determines the location of the implant (labeled IM), detects the contour of the femur (labeled FEM), and determines the location of the drill (labeled DR). 5. If the position of the drill tip cannot be determined, the system will instruct the user to improve the visibility of the drill tip (for example, by moving the C-arm). The user follows the instructions to obtain a new image and returns to step 4. 6. Based on the location of the implant in both images (labeled I.AP and I.ML in Figure 30), the system performs image registration as shown in Figures 30 and 31. 7. Based on the image alignment from step 6, the system fits a statistical model of the femur by matching its projected contour to the detected contour of the femur in both images (i.e., it determines the rotation and translation, scaling, and mode of the statistical model of the femur in both images). 8. With respect to the current image, the system defines a line from the tip of the drill to the focal point in the image plane. This line intersects the reconstructed femur twice (i.e., the entry and exit points). The point closer to the focal point is selected as the current 3D position of the drill tip. The system can calculate the length of the fixation screw based on the diameter of the diaphysis along the nail hole trajectory of the reconstructed femur. 9. Based on the known spatial relationship between the femur and the implant (resulting from image alignment and femoral reconstruction), the system calculates the spatial relationship between the drill and the implant. 10. When the drill trajectory passes through the nail hole, the system instructs the user to begin drilling, and the user begins drilling and proceeds to step 14. At any point during the drilling process, the user can verify the drilling trajectory according to the following workflow example. 11. If the drill trajectory does not pass through the nail hole, the system will provide instructions to move the drill tip and / or rotate the drill. The user will follow these instructions to obtain a new X-ray image. 12. The system evaluates whether the line of sight has changed (e.g., by image difference analysis). If the line of sight has not changed, the system can use most of the results from previous images to determine the drill position. If the line of sight or any other relevant image content has changed (e.g., due to the effects of image blur, occlusion, etc.), the system uses this information to improve image alignment (e.g., by using additional line of sight from the current image). The system determines the positions of the implant and drill, detects the contour of the femur, and fits the reconstructed femur to the current image. 13. The user returns to step 9. 14. If the user wishes to fix additional holes, the system displays entry points for all nail holes (given by the intersection of the 3D reconstruction of the femur and the implantation curve for the ideal fixation position) and provides instructions on how to move the drill tip to reach those entry points. An example is shown in Figure 32. The user places the drill tip over the calculated entry point (labeled EP) and returns to step 12.

[0157] For an example of a potential workflow (enhanced version), see Figure 36: 1. Optional: The user obtains an X-ray image of the distal femur (e.g., AP or ML as shown in Figure 28). The system locates the implant (labeled IM) and detects the contour of the femur (labeled FEM). If the location of either the implant or the femoral contour cannot be determined, the system instructs the user to increase visibility (e.g., by moving the C-arm). The user follows the instructions and returns to the beginning of this step. 2. The user places the drill on the surface of the femur (for example, along the nail hole trajectory). 3. The user obtains an X-ray image of the distal femur (e.g., ML or AP). The system locates the implant (labeled IM), detects the contour of the femur (labeled FEM), and locates the drill (labeled DR). If the location of the implant, the contour of the femur, or the tip of the drill cannot be determined, the system instructs the user to improve visibility (e.g., by moving the C-arm). The user follows the instructions and returns to the beginning of this step. Based on the 3D reconstruction of the bone relative to the nail coordinate system, the system calculates the required length of the sub-implant (e.g., the fixing screw) and displays it according to the information. 4. The user acquires an X-ray image from a different line of sight (for example, 25°-ML as shown in Figure 29). The drill tip must not move between images. If it moves, the system may be able to detect this and prompt the user to return to step 3. 5. This system locates the implant (labeled IM), detects the contour of the femur (labeled FEM), and locates the drill (labeled DR). 6. If the position of the drill tip cannot be determined, the system will instruct the user to improve the visibility of the drill tip (for example, by moving the C-arm). The user follows the instructions to obtain a new image and return to step 5. 7. Based on the location of the implant in at least two images (labeled I.AP and I.ML in Figure 30), the system performs image registration as shown in Figures 30 and 31. 8. Based on the image registration from step 7, and possibly using information from previous image registrations, the system fits a statistical model of the femur by matching its projected contour to the detected contour of the femur in the image (i.e., it determines the rotation and translation, scaling, and mode of the statistical model of the femur in both images). Optional: The system may update the calculated sub-implant length based on the reconstructed bone and determined nail hole trajectory. 9. With respect to the current image, the system defines a line L1 (labeled L1 in Figure 31) in the image plane from the tip of the drill to the focal point. L1 intersects the reconstructed femur twice (i.e., the entry and exit points). The point closer to the focal point is selected as the current 3D position of the drill tip. 10. For images from other line-of-sight directions that include the drill tip, the system defines a line L2 from the drill tip to the focal point in the image plane (i.e., within the corresponding coordinate system of the image). Based on image alignment, this line is transformed into the coordinate system of the current image. The transformed line is called L2' (labeled L2' in Figure 31). 11. If the shortest distance between L1 and L2' is greater than a certain threshold, the system may advise the user to return to step 4, as there is a very high probability that the drill tip has moved between images. Optional: If the user confirms that the drill tip has not moved between the generation of the image pairs used for image alignment, the system will improve image alignment by optimizing implant positioning in both images to minimize the distance between L1 and L2' (L1 and L2' will intersect if implant and drill tip positioning is complete in both images and the drill tip has not moved between images). 12. The point on L1 that has the shortest distance to L2' is selected as an additional initial value for the current 3D position of the drill tip. 13. Based on the two solutions for the 3D position of the drill tip (i.e., from steps 9 and 12), the system finds the current 3D position of the drill tip (e.g., by selecting the solution from step 12, or by averaging both solutions). Since the drill tip is on the surface of the femur, the system improves the 3D reconstruction of the femur, under the constraint that the estimated 3D position of the drill tip is on the surface of the reconstructed femur. The system may validate the previously calculated subimplant length based on the improved reconstruction of the femur. If the updated length deviates from the previously calculated screw length (presumably taking into account the available length increment of the subimplant), the system notifies the user. 14. Based on the known spatial relationship between the femur and the implant (resulting from image alignment and femoral reconstruction), the system calculates the spatial relationship between the drill and the implant. 15. When the drill trajectory passes through the nail hole, the system instructs the user to begin drilling, and the user begins drilling, inserts the sub-implant after drilling, and then proceeds to step 19. At any point during the drilling process, the user may verify the drilling trajectory according to the following workflow example. 16. If the drill trajectory does not pass through the nail hole, the system will provide instructions to move the drill tip and / or rotate the drill. The user will follow these instructions to obtain a new X-ray image. 17. The system evaluates whether the line of sight has changed (e.g., by image difference analysis). If the line of sight has not changed, the system can use most of the results from previous images to determine the drill position. If the line of sight or any other relevant image content has changed (e.g., due to the effects of image blur, occlusion, etc.), the system uses this information to improve image alignment (e.g., by using additional line of sight from the current image). The system determines the positions of implants and drills, optimized by determining the positions of already inserted sub-implants by taking into account available information about their entry points where possible, and detects the contour of the femur to fit the reconstructed femur to the current image. 18. The user returns to step 14. 19. If the user wishes to fix additional holes, the system displays entry points for all nail holes (given by the intersection of the 3D reconstruction of the femur and the implantation curve for the ideal fixation position) and provides instructions on how to move the drill tip to reach those entry points. An example is shown in Figure 32. The user places the drill tip over the calculated entry point (labeled EP) and returns to step 17.

[0158] At any time, if the user decides to check whether the holes are successfully secured, the user may acquire images in an imaging direction that is less than 8 degrees off the secured hole trajectory, and the system will automatically evaluate whether the securing is successful. If the last hole is secured, or if the system has information that requires verification of the secured procedure performed, the system may guide the user to reach the C-arm position relative to the secured hole trajectory.

[0159] To support performing skin incisions at the correct spot to position the drill on the proposed entry point, this system can project skin entry points and bone entry points based on the implantation curve by estimating the distance between the skin and bone.

[0160] See Figure 37 for an example of a potential workflow for verifying and correcting drill trajectories: 1. The user acquires an X-ray image from the current imaging direction. 2. The system aligns the drill and the nail, that is, it determines their relative 3D position and orientation based on acquired X-ray images. The ambiguity of 2D-3D matching can be resolved by taking into account prior information that the drill axis passes through an entry point (i.e., the starting point of drilling) whose 3D coordinates relative to the nail are previously determined within the workflow of Figure 35 or Figure 36. Further explanation of this is provided below. 3. If the current drill position and orientation relative to the nail indicate that the drill may miss the fixing hole if it continues along its current path, the system instructs the user to tilt the power tool, using the rotary drill bit, by a specified angle. By doing so, the drill bit widens its path through the cancellous bone and, as a result, returns to the correct trajectory. The angle provided in the instruction may take into account that the drill may bend inward into the bone when following the instruction, and the amount of bending may depend on the depth of drill insertion, bone density, as well as the stiffness and diameter of the drill. 4. The user can return to step 1 or resume drilling. This loop of steps 1-4 can be performed continuously with near real-time navigation guidance.

[0161] The resolution of the 2D-3D matching ambiguity in Step 2 is illustrated in Figures 38 and 39. Figure 38 shows three different drill positions (labeled DR1, DR2, and DR3) in 3D space, all of which correspond to the same 2D projection DRP in Figure 39. However, any ambiguity regarding the 3D position and orientation of the drill relative to the nail N can be resolved by taking into account the prior information that the drill axis passes through the entry point EP.

[0162] Note that as soon as the drill approaches the nail, the image acquired in step 1 no longer allows for resolving the ambiguity of the 2D-3D matching because the tip of the drill overlaps with the nail in the X-ray image. In this case, a possible solution may be to acquire additional X-ray images showing the tip of the drill (and the nail) from different imaging directions. In the additional X-ray images, the nail can also be located, and thus the additional X-ray images can be aligned with the original X-ray images. The tip of the drill can be detected in the additional X-ray images. The point defined by the tip of the drill detected in the additional X-ray images defines the epipolar line. The axis of the tool can be detected in the original X-ray image, defining the epipolar plane. The intersection of the epipolar plane and the epipolar line defines the position of the tip in 3D space relative to the nail.

Claims

1. A device having a processing apparatus configured to process X-ray images, wherein a software program, A step of receiving a first X-ray image which is a projection image of an object, wherein the object is at least partially visible in the first X-ray image, The steps include receiving a virtual representation of the object and applying the virtual representation to determine the position of the object in the first X-ray image, A step of identifying a first point visible in the first X-ray image, wherein the 3D position of the first point in 3D space relative to the object is known. A step of receiving a second X-ray image which is a projection image of a tool and the object, wherein the tool is at least partially visible in the second X-ray image and the object is at least partially visible in the second X-ray image, The steps include applying the virtual representation of the object to determine the position of the object in the second X-ray image, The steps include defining a coordinate system that has a fixed relationship with the aforementioned tool, The steps include determining and identifying known axes within the aforementioned coordinate system, The 3D position and orientation of the tool relative to the object are as follows: (i) The second X-ray image, (ii) 3D model of the tool, (iii) The object whose position has been determined, (iv) The knowledge that the 3D position of the first point relative to the object is the same when generating the first X-ray image and when generating the second X-ray image, and (v) Information regarding the distance between the first point and the axis Steps to decide based on and An apparatus that is performed by the processing apparatus in order to do so.

2. The apparatus according to claim 1, wherein the object includes a hole-filled implant, and the 3D position of the first point relative to the object is determined based on the axis of the hole in the implant.

3. The apparatus according to claim 1, wherein the tool is at least partially visible in the first X-ray image, the identified first point is a point on the tool, and the tool moves relative to the object between the generation of the first X-ray image and the generation of the second X-ray image.

4. The apparatus according to claim 1, wherein the 3D position of the first point with respect to the object is determined based on i) knowledge of the position of the bone surface, and ii) knowledge that the first point is located on the bone surface.

5. The apparatus according to claim 1, wherein the 3D position of the first point with respect to the object is determined based on further X-ray images from another line of sight direction.

6. The apparatus according to any one of claims 3 to 5, wherein the 3D position of the first point with respect to the object is determined based on the determination of the 3D position and orientation of the tool with respect to the object based on the second X-ray image.

7. The apparatus according to claim 1, wherein the tip of the tool is visible in the second X-ray image, and the 3D position and orientation of the tool with respect to the object is further determined based on the tip of the tool defining the second point.

8. The apparatus according to claim 1, wherein the tool is a drill that rotates during the generation of at least one of the X-ray images.

9. The apparatus according to claim 1, wherein the portion of the tool that is visible in the aforementioned X-ray image is rotationally symmetric.

10. The apparatus according to claim 1, wherein the portion of the tool that extends into the second X-ray image is partially shielded.

11. The apparatus according to any one of claims 8 to 10, wherein the software program is executed by the processing apparatus to perform a further step of receiving a third X-ray image generated from another line of sight, and the determination of the 3D position and orientation of the tool relative to the object is further based on the third X-ray image.

12. The apparatus according to claim 1, wherein the tool is a drill, and the software program is performed by the processing apparatus to perform a further step of providing instructions to a user, taking into account at least one of the embodiments from the group consisting of the depth of a hole already drilled, the density of the object, the diameter of the drill, and the rigidity of the drill.

13. The aforementioned software program, A further step of receiving a series of X-ray images, each of which is a projection image of the at least partially visible object and the at least visible tool, A further step of providing a continuous quasi-real-time determination of the 3D position and orientation of the tool relative to the object, during which the tool is moving relative to the object, and In order to do so, the processing apparatus performs the following: The apparatus according to claim 1.

14. A method to support musculoskeletal surgery, A step of receiving a first X-ray image which is a projection image of an object, wherein the object is at least partially visible in the first X-ray image, The steps include receiving a virtual representation of the object and applying the virtual representation to determine the position of the object in the first X-ray image, A step of identifying a first point visible in the first X-ray image, wherein the 3D position of the first point in 3D space relative to the object is known. A step of receiving a second X-ray image which is a projection image of a tool and the object, wherein the tool is at least partially visible in the second X-ray image and the object is at least partially visible in the second X-ray image, The steps include applying the virtual representation of the object to determine the position of the object in the second X-ray image, The steps include defining a coordinate system that has a fixed relationship with the aforementioned tool, The steps include determining and identifying known axes within the aforementioned coordinate system, The 3D position and orientation of the tool relative to the object are as follows: (vi) The second X-ray image, (vii) 3D model of the tool mentioned above, (viiii) The object whose position has been determined, (iX) Knowledge that the 3D position of the first point relative to the object is the same when generating the first X-ray image and when generating the second X-ray image, (X) Information regarding the distance between the first point and the axis Steps to decide based on and Methods that include...

15. The aforementioned method, A step of receiving a series of X-ray images, each of which is a projection image of the at least partially visible object and the at least visible tool, A step of providing near real-time continuous 3D alignment of the tool with respect to the object, during which the tool is moving relative to the object; Further including, The method according to claim 14.

Images