Determining relative 3D positions and orientations between objects in 2d medical images

Abstract

To provide an apparatus, method, and computer program for determining a 3D representation as well as relative 3D position and relative 3D orientation between objects based on an X-ray projection image.SOLUTION: An apparatus receives an X-ray image that is a projection image of a first object and a second object, classifies the first object and the second object, and receives a respective 3D model of the objects. At the first object, a geometrical aspect such as an axis or a line is determined, and at the second object, another geometrical aspect such as a point is determined. Finally, based on the 3D model of the first object, the 3D model of the second object, and information that the point of the second object is located on the geometrical aspect of the first object, a spatial relation between the first object and the second object is determined.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to the fields of artificial intelligence and computer-assisted surgery. In particular, the present invention relates to an apparatus and method for determining 3D representations and relative 3D positions and orientations between objects based on X-ray projection images. The method may be implemented as a computer program executable on a processing unit of the apparatus. [Background technology]

[0002] In orthopedic or orthopedic trauma surgery or spinal surgery, it is a common task to aim a relatively thin instrument at a target object or target structure (as part of the target object). The target structure may be an anatomical structure (e.g., a pedicle) or part of another instrument or implant (e.g., the distal fixation hole of a long antegrade intramedullary nail). Generally, the goal may be to determine the relative 3D position and relative 3D orientation between the instrument and the target object. Based on available intraoperative 2D imaging techniques, this can be challenging. This is particularly difficult when the exact geometry of the target object is unknown and / or when the instrument is known but not uniquely locatable in 3D space based on 2D X-ray images.

[0003] Preoperative CT scans may be performed for surgical procedures, allowing for more precise planning of the procedure. This is the case, for example, when operating within complex 3D structures or when drilling or placing screws within tight anatomical structures or near critical structures (e.g., spinal cord, nerves, aorta). A typical example of such a procedure is the placement of a sacroiliac joint or pedicle screw. When the target structure is a device or implant, its 3D geometry is typically known. One example is a distal fixation procedure, where a 3D model of or 3D information about the target object (the nail) and, in particular, the target structure "distal fixation hole" (cylinder) is available.

[0004] However, a high level of spatial awareness and imagination is required for the surgeon to utilize this 3D information and apply it to intraoperative 2D X-ray images.

[0005] In some cases, for some procedures, it may be possible to determine the drilling direction, for example, by aligning the drill with a particular viewing direction of the imaging device (e.g., a true medial-lateral view for distal fixation procedures). Furthermore, it is generally not possible to guarantee that the drill will proceed exactly along this direction in practice. This will now be illustrated by the example of distal fixation of a long antegrade intramedullary nail.

[0006] In a conventional distal fixation procedure for antegrade nailing, the surgeon moves the C-arm to a true lateral position, which means that the fixation hole appears perfectly circular on the X-ray image. This positioning is repetitive, tedious, and time-consuming, sometimes requiring several minutes and typically requiring the acquisition of 5 to 20 X-ray images with corresponding readjustments of the C-arm. A faster way to achieve this positioning is to use the C-arm's fluoroscopy mode (which generates a continuous X-ray video stream), but this results in a higher X-ray dose.

[0007] Furthermore, to ensure high accuracy for distal fixation, the hole not only must appear circular, but it must also be near the center of the X-ray image. In practice, however, if the hole appears sufficiently circular on the X-ray image, this C-arm position is typically used for distal fixation, even if the hole is not near the center of the X-ray image. Due to the conical shape of the X-ray beam fan, the more the X-ray beam direction is tilted, the further the beam moves away from the center of the X-ray image. Therefore, drilling through the hole must be in the direction of the X-ray source focal point and not parallel to the centerline between the X-ray source and detector.

[0008] In the next step, the drill tip can be positioned at the intended drill location, and an x-ray image is acquired. Here, the drill may be intentionally held at an oblique angle (i.e., not in the direction of the fixation track) so that the power drill and the surgeon's hands do not block the view. The goal is to position the drill so that the drill tip appears centered in the (circular) fixation hole on the x-ray image. This is also done iteratively, typically requiring 5-10 iterations and x-ray images.

[0009] Once this is achieved with sufficient accuracy, the drill is aligned with the target trajectory, leaving the drill tip in place. At this angle, this alignment is typically not confirmed by x-ray, because the power drill and the surgeon's hand block the view. Therefore, the surgeon attempts to align the drill parallel to the "C," using the position of the C-arm as a guide. Achieving and then maintaining such alignment while drilling requires a fairly high level of manual dexterity. Furthermore, surgeons typically fail to adhere to the requirement to aim at the focal point of the x-ray source. The greater the error introduced by failing to aim at the focal point, the further the fixation hole appears from the center of the x-ray image. Typical errors are in the 1-2 mm range at the target point (fixation hole). This is close to the 3 mm limit, beyond which distal fixation fails in most nail fixation systems. All of this translates into unsuccessful drilling attempts, especially for inexperienced or less skilled surgeons.

[0010] Because fixation to more than one hole is generally required, the entire procedure must be repeated for each hole. Therefore, completing the entire nail fixation procedure is typically very time-consuming, requires numerous x-rays, and often results in unsuccessful drilling attempts. This means that distal fixation is one of the most frustrating procedures in the field of osteosynthesis. This sometimes leads to the shortcut of using a short nail instead of a long nail, which then further worsens the patient's prognosis and subsequently results in a significant number of revision surgeries.

[0011] For this reason, some manufacturers offer flexible mechanical solutions (hereafter referred to as "long aiming devices") that adjust the bending of the nail in the medullary canal. Although long aiming devices simplify the procedure, their application is still not easy, as the X-ray images showing the long aiming device must be interpreted correctly and the C-arm position adjusted accordingly. Only after correct adjustment of the C-arm can the long aiming device be properly adjusted.

[0012] EP 2801320 A1 (European Patent Application Publication No. 2801320 A1) proposes the concept of detecting a reference body having a metal marker at a fixed, known position relative to a long aiming device and determining the imaging direction of the reference body. Based on this, the system can provide instructions on how to adjust the C-arm imaging device. A disadvantage of such a system is that the X-ray image must include the reference body. For the adjustment of a long aiming device in the case of an antegrade femoral nail with a fixation hole in the lateral direction, US 2013 / 0211386 A1 (US Patent Application Publication No. 2013 / 0211386 A1) uses a reference body to determine the bending of the nail in the ML and AP directions, respectively.

[0013] Beyond distal fixation, enhancing the safety and accuracy of minimally invasive procedures generally requires surgeons to have access to necessary intraoperative 3D information—namely, information regarding the relative 3D position and orientation between instruments and target objects / structures, or between multiple anatomical objects. This information may be displayed by a tracking-based navigation system, which requires preoperative 3D imaging and then records intraoperative 2D imaging data along with the preoperative 3D data during surgery. Another alternative is the use of a database (e.g., an implant database) containing 3D information about the target object and additional instruments, often in the form of additional hardware (e.g., long aiming devices). These systems are often cumbersome, time-consuming, and tedious to set up and use intraoperatively, and are typically expensive. Therefore, due to all these drawbacks, navigation-based systems are not always available or even feasible for use in orthopedics and trauma. The same explanation applies to systems for intraoperative 3D imaging (e.g., O-arms, 3D C-arms), which also add high X-ray doses.

[0014] This highlights the need for a non-invasive and easy-to-use system that can provide intraoperative 3D information without a tracking system and without requiring any additional hardware components. The present invention proposes a system and method that requires only a computer and a display and / or loudspeaker to process intraoperative 2D images. Both techniques that utilize deterministic 3D data of the target object (e.g., in the form of 3D preoperative imaging data or 3D model data of the implant) and techniques that do not require such data are provided. Summary of the Invention

[0015] It is preferable to work without the use of any reference bodies or other additional hardware (e.g., aiming devices) because it simplifies product development, is more cost-effective, allows for an operating room workflow that more closely resembles a typical workflow, and eliminates the additional uncertainty introduced by mechanical interfaces due to reference bodies (e.g., when using new implants).

[0016] The present invention, as described herein, proposes combining knowledge of the X-ray image generation process with artificial intelligence (a form of so-called deep morphing and / or the use of neural networks) in place of any reference object to provide information needed, for example, when treating a fractured bone. It may therefore be considered an object of the present invention to provide an apparatus and / or method that allows for the 3D representation and determination of the relative 3D position and relative 3D orientation between multiple objects that are at least partially visible in an X-ray projection image. The objects may be any object visible in an X-ray image, such as an anatomical structure, an implant, a surgical instrument, and / or part of an implant system.

[0017] The term "3D representation" may refer to a complete or partial description of a 3D volume or surface, and it may also refer to selected geometric aspects such as radii, axes, and planes. While it may be possible to determine complete 3D information about the 3D surface or volume of an object, in many applications it may be sufficient to determine only selected geometric aspects.

[0018] Throughout this application, the terms "locate" and "localization" refer to determining the 3D orientation of an object and the 2D spatial position of its projection onto the image plane. The imaging depth (which is the distance of the object from the image plane), on the other hand, is estimated (with some uncertainty) based on a priori information about the typical placement of objects in an operating room, e.g., the relative positions of the implant, patient, and imaging device. For most purposes of the present invention, such estimated imaging depth is sufficient. Some applications may require more accurate determination of the imaging depth, which is possible in certain cases, as discussed further below.

[0019] According to one embodiment, 3D reconstruction (i.e., determination of a 3D representation) and localization of objects with some variability in their shape and appearance is provided. This can be done based on a single X-ray image, or for greater accuracy, based on multiple X-ray images. 3D representation and localization of relevant objects, such as anatomical structures, implants, surgical instruments, and / or parts of implant systems, can also be provided, even if they are not visible or only partially visible in the X-ray image.

[0020] According to one embodiment, determination of relative 3D position and orientation between multiple objects is provided, even when at least one of the objects cannot be individually localized with sufficient accuracy. This may also be achieved by utilizing a priori geometric information regarding the relative position and / or orientation between sides of at least two objects, and possibly by limiting the range of allowable X-ray imaging directions. Potential clinical applications include freehand distal fixation, sacroiliac (SI) or pedicle screw placement, and evaluation of anatomical reduction of fractures.

[0021] It should be noted that image data for processed x-ray images may be received from an imaging device, such as a C-arm based 2D X-ray device, or directly from a database. Aspects of the present invention may also be used to process medical images acquired using other imaging modalities, such as ultrasound or magnetic resonance imaging.

[0022] The proposed system according to one embodiment generally comprises at least one processing device configured to execute a computer program product comprising a set of instructions that cause the processing device to (i) receive X-ray projection images, the properties of which depend on imaging parameters, (ii) classify at least one object in the X-ray projection images, (iii) receive a model of the classified object, and (iv) determine a 3D representation of the classified object and localize the classified object with respect to a coordinate system by matching a virtual projection of the model to an actual projection image of the classified object. This process may take into account the characteristics of the X-ray imaging method, and in particular the fact that Thales' theorem applies, as will be discussed in the examples below.

[0023] The X-ray projection images may represent anatomical structures of interest, particularly bones. The bones may be, for example, bones of the hand or foot, i.e., long bones of the lower limbs, such as the femur and tibia, and long bones of the upper limbs, such as the humerus or vertebrae or pelvis. The images may also include artificial objects, such as surgical instruments (e.g., drills) or bone implants, that have already been inserted or fixed into the imaged anatomical structures of interest.

[0024] In the context of the present invention, a distinction is made between "object" and "model." The term "object" is used for a real object, such as a bone or a part of a bone or another anatomical structure, or an implant such as an intramedullary nail, a bone plate, or a bone screw, or a surgical instrument such as a sleeve, k-wire, scalpel, drill, or aiming device that may be connected to the implant. Furthermore, "object" may describe only a part of a real object (e.g., a part of a bone), or it may be an assembly of real objects and therefore may consist of subobjects. Furthermore, an object may be referred to as a "structure" to emphasize that it is a subobject of another object. For example, a "fixation hole" in a nail may be considered a structure (or subobject) of the object "nail." As another example, a "pedicle" of a vertebra may be considered a structure of the object "vertebra." Nevertheless, a structure (such as a pedicle) itself may simply be referred to as an "object."

[0025] The term "model" is used for a virtual representation of an object (or sub-object or structure). For example, a dataset defining the shape and dimensions of an implant may constitute a model of the implant. As another example, a 3D representation of an anatomical structure, such as that generated during a diagnostic procedure, may be considered a model of the real anatomical object. Note that a "model" may describe a specific object, such as a specific nail or a specific patient's left femur, or a class of objects, such as femurs in general, with some variability. In the latter case, such objects may be described by, for example, a statistical shape or appearance model. Thus, the objective of the present invention may be to find a 3D representation of a specific instance from a class of objects depicted in an acquired X-ray image. For example, the objective may be to find a 3D representation of a vertebra depicted in an acquired X-ray image based on a general statistical shape model of vertebrae. It may also be possible to use a model including a discrete set of deterministic possibilities, and the system then selects which of these best describes the object in the image. For example, there may be several nails in the database, and the algorithm then identifies which nail is depicted in the image (if this information is not provided by the user in advance).

[0026] Since the model is actually a computer data set, it is readily possible to extract from that data certain information such as the geometric aspects and / or dimensions of the object that is substantially represented.

[0027] The model may include more than one portion of the imaged object, one or more of which may not be visible in the x-ray projection image. For example, a model of an implant may include a screw intended to be used with the implant, but if only the implant has already been introduced into the anatomical structure, only the implant will be visible in the x-ray projection image.

[0028] It should also be noted that the model does not have to be a complete 3D model of the real object in the sense that it describes only certain geometric aspects of the object, such as the fact that the femoral head can be approximated by a ball in 3D and a circle in a 2D projection image, or the fact that the pedicle of a vertebra has a cylindrical shape.

[0029] According to one embodiment, a system for processing X-ray images generally includes a processing device and a software program product, which, when executed by the processing device, causes the system to perform the following steps: first receive X-ray images, which are projection images of at least a first object and a second object; then classify the at least first object and the second object, and receive 3D models of each of these objects, e.g., from a database; and, based on the direct X-ray image and / or the respective 3D models, determine and identify a first geometric aspect of the first object with respect to the 3D model of the first object, and determine and identify a second geometric aspect of the second object in the 3D model of the second object. The second geometric aspect may be a point.

[0030] Furthermore, the spatial relationship between the first object and the second object is determined based on the 3D model of the first object, the 3D model of the second object, and information that the second geometric side (e.g., a point of the second object) is located on the geometric side of the first object. The geometric side of the first object may be a plane, a line, or a point. Of course, the geometric side may include multiple of the above aspects and combinations thereof. As a result, the geometric side used in this specification may be a more complex shape, such as the edge of a fragment in the case of a fracture.

[0031] According to one embodiment, the geometric aspect of the first object is a plane or a line, and an X-ray image is generated with the imaging direction tilted relative to the geometric aspect at an angle ranging from 10° to 65°. In fact, the plane or line associated with the first object may be tilted relative to the imaging direction. The X-ray image generated with the tilted imaging direction need only contain sufficient information about the first object to enable the determination of the geometric aspect. In other words, the appearance of the first object as visible in the X-ray image provides the processing unit of the system with sufficient information to classify the object and automatically identify the geometric aspect. The angle may range from 15° to 45°. Alternatively, the angle may range from 20° to 30°. Assuming the system provides instructions to the user for adjusting the C-arm, the system may instruct the user to orient the imaging direction relative to the geometric aspect of the first object, or may instruct the user to orient the geometric aspect of the first object at an angle of, for example, 25°.

[0032] According to a further embodiment, the system further determines deviations of the 3D position and 3D orientation of the second object from an intended spatial relationship of the second object relative to the first object.

[0033] For example, the first object may be an anatomical structure or a side of a first implant, and the second object may be an instrument or a second implant. As described in more detail below, the first object may be a vertebra and the second object may be a pedicle screw. Alternatively, the first object may be an intramedullary nail and the second object may be a fixation screw for distally securing the nail. Alternatively, the second object may be a drill or k-wire used to prepare a path for a screw through the bone. Alternatively, the first object and the second object may each be bone fragments that must be anatomically reduced.

[0034] According to one embodiment, the selected point of the second object may be an object (e.g., the tip of a drill) and the information about the 3D position of said tip may be the point of contact between the tip and a surface of the first object (e.g., the outer surface of a vertebra or a long bone such as a femur).

[0035] According to one embodiment, the system may include a device for providing information to a user, wherein the information includes at least one of the following: an X-ray image and instructions regarding the steps of the treatment. Of course, such a device may be a monitor for visualizing the information or a loudspeaker for providing the information audibly.

[0036] According to yet another embodiment, the characteristics of an X-ray imaging device intended to be used with the software program product executed on the processing unit of the system may be known. On the one hand, the imaging characteristics may be known and taken into account when processing the X-ray image data, and on the other hand, instructions to the user for adjusting the C-arm may be facilitated for imaging based on the known geometry of the imaging device and the possibility of changing the position and orientation of the C-arm. A C-arm-based X-ray imaging device may be part of the system.

[0037] The appearance of an object in a projection image may be affected by the X-ray imaging procedure. For example, imaging parameters such as imaging direction relative to gravity (which describes the direction the X-ray beam passes through the object, also called the "viewing direction"), zoom, radiation intensity, and / or the presence of a magnetic field may affect the appearance of the object in the projection image. These or additional imaging parameters may cause changes in properties of the projection image, such as deformation of the projected object due to the pillow effect, mechanical bending of the C-arm imaging device depending on the imaging direction, curvature, noise, and / or distortion. These changes are referred to herein as image properties.

[0038] Of course, it may be possible to determine these image characteristics with sufficient accuracy in the projected image. For example, the location of structures shown in the edge region of the image may be more affected by the pillow effect than structures in the center of the image. As a result, the pillow effect characteristics can be determined with sufficient accuracy based on structures of known shape extending from the edge region to the center region. Image characteristics determined for a region in the 2D X-ray may be extrapolated to the entire image.

[0039] The appearance of an object in an X-ray image is further determined, inter alia, by the attenuation, absorption, and deflection of X-ray radiation, which in turn is determined by the material of the object. The more material the X-ray beam must pass through, the less X-ray radiation is received by the X-ray detector. This can change not only the appearance of the object within its outline, but also the shape of the outline itself in the X-ray projection image, especially in areas where the object is thin. The strength of this effect is also determined by the X-ray intensity and the amount of tissue surrounding the object that the X-ray beam must pass through. The latter is determined by the patient's body mass index and the imaging direction. The amount of soft tissue surrounding the object can be obtained from a database, which takes into account, for example, ethnicity, gender, body mass index, and age.

[0040] Taking into account image and object properties and the effects of X-ray attenuation, absorption, and deflection, the virtual projection of the model may be deformed and / or distorted so that the object is deformed and / or distorted in the X-ray projection image. Such virtual projection may then be matched to the projection seen in the X-ray image. Of course, matching the object to the model in the X-ray projection image may include matching the image properties of the X-ray projection image to the image properties of the virtual projection of the model and / or matching the image properties of the virtual projection of the model to the image properties of the X-ray projection image. Of course, matching in a 3D projection volume by minimizing distance in 3D may also be possible.

[0041] Because the X-ray beam originates from an X-ray source (focal point) and is detected by an X-ray detector at the image plane, the physical dimensions of an object are related to the dimensions of its projection in the X-ray image by Thales' theorem (also known as the fundamental proportionality theorem). The exact imaging depth (which is the distance of the object from the image plane) is generally not required in the context of the present invention. However, if the object is sufficiently large, the imaging depth may be determined by Thales' theorem, and the larger the object, the more accurate this determination becomes. Even for small objects, an approximate estimation of the imaging depth may be possible. Alternatively, the imaging depth can be determined if the size of the X-ray detector and the distance between the image plane and the focal point are known.

[0042] According to one embodiment, a deep neural network (DNN) may be utilized to classify objects (e.g., proximal portion of femur, distal portion of femur, proximal portion of nail, or distal portion of nail) in an X-ray projection image. Note that the DNN may classify objects without determining their location (see, for example, Krizhevsky, A., Sutskever, I., and Hinton, G.E., "ImageNet classification with deep convolutional neural networks," in NIPS, pp. 1106-1114, 2012). It is further noted that objects can be classified even when it is known which objects should be recognizable in an X-ray image. Neural networks may also be used for rough classification of imaging orientation (see, e.g., AP vs. ML, paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jurgen Schmidt: "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge," 2018 (available online at https: / / arxiv.org / abs / 1808.04441). Such classification of objects and imaging orientations may be used to select appropriate models for subsequent processing steps. Note that classification may also be performed by other means or by a priori information about which objects are visible in the image.

[0043] According to one embodiment, the outlines of classified objects may be detected in X-ray images. For objects with variable shapes, such as anatomical structures, this may proceed using the "deep morphing" technique described in the above-cited paper by Pries et al. (2018). This paper proposes a deep neural network-based method for detecting bone structures in fluoroscopic X-ray images. This technique specifically addresses challenges in the automated processing of fluoroscopic X-rays, namely their low quality and the fact that only small datasets are typically available to train neural networks. This technique incorporates high-level information about the object in the form of a statistical shape model. This technique consists of a two-stage approach (called deep morphing): in the first stage, a neural segmentation network detects the outline of bones or other objects, and then in the second stage, a statistical shape model is fitted to this outline using a variant of the active shape model algorithm (although other algorithms can also be used for the second stage). This combination allows the technique to label points on the object outline. For example, in segmenting a femur, this technique can determine which points on the contour in a 2D X-ray projection image correspond to the lesser trochanter region, and which points correspond to the femoral neck region, etc. Objects described by deterministic models (e.g., nails) may also be detected by deep morphing, or simply by neural segmentation networks as in the first stage of deep morphing.

[0044] In a further step, the virtual projection of the model may then be adjusted to match the appearance of the object in the X-ray projection image, taking into account image and / or object properties and the effects of X-ray attenuation, absorption, and deflection. According to one embodiment, for objects described by deterministic models, this matching may proceed along the lines of, for example, Laugier C. (ed.), "Geometric Reasoning for Perception and Action," GRPA 1991, Lecture Notes in Computer Science, Vol. 708, Springer, Berlin, Heidelberg, in the article: Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps." This approach may compensate for the effects of image and object properties and X-ray attenuation, absorption, and deflection by introducing additional degrees of freedom into the parameter vector or by using a suitably adjusted model.

[0045] Neural networks may be trained on a large number of data sets comparable to the data they are being used to. For example, in the case of assessing bone structure in images, neural networks must be trained on a large number of X-ray images of the bones of interest. Naturally, neural networks may also be trained on simulated X-ray images, which may be generated, for example, from 3D CT data, as described in the appendix of the paper "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge" by Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, and Jurgen Schmidt (available online at https: / / arxiv.org / abs / 1808.04441).

[0046] According to one embodiment, two or more neural networks may be used, each trained specifically for a substep required to achieve a desired solution. For example, a first neural network may be trained to evaluate X-ray image data in 2D projection images to classify an anatomical structure, a second neural network may be trained to detect the location of that structure in the 2D projection images, and a third neural network may be trained to determine the 3D location of that structure with respect to a coordinate system. Neural networks may also be combined with other algorithms, such as, but not limited to, active shape models. Note that neural networks may also be trained to locate objects or determine imaging orientations without first needing to detect the object's outline in 2D X-ray images. Note that neural networks may also be utilized for other tasks, such as determining one or more image characteristics, such as the pillow effect.

[0047] According to one embodiment, objects may be manually classified and / or identified in the X-ray projection images, and such classification or identification may be supported by the device by automatically referencing structures recognized by the device.

[0048] According to one embodiment, the system may calculate the geometric aspects of the object (e.g., axes, planes, orbits, outlines, curvatures, center points, or 1D or 2D manifolds) and dimensions of the object (e.g., lengths, radii or diameters, distances). This may be achieved by correspondence between the model and a virtual projection that is matched to the projection seen in the X-ray image.

[0049] When displaying an X-ray projection image, geometric aspects and / or dimensions may be shown as an overlay in the projection image. Alternatively and / or additionally, at least a portion of the model may be shown in the X-ray image, for example as a transparent visualization or 3D rendering, to facilitate a user's identification of structural aspects of the model and thus the imaged object.

[0050] The present invention provides 3D reconstruction and localization of objects whose shape and appearance have some variability. Such objects may be described, for example, by a 3D statistical shape or appearance model. This can be done based on a single X-ray image or multiple X-ray images. Based on one image of an anatomical object, the model may be deformed in such a way that its virtual projection matches the actual projection of the object in the X-ray image. When multiple X-ray images are acquired, information from them may be fused (registered) to improve the accuracy of the 3D reconstruction and / or determination of spatial position or orientation. If the imaging direction is known or can be determined, or if the 3D angle between the imaging directions (which may be expressed, for example, by Euler angles) is known or can be determined when registering multiple X-ray images, the virtual projection can be matched to the actual projection in the X-ray image (e.g., using deep morphing) with greater accuracy.

[0051] According to one embodiment, determination of relative 3D position and 3D orientation between multiple objects is provided even when at least one of the objects cannot be individually localized with sufficient accuracy. This may be addressed by utilizing a priori geometric information regarding the relative 3D position and / or 3D orientation between at least two objects / structures in the X-ray image. This may be, for example, information that a point of one object lies on a line whose relative 3D position and orientation with respect to another object is known. Another example is that the relative 3D position and 3D orientation between a geometric aspect of one anatomical object and a geometric aspect of another anatomical object are known. Since residual ambiguities may still exist, it may be necessary to limit the X-ray imaging direction to (i) a specific anatomically relevant view (e.g., true ML) or (ii) an angular range that allows one of the objects to be viewed from a specific direction.

[0052] The processing device may be implemented by a single processor performing all steps of the process, or by a group or multiple processors that do not necessarily need to be co-located. For example, cloud computing allows processors to be located anywhere. For example, the processing device may be divided into (i) a first sub-processor implementing a first neural net that evaluates image data including classifications of anatomical structures such as bone surfaces, (ii) a second sub-processor implementing a second neural net specialized in determining imaging directions for the classified anatomical structures, and (iii) an additional processor for controlling a monitor for visualizing the results or a loudspeaker for providing audible instructions to the user. Furthermore, one of these processors or the additional processors may control, for example, the movement of a C-arm of an X-ray imaging device.

[0053] According to one embodiment, the apparatus further comprises storage means providing a database for storing, for example, X-ray images. It will be understood that such storage means may also be provided in a network to which the system may be connected, and that data relating to the neural net may be received through that network.

[0054] Additionally, the apparatus may include an imaging device for generating at least one 2D X-ray image, where the imaging device may be capable of generating images from different directions.

[0055] The apparatus may further comprise input means for manually determining or selecting the location or portion of an object in the X-ray image, such as a bone outline, for example, to measure distances in the image. Such input means may be, for example, a computer keyboard, a computer mouse or a touch screen for controlling a pointing device such as a cursor on the monitor screen, which may also be included in the apparatus.

[0056] It should be noted that all references to C-arm movement or rotation in this application always refer to relative repositioning between the C-arm and the patient. Therefore, any C-arm movement or rotation may generally be replaced with a corresponding movement or rotation of the patient / operating table, or a combination of C-arm movement / rotation and patient / operating table movement / rotation. This may be particularly relevant when working with extremities, since actually moving a patient's extremity may be easier than moving the C-arm. It should be noted that the required patient movement / rotation is generally different from the C-arm movement / rotation, especially since patient translation is typically not required if the target structure is already in the desired position in the x-ray image. The system may calculate C-arm adjustments and / or patient adjustments.

[0057] The methods and techniques disclosed herein may be used in systems that support a human user or surgeon, or in systems where some or all of the steps are performed robotically. Thus, all references to a "user" or "surgeon" in this patent application may refer to a human user as well as a robotic surgeon, mechanical support device, or similar device. Similarly, whenever instructions on how to adjust a C-arm are mentioned, it should be understood that such adjustments may also be performed automatically by the robotic C-arm without human intervention, i.e., without human intervention, or by surgical staff with some automated support. It should be noted that because a robotic surgeon and / or robotic C-arm can perform surgery with greater precision than a human, iterative procedures may require fewer repetitions and may execute more complex instructions (e.g., combining multiple iterative steps).

[0058] The computer program product may preferably be loaded into the random access memory of a data processor. Thus, according to one embodiment, a data processor or processing unit of a system may be configured to perform at least part of the described process. The present invention further relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program may be stored. However, the computer program may also be provided over a network, such as the World Wide Web, and downloaded into the random access memory of a data processor from such a network. Furthermore, the computer program may be executed on a cloud-based processor, and the results provided over the network.

[0059] Additionally, prior information about the implant (e.g., nail size and type) may be obtained before or during surgery by simply scanning the implant packaging (e.g., bar code) or anything written on the implant itself.

[0060] To further understand the present invention, an exemplary method for distally fixing a bone nail into a long bone is described, in which a bone screw is inserted into the long bone through a hole in the bone nail. The hole has a hole axis, which may be considered a geometric aspect of the nail. The method may include placing a drill with its tip in contact with the outer surface of the long bone so that the tip is positioned on the hole axis of the hole in the bone nail, with the drill axis oriented at an angle of 10 to 70 degrees relative to the hole axis. Here, the hole axis is a line representing a first object, i.e., a geometric aspect of the bone nail, and the tip of the drill is a point representing a second object, i.e., a geometric aspect of the drill.

[0061] A first X-ray image of the bone drill and bone nail in the long bone with the tip of the drill on the lateral surface of the long bone is generated with the imaging direction in the direction of the hole axis of the bone nail. Those skilled in the art will understand that the imaging direction may be true medial-lateral, and therefore the hole should be visible as a circle in the case of distal fixation of the femoral nail.

[0062] The system may then determine the actual angle between the drill axis and the hole axis based on knowledge of the contact point, based on the 3D model of the drill, and based on the 3D model of the bone nail. Based on the determined angle, the system may provide instructions to reorient the bone drill so that the tip is still on the bore axis and the drill axis is close to the hole axis, where "close" means having a deviation of up to 15° from the hole axis. It may be sufficient to roughly align the drill drilling trajectory with the imaging direction.

[0063] A second X-ray image of the drill and bone nail in the long bone may be generated with a second imaging direction oriented relative to the first imaging direction at an angle ranging from 10 to 65° while maintaining the drill tip position. An easy way to change the orientation may be to move only the C-arc in the anterior-posterior direction when starting from a medial-lateral imaging direction. It is also crucial here that the drill, i.e., the second object, be sufficiently visible in the next X-ray image to allow automatic determination of, for example, the position and orientation of the drill axis. Therefore, the angle may be in the range of 10 to 65°, preferably 15 to 45°, and most preferably 20 to 30°.

[0064] Based on the second X-ray image, deviation of the 3D position and 3D orientation of the drill axis from the hole axis of the bone nail hole may be determined. If deviation occurs, the position and orientation of the bone drill may be adjusted, and a bore may be drilled through the bone nail hole into the long bone. Drilling along the target trajectory may be confirmed by one or more X-ray images at the same angle as the imaging direction relative to the drill axis.

[0065] The principles of the present invention may also be applied to a method of inserting a bone screw into the pedicle of a vertebra, which may include placing a drill with its tip in contact with the outer surface of the vertebra such that the tip is positioned on an axis extending through the pedicle of the vertebra, the drill axis of the drill being oriented at an angle of 10 to 65 degrees relative to a target axis through the pedicle.

[0066] As in the above method, a first X-ray image including the drill and vertebrae is generated from an imaging direction, e.g., true AP, such that the pedicle opening is clearly visible. The difference between these two methods is that the imaging direction of the first X-ray image can be considered to be the anterior-posterior direction when the patient is lying flat on his / her stomach, and the imaging direction does not need to coincide with the pedicle axis (target trajectory) because both objects are in contact with each other. Using such a tilted view, the relative 3D position and 3D orientation between the drill and pedicle may be determined based on knowledge of the contact points.

[0067] As a next step, the actual angle between the drill axis and the pedicle axis may be determined based on the 3D model of the drill and the 3D model of the vertebra. According to instructions provided by the system, the drill may be reoriented so that the tip of the drill is still on the pedicle axis and the drill axis is near the target pedicle axis. If the inclination of the pedicle axis relative to the viewing direction is large enough so that neither the power tool nor the hand obstructs the view, a second X-ray image may be generated from the same direction. Typically, the inclination between the pedicle axis and the true AP viewing direction of the corresponding vertebra is between 10 and 45°. The drill's position and orientation may be adjusted if necessary, and then a hole may be drilled through the pedicle into the vertebra.

[0068] The principles of the present invention may also be applied to a method of inserting a bone screw into a sacroiliac (SI) joint. The method may include positioning a drill with its distal tip in contact with the outer surface of a vertebra such that the tip of the drill is positioned on an axis extending through a desired drill path through the SI joint, with the drill axis oriented at an angle of 10 to 65 degrees relative to a target axis through the pedicle.

[0069] As in the method described above, a first X-ray image is generated including the relevant portion of the drill, the ilium, and the sacrum, with the imaging direction aligned with the direction of the drill path. As a next step, based on knowledge of the contact points, based on the 3D model of the drill, and based on the 3D model of the vertebrae, the actual angle between the drill axis and the axis through the drill path may be determined. According to instructions that may be provided by the system, the drill may be reoriented so that the tip of the drill is still on the axis of the drill path and the drill axis is near the target axis of the drill path.

[0070] A second X-ray image of the drill, the relevant portion of the ilium, and the relevant portion of the sacrum is then generated with the drill and imaging orientation swapped. The drill may now be positioned approximately on the target trajectory through the drilling path, and the second imaging orientation may be oriented at an angle ranging from 10 to 65° relative to the first imaging orientation. The C-arc of the X-ray imager may be rotated relative to the tilted viewing orientation. The angle between the two imaging orientations may also range from 15 to 40°, or alternatively, from 20 to 30°. Such tilted views may be used to determine the deviation of the 3D position and orientation of the drill axis from the target axis through the drilling path. The drill position and orientation may be adjusted, if necessary, before drilling into the ilium and SI joint.

[0071] As will become clear from the above description, a main aspect of the present invention is the processing of X-ray image data that allows for the automatic interpretation of visible objects. The method described herein is to be understood as a method for assisting the surgical treatment of a patient. Therefore, according to one embodiment, the method does not include any steps of treatment of an animal or human body by surgery.

[0072] It should be noted that the embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims (computer programs), and other embodiments are described with reference to apparatus-type claims (systems / apparatuses). However, a person skilled in the art will infer from the above and following description that, unless otherwise specified, all combinations of features belonging to one type of subject matter and all combinations between features relating to different subject matters are considered to be disclosed by the present application.

[0073] The above-defined and further aspects, features and advantages of the present invention may be obtained from the example embodiments described hereinafter and will be explained with reference to the example embodiments also shown in the drawings, to which the present invention is not limited. [Brief explanation of the drawings]

[0074] [Figure 1] An example of 3D registration of AP and ML images is shown. [Figure 2] An example of 3D registration of AP and ML images is shown, and the effect of an inaccurately estimated C-arm width is illustrated. [Figure 3] Compare the situations in Figures 1 and 2. [Figure 4] 1 shows an example of 3D registration of AP and ML images and illustrates the effect of zooming. [Figure 5] Compare the situations in Figures 1 and 4. [Figure 6] 1 shows an example of 3D registration of AP and ML images and illustrates the effect of X-ray receiver size. [Figure 7] 1 shows an example of image distortion for an intramedullary nail. [Figure 8] The definition of drill inclination is shown below. [Figure 9] 3D array with two different drill positions is shown. [Figure 10] 10 shows the outline of the drill in an X-ray projection image corresponding to the 3D array of FIG. 9. [Figure 11] A zoom into the X-ray image showing the outline of two drills corresponding to different inclinations (43–45°) is shown. [Figure 12] A zoom into the X-ray image showing the outline of two drills corresponding to different inclinations (23–25°) is shown. [Figure 13] Figure 1 shows accurately and inaccurately determined outlines of the proximal femur, the latter corresponding to an angular error of 2.5°. [Figure 14] Shown are correctly and incorrectly determined outlines of the proximal femur, the latter corresponding to an angular error of 6°. [Figure 15] Axonal X-ray of the lumbar spine. [Figure 16] Determine the rotation axis of the C-arm. [Figure 17] Shown are circular and oblong holes in nails including chamfers. [Figure 18] 1 is an x-ray image showing a titanium nail rotated 25° about its axis away from the fixation plane. [Figure 19] 1 is an x-ray image showing a titanium nail rotated 45° about its axis away from the fixation plane. [Figure 20] 1 is an X-ray image showing the distal portion of the nail from an incorrect imaging orientation. [Figure 21] 1 is an X-ray image showing the distal portion of the nail from the correct imaging orientation. [Figure 22] 1 is an x-ray image showing a nail and a drill with an incorrectly placed drill tip. [Figure 23] 1 is an x-ray image showing the nail and the drill with the drill tip correctly positioned. [Figure 24] The general workflow for the proposed procedure is shown. [Figure 25] Details for rapid implementation of the general workflow of FIG. 24 are provided. [Figure 26] Details for high fidelity execution of the general workflow of FIG. 24 are provided. [Figure 27] 1 shows an axial view of the proximal end of the tibia.

[0075] Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. The present disclosure will now be described in detail with reference to the drawings, which are described in terms of exemplary embodiments and are not limited by the specific embodiments shown in the drawings. DETAILED DESCRIPTION OF THE INVENTION

[0076] 3D Reconstruction and Localization of Anatomical Objects from a Single X-ray Image The above-cited paper on deep morphing by Pries et al. (2018) proposes a method that allows a system to detect bone outlines / contours (in 2D projection images) and label points on the contours. For example, in femur segmentation, this technique can determine which points on the contour in 2D X-ray projection images correspond to the lesser trochanter and which points correspond to the femoral neck. Considering a 3D statistical shape or appearance model of the same anatomical structure, this model can then be deformed in such a way that its virtual projection matches its actual projection in the X-ray image, thus obtaining a 3D reconstruction of the anatomical structure and enabling object localization and imaging orientation determination. On the other hand, if the imaging orientation is already known, 3D reconstruction of the anatomical object can be performed with greater accuracy. For example, the imaging orientation may be known because the surgeon has been instructed to acquire X-ray images in a specific orientation (e.g., true AP or true ML), or because the specific imaging orientation has been detected, for example, by an algorithm invented by Blau, which was filed as a patent application on August 23, 2018.

[0077] The accuracy of 3D reconstruction of anatomical objects can be improved by using a priori information. This a priori information can be the patient's size or gender, but it can also be more anatomical-specific information, such as geometric information about the anatomical object being reconstructed. In the example of a 3D reconstruction of the proximal femur based on ML images, this information might include the femoral neck length or CCD angle. However, because such information may not be determined with sufficient accuracy in typical ML images, it can be extracted from AP images routinely acquired earlier in the surgical procedure on the proximal femur. The more information used from earlier images, the more accurate the 3D reconstruction can be. Another way of describing this procedure is that a 3D reconstruction of the proximal femur can be performed based on AP images with typical remaining uncertainties (such as the width of the femoral neck in the AP direction), and this 3D reconstruction serves as the starting point for a 3D reconstruction based on a priori information or later ML images.

[0078] The geometric a priori information may also consist of known correspondences between points in the 2D projection images and points in a 3D model of the anatomical object. - the points in the 2D projection images correspond to points on a line whose position and orientation relative to a 3D model of the anatomical object is known, or - points in the 2D projection image correspond to points on a plane whose position and orientation relative to a 3D model of the anatomical object are known If , is known, less specific geometric information may still be useful.

[0079] Such geometric a priori information may be provided by a surgeon, for example, by user input through a user interface, placing an object (e.g., an instrument such as a drill or k-wire) at a specific anatomical point visible in the 2D projection image. This may possibly be achieved due to prominent anatomical features in a particular imaging direction (e.g., true AP or true ML), or by palpation or visual identification of the actual object. All of this a priori information significantly reduces ambiguity in the 3D reconstruction.

[0080] Determining relative 3D position and 3D orientation between objects by reducing or eliminating ambiguity The invention by Blau, filed as a patent application on November 26, 2018, considers how, if two objects can be located based on 2D X-ray images, and if the two objects are known to be in contact with each other in physical 3D space, the relative 3D position and 3D orientation between both objects can be determined.

[0081] The present invention proposes how to determine the relative 3D position and 3D orientation between two (or more) objects (or structures) based on 2D X-ray images when at least one object cannot be located with sufficient accuracy. Such determination of the relative 3D position and 3D orientation may be possible based on a priori information about the 3D positions of specific points of one object relative to another, which may for example be obtained from an X-ray image previously acquired from a specific imaging direction that allows the location of at least one structure of the other object. For this to work, it may be necessary to restrict the allowable range of imaging directions in the current image.

[0082] For illustrative purposes, let us assume that one of these objects is a drill and the other is a nail or some other anatomical object (e.g., a vertebra). The anatomical object may be described using either a deterministic 3D model (generated for a specific patient, e.g., using 3D imaging methods) or a statistical 3D model describing general bone variability. In the former case, greater accuracy can be achieved. Even if a complete and accurate 3D model of the drill is available, it is not always possible to locate the drill with sufficient accuracy. As explained above, there is residual uncertainty in the imaging depth (the distance of the object from the image plane) when locating an object. This uncertainty in the imaging depth in determining the 3D position of the drill tip also creates ambiguity regarding the inclination of the drill in the direction of the imaging depth. As shown in Figure 8, the inclination of the drill is defined as the viewing angle relative to the drill tip, denoted as 8.DT. Since the drill is very thin and has a straight structure with a clearly defined axis, the inclination of the drill may be defined as the angle between the dashed line designated 8.L1 connecting the drill tip 8.DT and the X-ray focus 8.FP and the solid line designated 8.L2, which is the drill axis.

[0083] Consider, for example, a 3D configuration with two different 3D drill positions (designated 9.D1 and 9.D2) shown in Figure 9. The two drill positions differ in imaging depth and also in drill inclination (by 6.5°). Furthermore, the two drill positions in the x-ray projection image may not be distinguishable because they result in essentially the same x-ray projection, as shown by the nearly identical drill outline 10.O in Figure 10.

[0084] There are two main reasons for this:

[0085] 1. A drill is a relatively thin instrument with an almost constant diameter, which in a typical X-ray image is only a few pixels wide. In an X-ray projection, an object tilted in the imaging direction is depicted as wider at one end and smaller at the other end. However, this may only be detectable if this change in width is sufficiently severe (e.g., at least one pixel wide). For the accuracy required for angle detection of, e.g., less than 3° for a thin drill with a diameter of, e.g., less than 4 mm, this may not generally be the case.

[0086] Because the sine function for small angles has a slope close to zero, it may generally not be possible with sufficient accuracy to locate instruments such as drills (with a diameter of a few mm), only the front part of which is visible in X-ray images from a viewing angle close to 90° (which is the tilt of the drill). For example, at an angle of 70°, the projection of the drill is shortened by only 6 percent, which results in a slight change in the projected shape of the drill tip. Such a small change may not be sufficient to determine the tilt of the drill with an accuracy of about 3°. The limit of detection capability is an angle of about 65°, where the projection of the drill is shortened by 9.4 percent. Depending on the instrument, this may or may not be sufficient for the required accuracy.

[0087] As the viewing angle (tilt) decreases, it becomes easier to distinguish a difference in tilt of, say, 2°. This is shown in the x-ray image of FIG. 11, which shows the projections and outlines of two drills. The solid white line labeled 11.D1 corresponds to a drill with a 45° tilt, and the dashed white line labeled 11.D2 corresponds to a drill with a 43° tilt. These outlines differ in several places, allowing them to be distinguished by the system. As the viewing angle decreases, more clearly distinguishable outlines result. This can be observed in the x-ray image of FIG. 12, which shows the projections and outlines of two drills. The solid white line labeled 12.D1 corresponds to a drill with a 25° tilt, and the dashed white line labeled 12.D2 corresponds to a drill with a 23° tilt. These outlines are now clearly different in several places, allowing them to be easily distinguished by the system.

[0088] 2. In a typical X-ray, only the tip and top of the drill are visible, while the other end is not. If both ends are visible, the drill's inclination can be determined with high accuracy based on the shortened length of the drill in the projection image. This is also possible if the drill has a marking that is clearly visible in the X-ray image, for example, halfway along the shaft. Marking such a drill is easy, but it also represents a change to the existing instrument. Another option is to use the start of the drill's thread as such a marking. However, with a typical drill, the start of the thread may not be clearly visible enough in the X-ray projection image, and therefore it may not generally be possible to use it for this purpose.

[0089] This problem may be addressed by more accurately determining the imaging depth of the drill tip, thereby sufficiently reducing or eliminating the ambiguity. This may be possible when determining the relative 3D position and 3D orientation between two objects in an X-ray image. The other object is referred to as the target object, and may be, for example, a nail. The ambiguity of the position of the drill tip relative to the target object may be reduced, for example, by defining a trajectory whose 3D position and 3D orientation relative to the target object are known, provided that the position of the drill tip lies on this trajectory and the imaging direction of the trajectory at the position of the drill tip is sufficiently large (in other words, the trajectory must be sufficiently different from a line parallel to the line connecting the drill tip and the X-ray focal point).

[0090] It may be even more useful if the 3D position of the drill tip relative to a point on the target object is known. This is the case, for example, when the drill tip is in contact with the target object, such as when the drill tip is in contact with the ilium in a sacroiliac screw fixation procedure. However, it may still be sufficient if the drill tip is known to be on a plane whose 3D position and 3D orientation relative to the target object are known. This is the case, for example, for distal fixation of a further hole after completion of fixation of the first hole. Here, two degrees of freedom (DoF) are not determined. For a more detailed description of the distal fixation procedure for nails, please refer further to the corresponding section below.

[0091] Furthermore, there may be ambiguity regarding the target object, especially when the target object is an anatomical object. However, when the device is in contact with the target object, there are imaging directions that involve different directions of ambiguity in locating each of the objects. Therefore, even in such cases, sufficiently accurate determination of relative 3D position and 3D orientation may be possible.

[0092] Below, we next present these considerations for a drill contacting the trochanter of the proximal femur. In this imaging orientation, it may be clearly determined, for example by palpation, at which point the drill tip contacts the femur. Due to the ambiguity explained above, there are several possibilities for the 3D position and 3D orientation of the drill relative to the femur, all of which result in the same projection of the drill in the X-ray image, each corresponding to a different relative 3D position and 3D orientation of the depicted anatomical structures. By jointly considering the depicted anatomical structures and using a priori information about the contact point, it may be possible to select which of these possibilities is the correct one.

[0093] Figure 13 shows an X-ray image of such a scenario. The solid white line (denoted 13.CO) is the femur outline corresponding to the femur's correct 3D position and orientation, and the dashed white line (denoted 13.IO) is the femur outline corresponding to one of the possible inaccuracies for the femur's 3D position and orientation. By comparing these possible outlines with the segmented and labeled femur in the X-ray image (which may be achieved, for example, by deep morphing), the one that best matches the segmented femur is selected. In the depicted scenario, the inaccurate outline 13.IO is clearly different from the correct outline 13.CO and may therefore be discarded, even though the inaccurate outline corresponds to only a 2.5° angular error (with respect to the drill tilt). Larger angular errors can result in even more clearly inaccurate outlines, as depicted in Figure 14, where the inaccurate outline 14.IO corresponds to an angular error of 6° (in terms of drill inclination) and is clearly distinguishable from the correct outline 14.CO.

[0094] A further example may be determining the relative 3D position and orientation of an instrument with respect to the pedicle of a vertebra. Figure 15 shows an A-P X-ray image of a lumbar spine, in which a surgeon has placed a Jamshidi needle (labeled 15.JN) in the right pedicle of the lumbar spine. The pedicle opening (labeled 15.OP) is clearly visible in this particular imaging direction as a brighter area. The center of the pedicle can therefore be clearly identified, and the instrument can be placed at the center of this opening (pedicle axis). Based on the a priori information that the instrument is positioned on the pedicle axis, the relative 3D position and orientation between the instrument and the pedicle may be determined within a suitable angle range for many other imaging directions, by contacting the bone surface and following the method outlined above.

[0095] The process of aligning two or more X-ray images taken from different directions Depending on the shape of the bone, there may still be residual ambiguities or matching errors in 3D reconstructions based on only one image. This may be mitigated by acquiring multiple images, potentially from different viewing directions, by rotating and / or translating the C-arm between images. In general, additional images from different imaging directions are more useful, and the more different the imaging directions (e.g., AP and ML images), the more useful the additional images are in determining 3D information. However, instead of changing to a completely different view (AP to ML or vice versa), it may even be beneficial to add images from only slightly different viewing angles that can be more easily acquired during surgery.

[0096] The present invention also allows for recording multiple X-ray images of at least one common object taken from different directions, which is important because 3D registration allows for the determination of relative 3D positions between multiple objects without the explicit determination of imaging depth.

[0097] For 3D reconstruction of a deformable-shape object (typically an anatomical structure described by, for example, a statistical shape or appearance model, referred to in this section as "object F") based on two or more X-ray images, the procedure outlined above for one image may be extended to two or more images. That is, deep morphing may be used to detect the contour of object F and label points on that contour in each of the two X-ray images. Given a 3D statistical shape model of object F, this model can then be deformed so that its virtual projection matches the actual projection of object F as closely as possible in two or more X-ray images simultaneously. This procedure does not require a priori information about the imaging orientation, as it implicitly determines the imaging orientation for each X-ray image.

[0098] As an alternative to aligning a pair of X-ray images taken from two different imaging directions, it may be possible to increase the accuracy of the alignment process by taking into account the 3D angle between the imaging directions, which can be determined using two different procedures. The more accurately this angle can be determined, the more accurate the 3D alignment may be.

[0099] One way to determine this angle is to determine the imaging direction for each X-ray image, for example, using the invention by Blau, filed as a patent application on August 23, 2018, and compare the difference between them. Another method may utilize another object (referred to as "object G") in the X-ray image, whose model (e.g., a nail possibly connected to a targeting device or instrument) is deterministic. Object G may be located by matching a virtual projection of object G to its actual projection in each X-ray image. Note that, without further conditions or a priori information, some objects, particularly instruments such as drills or k-wires, may generally not have a sufficient geometry or size to be localized. However, even in such cases, it may be possible to locate object G with sufficient accuracy, provided that (i) object G is visible in a certain angular range in all images to be registered, and (ii) some prior information regarding the relative 3D positions between objects F and G is available. The prior information in (ii) is particularly (a) The relative 3D positions of a point on object G and a point on object F are known, or (b) a point of object G lies on a line in physical 3D space whose relative 3D position and 3D orientation with respect to object F is known, or (c) A point of object G lies on a plane in physical 3D space whose relative 3D position and 3D orientation with respect to object F is known. It may be something like this.

[0100] However, the relative 3D position and 3D orientation between objects F and G must be the same in both X-ray images, ie with as little movement as possible between the objects.

[0101] In general, two (or more) images may be registered if they contain objects that can be located with sufficient accuracy. Image registration can be performed with high accuracy if the images contain two (or more) objects that do not move relative to each other during image acquisition. One example in which this procedure can be used is a situation in which an implant (e.g., a nail) has already been inserted into a bone and a 3D model of the bone (e.g., a statistical shape model) is available. In such a scenario, if a drill is visible in all images, albeit at different orientations, and its tip (also visible in all images) remains at the same point (e.g., a point on the bone surface), the 3D position of the drill tip can be determined relative to either of the two objects. This means that in the first image, the instrument can be positioned at any point, and in the second image (obtained from a different viewing direction), the instrument may be reoriented (e.g., by aiming approximately at the target trajectory), but the tip of the instrument remains in the same position. Based on the localization of the target object / structure (e.g., nail / nail hole), both images may be registered, which may allow the determination of the 3D position of a point (tip of the instrument) relative to the target object. This point may then be used to determine with sufficient accuracy the relative 3D position and 3D orientation between the instrument and the target object / structure.

[0102] In other words, a software program product executing on a processor of the system may cause the system to receive a first X-ray image, which is a projection image of at least one object, classify the at least one object, and determine at least one point in the first X-ray image. The system may then receive a second X-ray image, which is a projection image generated in an imaging direction different from the imaging direction used to generate the first X-ray image. In the second image, the at least one object is again classified and at least one point is determined. Based on the classification of the at least one object in the first and second X-ray images and based on the determination of the at least one point in both X-ray images, the two images may be registered, and a 3D position of the point relative to the at least one object may be determined.

[0103] If the at least one object includes two objects and the at least one point is a point on one of the two objects, the system may determine the spatial relationship between the two objects, i.e., the 3D orientation and 3D positioning, based on the aligned images.

[0104] Additionally, the system may determine deviations of the 3D position and 3D orientation of one of the objects from its intended spatial relationship to another object, for example, one object may be a drill, and it is intended to place the drill parallel to and on a trajectory through another object, which may be a bone or an implant.

[0105] The above-described method of registering two X-ray images may also be useful for 3D reconstruction and / or localization of an anatomical object, for example, into which a known implant has been inserted. Locating the implant allows for image registration, which in turn allows for the determination of the 3D position (relative to the implant) of the tip of the drill located on the bone surface. The points thus determined may serve as anchor points for the 3D reconstruction and / or localization of the anatomical object. Following this approach, it may be possible to determine multiple surface points, which means sampling the 3D bone surface at separate points relative to the implant, thereby obtaining a point cloud. Each sample point added to this point cloud may reduce ambiguity in the 3D reconstruction and the determination of the 3D position and orientation of the anatomical structure relative to the implant. If the drill tilt is within the range of 10-55°, this may also allow for the determination of the 3D position and orientation of the anatomical structure (or implant) relative to the drill. Thus, even if a deterministic 3D model of the anatomical structure (e.g., a CT scan) is available, this procedure can be used to determine the 3D position and orientation. The point sampling method can also be used when there are no known implants in fixed positions in the bone, in which case the reconstruction and / or localization and / or registration proceeds directly based on the anatomy.

[0106] In the following, examples are given to illustrate the influence of C-arm width, image detector size, zoom, etc. on 3D registration. In all of these examples, it is shown that determining the imaging depth is not necessary.

[0107] Effect of C-arm width: Figure 1 shows the left femur (designated LF) and nail implant with the attached aiming device (designated NAD). It also shows AP X-ray images (designated 1.AP) and ML X-ray images (designated 1.ML) and their corresponding focal points (designated 1.FP.AP and 1.FP.ML). The 3D ball approximates the femoral head (designated FH), and the dashed white circle is its 2D approximated projection in the image (designated 1.FH.AP and 1.FH.ML). The C-arm has a width of 1000 mm (defined here as the distance between the focal point and the image plane). The cone represents the portion of the X-ray beam passing through the femoral head. Note that throughout this application, we follow the convention of referring to images taken in the posteroanterior direction as "AP" images and images taken in the anterior-posterior direction as "PA" images. Similarly, we refer to images taken in the lateral-medial direction as "ML" images, and images taken in the medial-lateral direction as "LM" images.

[0108] In Figure 2, the C-arm width was incorrectly estimated as 900 mm instead of the true 1000 mm. Therefore, all objects in the image, including the femoral head (FH), appear smaller than they should in the X-ray image. Therefore, it appears as if the objects are moving toward the AP image plane (denoted by 2.AP) and toward the ML image plane (denoted by 2.ML). The corresponding focal points are denoted by 2.FP.AP and 2.FP.ML. The 3D reconstruction of the femoral head (FH) based on the 2D projection of the approximated femoral head (white circles 2.FH.AP and 2.FH.ML) remains unchanged compared to Figure 1. The only parameter that changes is the apparent imaging depth. However, imaging depth is not relevant in this scenario because the relative 3D positions of the femoral head and nail remain unchanged.

[0109] To illustrate that the only difference between Figures 1 and 2 is the apparent imaging depth, Figure 3 shows both scenarios simultaneously.

[0110] Effect of zoom: When one of the images is captured with a zoom ratio, objects appear larger than without zoom. In Figure 4, the AP image (represented by 4.AP) was captured with a zoom ratio of 1.5. Therefore, all objects in the image, including the femoral head (FH), appear as if they have moved toward the focal point in AP (represented by 4.FP.AP). As previously mentioned, the 3D reconstruction of the femoral head (FH) based on the 2D projection of the approximated femoral head (dashed white circle 4.FH.AP and 4.FH.ML) remains unchanged compared to Figure 1. The only parameter that changes is the apparent imaging depth. However, since the relative 3D positions of the femoral head and nail remain unchanged, imaging depth is not relevant in this scenario. A similar observation applies when both images have zoom. Figure 5 compares the situation with zoom (similar to Figure 4) with the situation without zoom (similar to Figure 1).

[0111] Effect of X-ray detector size: If the assumed size of the X-ray detector is 12" instead of the true 9", the object will appear larger in the image and it will appear as if the object has moved towards the focus in both images. This is shown in Figure 6, where · 6.AP.9'' refers to AP images with a 9'' X-ray detector with a focal spot represented as 6.FP.AP.9'' · 6.AP.12'' refers to AP images with a 12'' X-ray detector with a focal spot represented as 6.FP.AP.12'' · 6.ML.9'' refers to ML images from a 9'' X-ray detector with a focal spot represented as 6.FP.ML.9'' · 6.ML.12'' refers to ML images with a 12'' X-ray detector with a focal spot represented as 6.FP.ML.12''.

[0112] The effect is equivalent to a zoom factor applied to both images, so the same conclusions can be drawn as with zooming.

[0113] Measuring the characteristics of classified objects The present invention does not require a priori calibration. Measurements may be made in mm if a known object located near (at a similar depth to) the structure being measured is present in the image. Since the known object has known dimensions, it can be used to calibrate the measurements. This is similar to the procedure proposed by Baumgaertner et al. to determine the TAD value (see Baumgaertner MR, Curtin SL, Lindskog DM, Keggi JM: The value of the tip-apex distance in predicting failure of fixation of peritrochanteric fractures of the hip. J Bone Joint Surg Am. 1995, 77:1058-1064).

[0114] Example 1: A nail has been inserted and an AP image is available. The nail has been identified and located. Because the nail is located mid-diaphysis and therefore at a similar imaging depth as the depicted lateral cortex of the diaphysis, the known nail geometry can be used for calibration. This allows for scaling to determine the distance between the nail axis and the lateral cortex of the diaphysis.

[0115] Example 2: If the imaging depth of object A is known (e.g., because object A is large enough or the size of the X-ray detector and the distance between the image plane and the focal point are known), and if information exists about the difference in imaging depth between object A and object B (e.g., based on anatomical knowledge), it may even be possible to calculate the size of a different object (call it "object B") at different imaging depths based on Thales' theorem.

[0116] Dealing with image distortion for the example of an intramedullary nail Generally, there are two ways to deal with image distortion, which may be combined. 1. De-emphasize areas in the X-ray image that are known to be highly distorted (e.g., image boundaries), and place more emphasis on areas that are less affected by distortion. 2. Determine the distortion and compensate for it

[0117] These are now illustrated in the example of an AP image of a femur with an inserted nail.

[0118] Reference 1. The following letters are used to label Figure 7: The solid lines are the outlines of the nail and aiming device as seen in the distorted X-ray image. The white dashed lines indicate the hypothetical outlines of the nail and aiming device as shown in the undistorted image. 7.D: Distal portion of intramedullary nail 7.C: The center part of the nail containing the hole for the neck screw 7.P: Proximal part of intramedullary nail 7.A: Aiming device

[0119] Typically, 7.D is located in a more distorted region of the X-ray image. Furthermore, the exact location of 7.D is not important when predicting the trajectory for a screw inserted through a hole in 7.C. Therefore, in predicting the screw trajectory, the locations of 7.C and 7.P may be given a higher weighting than 7.D, and the exact weighting may be determined based on their visibility and detection reliability. Also, a higher weighting for 7.C and 7.P may be reasonable because these regions are closer to the regions of interest (screw hole and femoral head). Furthermore, the appearance of 7.C carries information about the rotation of the nail around its axis.

[0120] Reference 2. The distortion in an image may be determined by: a) Earlier surgical procedures (which can be trained on specific C-arms) b) Pre-surgical calibration: A known object (e.g., nail, k-wire, etc.) can be placed directly on the image intensifier / X-ray detector at a known distance relative to the image plane. This can also be used to determine the size of the X-ray detector and the distance between the focal point and the image plane. c) Images acquired earlier (allowing the algorithm to learn during surgery) d) A database with typical distortion effects (e.g., typical pillow effect for typical C-arm positions, Earth's magnetic field). The device may use the knowledge that digital X-ray devices do not introduce distortion.

[0121] If such information is available, it may be used when matching the virtual projection of the model to the projection in the X-ray image. The distortion may be applied to the entire image or specifically to the shape being matched.

[0122] Alternatively and / or additionally, distortions may be explicitly or implicitly determined during the process of matching a virtual projection of an object (e.g., a nail) with a known deterministic 3D model to the appearance of the object in an X-ray projection image. According to one embodiment, this matching may proceed along the lines of the article: Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps" in Laugier C. (ed.), "Geometric Reasoning for Perception and Action." GRPA 1991. Lecture Notes in Computer Science, Vol. 708. Springer, Berlin, Heidelberg. According to one embodiment, the distortions may be described by suitable mathematical models (e.g., radial and / or sigmoidal functions as described by Gronenschild E., "Correction for geometric image distortion in the X-ray imaging chain: local technique versus global technique", Med Phys., 1999 Dec;26(12):2602-16). Distortions modeled in this way may be compensated for by introducing additional degrees of freedom into the parameter vectors of the above-cited paper by Lavallee et al. (1993) when matching virtual projections to projections in the X-ray image.

[0123] Handling exchanges of X-ray source and receiver positions Since the X-ray imaging device allows for left-right image flipping and the present invention does not use a calibration reference attached to the image detector throughout the surgical procedure, it is not necessary to detect the exchange of positions of the X-ray source and receiver device even if the treatment side (left or right bone) is known. The user may be required to provide information on whether the left-right flip function is activated.

[0124] However, even in the absence of such information, it is possible to detect an exchange of the positions of the X-ray source and the receiver, since the instrument or device spans a large range of imaging depth in X-ray images viewed from imaging directions much smaller than 90°. Therefore, an exchange of the X-ray source and the receiver can be detected, since parts of the instrument or device closer to the imaging plane (receiver) are depicted smaller than those further away.

[0125] A method for determining rotation / horizontal / vertical inversion of X-ray images Methods are provided that allow the system to determine rotation, left-right flipping, and upside-down flipping of the x-ray image, which may be used to display the x-ray image so that, for example, the nail appears as if it were actually positioned in front of the surgeon.

[0126] The following is known for the system based on the surgical procedure being performed: Patient positioning (e.g., supine) C-arm position (e.g., image intensifier on the inside, X-ray source on the outside) On which part of the patient's body the surgery will be performed (e.g., left / right leg: this may be known, for example, based on previous proximal procedures, user input, or scans of nail packaging)

[0127] Detection of left-right flip (i.e., determining which side is anterior and which side is posterior) may be based on determining the orientation of the implant, possibly supported by determining the imaging orientation relative to the anatomy (e.g., the condyles face downward in the image when the patient is supine). Alternatively, the user may be instructed to orient the instrument (e.g., a drill) in a particular direction, which may then be used to identify this direction as anterior or posterior.

[0128] Method for locating target objects / structures in the field of view of a C-arm from a desired viewing direction For definition of the rotation axes of the C-arm, refer to FIG. 16. In this figure, the X-ray source is represented by XR, the rotation axis represented by the letter B is called the vertical axis, the rotation axis represented by the letter D is called the propeller axis, and the rotation axis represented by the letter E is called the C-axis. Note that for some C-arm models, axis E may be closer to axis B. The intersection of axis D and the central X-ray beam (labeled XB) is called the center of the "C" of the C-arm. The C-arm may move up and down along the direction indicated by the letter A. The C-arm may move along the direction indicated by the letter C. This terminology is used throughout this application. The distance of the vertical axis from the center of the "C" of the C-arm may vary between C-arms.

[0129] Below, a method is proposed for providing user instructions on how to adjust the C-arm so that the target structure appears at the desired position in the X-ray projection image and is visible from the desired imaging direction (e.g., the fixation hole must appear circular, and the projection of the nail shaft must pass through the center of the X-ray image). Even if the required rotation and translation, for example, based on object localization, are accurately determined, it can be important to determine appropriate user instructions for repositioning the C-arm. As shown in Figure 16, the C-arm has multiple rotation and translation axes. Furthermore, its wheels allow the C-arm to be moved to different positions in the operating room. It also allows translation parallel to the floor and rotation around an axis parallel to the vertical axis, which is typically a large distance (more than 1 m) from the vertical axis.

[0130] The many available options for moving the C-arm make it difficult for the user to determine which option is best for reaching the desired position (for the desired imaging direction) most quickly or with the least effort. Furthermore, there are constraints arising from the operating room setup that prevent the user from moving the C-arm to a specific position. Thus, in some cases, the user may choose to move the patient (or operating table) rather than the C-arm, especially in surgeries involving the upper extremities.

[0131] It may (i) determine the necessary information on how to reposition the C-arm and / or the patient, (ii) translate this information into guidance for the user to select from available means of moving the C-arm (by translation along or rotation around its axis, or by moving the C-arm on its wheels) or moving the patient, and (iii) propose a method for determining the amount of movement required in each case.

[0132] In other words, a method for assisting in adjusting an imaging direction of a C-arm-based imaging device may include receiving information about a current imaging direction, receiving information about a target imaging direction, determining a first one of a plurality of means for rotating and translating the X-ray source and the detector and an amount of movement for the first means, and achieving an imaging direction closest to the target imaging direction. Of course, such a method may be implemented as a software program product that causes a system to perform the method.

[0133] These methods may take into account possible constraints imposed by the construction of the C-arm and the operating room setup, and may select those movements that are easiest for the user to perform and require the fewest number of movements.

[0134] These methods may determine the current imaging direction based on an X-ray image. For example, the viewing or imaging direction may be determined based on an anatomical structure (e.g., using the patent application filed by Blau on August 23, 2018), and / or the target structure (or object) may be located by optionally taking into account a priori geometric information about the implant, instrument, or anatomical structure, and optionally taking into account the 3D position and 3D orientation of the target structure in a coordinate system spanning the image plane, and optionally taking into account the typical operating room setup for the current surgical procedure (position of the patient on the operating table, position of the C-arm relative to the operating table, position of any objects that may prevent the C-arm from moving to a specific position relative to the operating table / patient), and optionally taking into account the typical rotation axis of the C-arm.

[0135] For example, for a femoral nailing procedure, it may be assumed that the patient's typical position relative to the C-arm is known, i.e., the patient is supine, the treatment side is known, and the image receiver is between the patient's legs. Additionally or alternatively, the user may select an operating room setup from a set of provided options. These may be based on information the system gathers, for example, from scanning the implant packaging, previously performed surgical procedures that the system has learned about, such as the patient's positioning on the operating table, where the implant is used, and / or where the C-arm machine is positioned relative to the operating table (e.g., between the patient's legs).

[0136] Translating calculated deviations from the desired position / orientation into instructions for adjusting the C-arm can be important, as these instructions are likely to be as easy to implement as possible. Instructions that do not require moving the C-arm on its wheels may be preferred, as this is less precise and may be more difficult to implement in an operating room setting. In general, it may be preferable to keep the number of instructions low. A neural network may assist with this entire procedure.

[0137] For example, if a large rotation about the vertical axis is required, this should be performed by moving the entire C-arm on its wheels, since this may allow isocentric rotation about the target structure (keeping the target structure close to the central C-arm beam). If the required rotation is about an axis parallel to the C-arm's vertical axis and the rotation is relatively small, the C-arm's vertical axis should be used. If the desired rotation axis is far from the C-arm's vertical rotation axis, such rotation will include a relatively large translation component; however, it must be kept in mind that any such translation can be compensated for when determining any potentially required translation. As explained above, a rough determination of the imaging depth may be possible (up to a few centimeters). Since the distance of the C-axis from the target structure is roughly known, this may be sufficient to calculate, for example, the AP translation resulting from rotation about the C-axis. The offset between the C-axis and the central X-ray beam may also be calculated from the rotation about the C-axis. Furthermore, due to the fact that perspective projection is applied, any translation of the C-arm automatically includes a rotational component.

[0138] Making a rough assumption about the 3D position and orientation of the C-arm axis relative to the target object / structure may be sufficient for the first iteration of the positioning procedure, sometimes resulting in a position sufficiently close to the desired position. For greater accuracy, or to reach a sufficiently accurate position in the fewest number of steps, the system may use information gathered from previous iterations in subsequent steps, thereby enabling a more accurate determination of the 3D position and orientation of the axis relative to the target object / structure. As an example, if translation along or rotation about two or more C-arm axes is required to reach the desired position of the C-arm, the system may instruct the user to move or rotate the C-arm about one axis (e.g., rotate about the C-axis). This may allow for approximation of the desired position and simultaneously determine the 3D position and orientation of the C-axis relative to the target object and the offset between the C-axis and the center of the "C" of the C-arm.

[0139] Alternatively, depending on available a priori information, the 3D position and 3D orientation of the C-arm axis relative to the target object / structure may be determined by either (i) translating the C-arm along two mutually perpendicular axes (if these axes do not intersect, one parallel axis of these axes must be perpendicular to the other axis), or (ii) rotating the C-arm along two mutually perpendicular axes (if these axes do not intersect, one parallel axis of these axes must be perpendicular to the other axis), or (iii) translating the C-arm along one axis and then rotating the C-arm about another axis parallel to the axis of translation (e.g., an anterior translation combined with a rotation about a vertical axis, or a rotation about the C-axis combined with a rotation about a propeller axis, or a proximal translation combined with a posterior translation).

[0140] As an example, for an antegrade femoral nailing procedure (with the patient positioned supine, with the C-arm positioned between the patient's legs), anterior translation of the "C" may, on the one hand, cause a rotation of the viewing direction about the nail axis, thereby enabling a more accurate calculation of the target object's 3D position relative to the C-axis for more accurate determination of imaging depth. On the other hand, the system may determine the rotation of the nail about its axis relative to the operating room floor. At a later point in time, when calculating guidance instructions for anterior / posterior translation, the system may calculate the required translation by using a simple trigonometric relationship, taking into account the effect of the rotation of the nail about its axis relative to the operating room floor determined above on the required translation along the vertical axis. This information may also be used when calculating other translation instructions provided by the system, such as when calculating the required rotation about the vertical axis, to more accurately determine the distance between the object and the vertical axis (because the distance of the axis perpendicular to the C-axis can be more accurately determined). Therefore, the effect of translation in the proximal / distal direction caused by rotation about the vertical axis can be taken into account more accurately, since the distance between the central X-ray beam and the C-axis has already been determined.

[0141] In practice, in the first iteration, it may be sufficient to determine one or two steps of translation or rotation to approach the final desired position. By observing these one or two translations / rotations, the system can obtain sufficient information about the 3D position and 3D orientation of the axis relative to the target object / structure so that in the second iteration, all remaining steps necessary to reach the final desired position can be determined with sufficient accuracy and provided to the user. As described below in the "Examples for Potential Processing Workflows for Distal Fixation Procedures" section, the system does not require very precise positioning of the C-arm; for example, a perfectly circular projection of a circular hole may not be required.

[0142] If the user moves the patient rather than the C-arm in response to a system instruction, checking the image background may help detect such movement and prevent inaccurate determination of the movement / rotation axis. The "image background" in the preceding sentence may also be any object that does not move with the patient and is not entirely radiolucent (e.g., part of the operating table). Simple image difference analysis may be used to reveal whether the C-arm has moved. In this context, it may be required that there is no rotation of the digital image at the C-arm between the C-arm movement and image acquisition.

[0143] Based on the image of the target structure and the initial positioning of the target structure within the line of sight, the system may determine which movements to initiate and / or proceed with.

[0144] Instructions may be given on how to rotate the C-arm about the C-axis and how to move the C-arm up and down. If the target structure is not positioned at the center "C" of the C-arm, translation instructions may be given on how to restore its previous position in the X-ray image or how to reach the desired position. Instructions may be given on how to rotate the C-arm about its vertical axis. This may take into account typical C-arm geometry (the distance between the vertical axis and the center of the "C" of the C-arm), or the system may learn the specific C-arm geometry from previous rotations. If the target structure does not appear in the desired position in the X-ray image or no longer appears in the desired position after rotation, instructions may be given on how to translate the C-arm. If the position of the target structure is already correct in the X-ray image and remains correct after rotation (e.g., if the C-arm is moved by its wheels so that the target structure remains in the center of the "C" of the C-arm), no translation instructions are given.

[0145] This procedure may be applied to other axes of the C-arm as well, such as the propeller axis.

[0146] Regarding the optimized adjustment of the C-arm, the following general aspects can be mentioned as a summary:

[0147] First, it is intended to use only one means for translating or rotating the C-arm device in order to adjust the position and orientation of the imaging direction as close as possible to, or at least with sufficient accuracy to, the target imaging direction, i.e., the optimal direction.

[0148] If necessary, it may be suggested to use a second means to further adjust the imaging direction. Additional means may be utilized to further increase accuracy.

[0149] The current imaging orientation, i.e., the starting point for the adjustment, may be determined based on the location of the object or structure in the x-ray image generated using the current imaging orientation. Of course, the object or structure may be a substructure, and may also be an anatomical structure, an implant, or a device / instrument, or a combination of such objects.

[0150] Target imaging directions may be identified based on their known geometric aspects, 3D positions, and 3D orientations, where such geometric aspects may be known from preoperative planning or the geometric aspects are available from a database.

[0151] Additionally, the position and orientation of the object may generally be known relative to the C-arc of the C-arm based imaging device. In a typical operating room setting, the anatomical structures may be positioned relative to the C-arm device in a known manner that allows for prediction of the imaging direction. The system may optionally provide information for confirmation to the user.

[0152] Alternatively or additionally, the system may learn how users tend to use C-arc translations and rotations and take that usage into account. For example, the system may calculate a rotation axis or translation of the C-arm device relative to the object from two images, where the imaging direction is rotated or translated about the rotation axis between image generation. In particular, the system may learn whether users tend to move the C-arm device approximately the amount currently instructed and take that into account when providing further instructions to move the C-arm device.

[0153] Example of a potential processing workflow for a distal fixation procedure The following distal fixation procedure describes a long antegrade nail. Nevertheless, it may also be applied to a proximally fixed retrograde nail. The following fixation procedure is provided for a hole whose axis is approximately in the ML direction, but may also be applied to a hole whose axis is in a different direction, for example, in the AP direction.

[0154] For such procedures, it may be assumed that a complete 3D model for the target object (i.e., the nail) is available. Nevertheless, the procedure may be performed successfully when only an incomplete or partial 3D model is available, for example, when only approximate information about the shape of the nail (e.g., a cylindrical object whose diameter slightly decreases toward its tip, including a cylindrical fixation hole) is known.

[0155] In the following, the fixation hole will be referred to as the "target structure." In principle, it is sufficient to know the relative 3D position and 3D orientation between the instrument (e.g., a drill, an awl, a sleeve, or even an implant such as a screw) and the target structure. Therefore, in the following, both the target object (nail) and the target structure (fixation hole) will be considered. In this description, it is assumed that a circular hole is first fixed.

[0156] 1. The user acquires an X-ray image of the nail in the approximate ML direction.

[0157] 2. The system may determine the imaging direction of the target structure (e.g., by detecting or locating the target object (in 2D)). The system may search for or determine a target trajectory (or possibly a target plane) relative to the target object or structure. The system may then determine and inform the user how to adjust the C-arm to reach the desired imaging direction of the target trajectory. Often, the desired imaging direction is aligned with the target trajectory, where the distal fixation hole is depicted as a circle. Figure 20 is an x-ray image in which the nail labeled 20.N is visible along with a non-circular fixation hole labeled 20.H. Therefore, the imaging direction is not the intended ML imaging direction. Figure 21 is an x-ray image in which the nail labeled 21.N is viewed from the true ML imaging direction, as evidenced by the circular fixation hole labeled 21.H.

[0158] It is desirable that the nail shaft passes through the image center and that the fixation holes are near the image center. This is desirable if there are additional holes that can be fixed without readjusting the C-arm. Ideally, these holes should therefore be on the central X-ray beam. The C-arm adjustment may be repeated, completed with a new X-ray image that meets the requirements.

[0159] 3. At this point, the system may highlight the center of the fixation hole on the X-ray image where the instrument must be aimed. This highlighted (target) point is on the target trajectory, which in the described scenario is the center of the circle. The system then detects the tip of the instrument in 2D and calculates the required movement of the tip to reach the target point. The system may support the user in an iterative process (each iteration consisting of acquiring a new X-ray image and repositioning the instrument) to reach the target point. Figure 22 shows X-ray images of a nail (22.N) and a drill (22.D) with an incorrectly positioned drill tip. Figure 23 shows X-ray images of a nail (23.N) and a drill (23.D) with an correctly positioned drill tip.

[0160] 4. Once the tip of the instrument (here a scalpel) is located at the point highlighted in the X-ray image, the surgeon may make the incision and insert the drill (optionally with a soft tissue protection sleeve) and repeat step 3. The user may then decide to align the drill with the target trajectory (as in conventional procedures) without moving the drill tip.

[0161] 5. The C-arm is rotated, for example, 25° around the C-axis, and a new X-ray image is acquired. The system may relocate the target object (or possibly just the target structure). Based on the a priori knowledge that the instrument tip is on the target trajectory (remaining in a known 3D position and 3D orientation relative to the target object / structure), the relative 3D position and 3D orientation between the drill and the target object / structure may be determined. Even though the tip of the drill in the distal-proximal direction is no longer located exactly on the target trajectory, the system can calculate the corresponding deviation. This is because it may be sufficient to have a priori information that the tip of the drill is located in a plane spanning the target trajectory and the nail axis when the C-arm is rotated around the C-axis.

[0162] At this point, the system may calculate any deviation from the target trajectory and may communicate this to the user, for example, by displaying the required angular corrections in the proximal-distal and anterior-posterior directions. If necessary, the system may also instruct the surgeon on how to adjust the instrument tip position in the proximal-distal direction. The system may also calculate the penetration depth, in this case, for example, the distance between the drill tip and the nail, and may communicate this to the user. This step 5 may be repeated by acquiring new x-ray images, providing information / instructions to the user, and re-adjusting the instrument.

[0163] The user may be informed on a display and / or audibly. An advantage of audible information may be that the surgeon does not have to look away from the drill and therefore achieves the correct drilling direction with fewer repetitions.

[0164] 6. Step 5 may also be performed during drilling to adjust drilling direction and / or to obtain information on how far to drill further (this is regardless of the fact that in a typical distal fixation situation, drilling continues down to the next cortex after hitting the nail hole).

[0165] Fixing to additional holes (e.g. oblong holes):

[0166] To save time and reduce x-ray exposure, the following procedure is provided for fixation of additional holes (hereafter assumed to be oblong holes) after fixation of the first hole as described above.

[0167] Assuming the target trajectory for the oblong hole is in the same plane as the first hole, rotate the C-arm back to its original position (where it was before step 5) showing the first hole (with or without the screw) as a circle. Thus, the nail axis again passes through the center of the X-ray image, and the oblong hole is close to the image center. No readjustment of the C-arm is necessary unless correction of the rotation about the C-axis is required because the original angle was not reached accurately enough. Thus, the oblong hole appears at its largest diameter in the AP direction but is compressed vertically. On the other hand, if the target trajectory for the oblong hole is not in the same plane as the first hole, the required readjustment of the C-arm may be supported by the system as described above.

[0168] Since the system knows the approximate distance between the bone surface and the nail in the medial-lateral position from fixation in the first hole, this value (and possibly a statistical model of the bone) may be used to correct the target position of the drill tip (see step 3 above). Thus, to hit the oblong hole that is centered (both AP and distal-proximal), the target point in the 2D X-ray image does not appear centered, but has shifted in the distal-proximal direction.

[0169] In Figure 17, the circular nail hole, designated 17.RH, is perfectly circular in 2D, i.e., X-ray. The tip of the opening tool (designated 17.OT) has a specific distance from the center of the oblong nail hole due to its position on the bone surface, so it is not at the 2D center of the oblong nail hole, but its tip is perfectly positioned on the central trajectory of the oblong nail hole in 3D. The two black arrows designated 17.C1 and 17.C2 indicate chamfers, which appear different sizes on either side of the oblong nail hole in perspective.

[0170] Any potential inaccuracies in the distal-proximal direction may not be an issue, since the inaccurate positioning of the instrument tip in the distal-proximal direction is detected and calculated after (approximately) aligning the instrument with the target trajectory, rotating the C-arm about the C-axis, and acquiring another X-ray image. Instructions may then be given to correct the instrument tip position in the distal-proximal direction, if necessary. As discussed above in step 5, it may be sufficient for the instrument tip to be in a plane spanning the target trajectory and the nail axis. The remainder of the procedure follows the steps for the first hole.

[0171] If the oblong hole is tilted relative to the nail axis, the tilt may be compensated for when rotating the C-axis with the potential fine adjustments supported by the system.

[0172] The entire discussion applies to circular holes as well. Furthermore, the same procedure can be followed to fasten further holes.

[0173] As discussed above, the view direction to the hole does not need to be perfect (i.e., in the case of a circular hole, for example, having the maximum projected hole width and height be perfectly circular) in order to position the drill tip on the target trajectory. Depending on the available a priori information and how stringent the requirements for angle determination are (the less stringent they are, the smaller the drilled distance between the bone surface and the nail will be), the view direction to the hole may deviate more or less from the target trajectory. For example, if an AP image is acquired, the distance between the outer bone surface and the nail along the target trajectory may be approximately determined (or simply estimated). Based on this information and the outer X-ray image, the point in the 2D X-ray image where the drill tip must be positioned to be on the target trajectory may be calculated (and then displayed) based on the oblique view angle, as discussed above. This point does not need to be perfectly on the drill trajectory. Instead, a new target trajectory may be calculated using the deviation (in 2D spatial coordinates) from this point as determined in the 2D X-ray image based on the estimated or previously determined distance between the bone surface and the nail along the fixation trajectory. The new target trajectory may then be used in the next image to orient the drill. This may make it easier for the surgeon to position the drill tip at the correct point, as the drill tip does not need to be perfectly on target. It may also allow for greater accuracy in orienting the drill to hit the target hole.

[0174] Assuming that sufficiently accurate positions of points of a second object relative to the geometric aspects of the first object are known, image registration may enable 3D reconstruction and / or determination of relative 3D position and 3D orientation.

[0175] Deviations from the original target trajectory may also be resolved by the following method: in a first image, the instrument may be placed on an arbitrary point (which may or may not be on the target object), and in a second image (obtained from a different viewing direction), the instrument may be tilted (e.g., by aiming at the target trajectory), but the tip of the instrument remains in the same position. Based on the localization of the target object / structure, both images may be aligned, which may allow the determination of the 3D position of the point relative to the target object. This point may then be used to determine with sufficient accuracy the relative 3D position and 3D orientation between the instrument and the target object / structure.

[0176] In steps 5 or 6 above, if the tip of the instrument is blocked by the target object (e.g., a steel nail) and therefore localization is not sufficiently accurate, the system may provide instructions on how to make the tip of the instrument visible to the system. This may proceed by the system calculating and instructing how the C-arm needs to be repositioned (e.g., rotation about the C-axis instead of rotation about an axis perpendicular to the nail axis). Also, if the material of the instrument absorbs significantly more X-rays than the target object (e.g., a steel instrument, a titanium nail), this may be achieved, for example, by increasing the voltage and / or current, or by selecting different C-arm program settings.

[0177] Additionally or alternatively, the system may fit a statistical 3D model of the bone, thus determining the 3D position of the nail relative to the 3D position of the bone, thus enabling the determination of the required fixation screw length in 3D.

[0178] It should be noted that for a typical steel nail in which all fixation holes face the same direction, the fixation holes may not be visible in the X-ray image at rotations greater than 30–35° (as in step 5), meaning that the steel nail may not be locatable for rotations greater than 30–35°. On the other hand, instruments or implants, such as titanium, absorb much less radiation than, for example, steel instruments or implants. Therefore, for titanium nails, tilted holes create a concentration gradient at the hole boundary. This is shown in FIG. 18 for a 25° rotation away from the fixation plane and in FIG. 19 for a 45° rotation away from the fixation plane. This effect means that it may be possible to locate tilted titanium nails over a much larger angular range compared to steel nails. Another beneficial effect of titanium nails is that they may make the drill, which is typically made of steel, visible relative to the nail. This may increase the accuracy of locating the drill, for example, when the drill tip is close to the nail during drilling. It may also be possible to rotate the C-arm around a different axis, for example the propeller axis, where the X-ray typically shows the tip superimposed on the nail.

[0179] If the C-arm is rotated around a vertical axis instead of the C-axis in step 5, the system requires the drill tip to be in a plane whose normal is the nail axis and contains the target trajectory. In this case, the deviation of the drill tip from the target trajectory in the AP direction may be calculated.

[0180] Therefore, an alternative to the above workflow may be to acquire X-ray images from both viewing directions obtained by rotating about the C-axis away from the fixed plane and by rotating about the vertical axis. In this case, no a priori information is required about the position of the drill tip relative to the target trajectory. Therefore, positioning the C-arm in the true ML direction is not required.

[0181] Examples of potential processing workflows for sacroiliac (SI) or pedicle screw placement The target object and target structure may also be anatomical. An example of an anatomical target structure is a vertebral pedicle. It may be sufficient to achieve the required accuracy for the target structure and therefore the target trajectory, taking into account the 3D reconstruction and the relative 3D position and 3D orientation to the instrument.

[0182] The procedure is similar to the distal fixation procedure, except that the anatomical target structure is localized, which may proceed with or without a deterministic 3D model. The deterministic 3D model may be obtained either preoperatively (e.g., a preoperative CT scan) or intraoperatively (e.g., an intraoperative CT scan or O-arm). If a deterministic 3D model is not available, a statistical 3D model (e.g., a statistical shape or appearance model) may be used for 3D reconstruction, as discussed above in the section "3D Reconstruction and Localization of Anatomical Objects Based on a Single X-Ray Image."

[0183] In this procedure, the instrument may be manually held rather than fixed to the target point of the anatomical structure and then roughly aligned with the target trajectory. The C-arm may be rotated, for example, 25° around the C-axis without acquiring a new X-ray image. After this iterative process, a new X-ray image is acquired, and the system may calculate the relative 3D position and orientation between the instrument and the target structure / object, taking into account that the tip of the instrument is on the target trajectory. This assumes that the viewing angle to the instrument is within a range that allows for sufficiently accurate determination of the 3D position and orientation between both objects. If only a statistical model of the anatomical structure is available, this process involves 3D reconstruction of the target structure / object.

[0184] In the next step, the system may calculate the deviation between the instrument axis and the target trajectory in terms of angular deviation, and optionally the deviation in tip position in a direction parallel to the rotation axis of the C-arm used for rotation between both images. The system may also calculate the penetration depth of the drill. The system may provide this data to the user (e.g., in the form of two angle values, the translation required for the instrument tip position, and the remaining insertion depth). The user may then readjust the instrument accordingly, acquire new X-rays, and / or drill / insert the instrument. Because the a priori information and lateral constraints remain unchanged, the procedure also works well when drilling has already begun and the instrument has already penetrated the anatomy.

[0185] Example of a potential processing workflow to determine the 3D position and orientation between the instrument and the anatomical structure based on image registration for increased accuracy By adjusting (translating and rotating) the C-arm bearings to align with specific portions of the anatomy (e.g., narrow passages such as pedicles), the tip of an instrument (e.g., an implant such as a drill, k-wire, or Jamshidi needle, or even a screw) may be placed on a specific anatomical reference point. This process may be supported by the system by displaying the reference point in the acquired 2D X-ray image, or alternatively, the surgeon's identification of the reference point may be used by the system to enhance its accuracy. The C-arm is then rotated, for example, 20-30° around the C-axis (or a comparable rotation around the propeller axis) while leaving the instrument in place, and another X-ray image is then acquired. The fact that the instrument contacts the surface of the anatomical object at the reference point may be used to reduce or even eliminate ambiguity caused by the instrument's ambiguous localization. The movement of the C-arm relative to the previous image may then be determined, and thus the viewing direction to the anatomical object may be determined with high accuracy. This requires that the instrument not move between x-ray images, which can make it easier to fixate the instrument to the anatomy, and this can be done not only with a drill but also with a Jamshidi needle or k-wire.

[0186] procedure: 1. Based on the preoperative CT scan, reference points and reference trajectories (i.e., intended drilling or insertion trajectories) and target endpoints may be planned prior to surgery. This may include planning along the intended imaging direction of the C-arm, e.g., true lateral or true AP, or along the pedicle or other easily recognizable landmarks. This step 1 may also be performed automatically by the system and / or during surgery (using intraoperative 3D imaging) with user interaction.

[0187] 2. During surgery, improved localization accuracy may be achieved by using either a predefined or online-calculated C-arm imaging orientation. The system may provide user instructions to help achieve the required C-arm bearing by detecting the relative positions of specific anatomical features, such as edges or points (see, e.g., patent application by Blau, filed August 23, 2018). The system may display a reference point in the X-ray image based on matching the entire structure or object from the CT scan to the X-ray image. The surgeon may then position the tip of the instrument (e.g., a drill, Jamshidi needle, or k-wire, or even an implant such as a screw) on this reference point, which may also be supported by the system by detecting the tip of the instrument in the 2D X-ray image. If necessary, the instrument is intentionally held at an angle such that the instrument (or power tool) and the surgeon's hands do not block the view.

[0188] 3. The instrument is then aligned in approximately the desired orientation.

[0189] 4. If possible, secure the instrument to the anatomy using designated markings on the instrument to allow accurate penetration depth to be determined.

[0190] 5. Acquire another X-ray image from the same imaging direction. At this point, the position of the instrument tip is constrained because it is located approximately on the drilling or insertion trajectory. If the instrument penetration depth is precisely known, there is less ambiguity than if the instrument penetration depth is unknown. The system may check (e.g., by image difference analysis) whether the anatomical structures shown in the X-ray image remain unchanged. Alternatively, in this step, it may be sufficient for the user to simply indicate in which plane (relative to the anatomical structures) the instrument tip is located. If step 5 cannot be performed because the instrument or the surgeon's hand blocks the view, steps 4 and 5 are performed without aligning the instrument in step 3.

[0191] 6. Move or rotate the C-arm to a different position. Based on the localized and fixed instruments, the C-arm movement relative to the previous image may be determined. Together with the localized anatomical structures, this allows for the determination of the relative 3D position and 3D orientation between the instruments and the anatomical structures. The determined quantities may be optimized together.

[0192] 7. If the instrument has such a small diameter that it can be located with sufficient accuracy only for a specific angle, the instrument must be viewed at a suitable angle (e.g., within a range of 10 to 55°) in all acquired X-ray images to be registered. Furthermore, the anatomical structure may be located more accurately for a specific imaging direction. Therefore, the accuracy for determining the relative 3D position and 3D orientation in step 7 may be increased by selecting a specific imaging direction to the anatomical structure. However, since such a specific imaging direction is typically true AP and true ML, this means that the angle between the two X-ray images is close to 90°. Furthermore, when fixing the instrument in step 4, it must already be ensured that the tip of the instrument is viewed at a suitable angle (e.g., within a range of 10 to 55°) in all acquired and registered X-ray images. Therefore, in such cases, the appropriate angle for fixing the instrument is midway between the true AP viewing direction and the true ML viewing direction, i.e., approximately 45°.

[0193] 8. Once the X-ray image is acquired and the above conditions are met, the system may calculate the deviation between the axis of the instrument and the reference trajectory and provide this to the user (e.g., by displaying two angle values). The user may then retract the instrument to the original reference point and realign it with the reference trajectory. The system may assist the user in locating the original reference point.

[0194] Reaching the correct reference trajectory may require an iterative process of acquiring additional X-ray images. After acquiring a new X-ray image, the system may then verify (e.g., by image difference analysis) whether the anatomical structure is still shown in the X-ray image with the same orientation and position. If so, the system may again calculate the relative 3D position and 3D orientation between the instrument and the anatomical structure with the a priori information that the instrument tip is on the reference trajectory. The system may also calculate and therefore communicate to the user the penetration depth, in this case, for example, the distance between the instrument tip and the target endpoint. During the insertion of the instrument, additional X-ray images may be acquired, and the above process may be repeated.

[0195] Also, if it is not feasible or desirable to ensure that the instrument is visible within a preferred angle range (e.g., 10–55°) on all acquired x-rays, the power tool holding the instrument (e.g., a power drill holding a drill) may be removed (as is evident when using k-wires or Jamshidi needles). If the entire instrument (tip and base) is visible in the x-ray image, the length of the instrument's projection on the x-ray image may be determined, thus allowing for sufficient accuracy in locating the instrument. In such cases, the instrument may even be viewed at an angle approaching 90°. This allows the instrument to be initially fixed at approximately the correct angle (eliminating the need to initially fix the instrument at an intentionally incorrect angle; see step 7), thus reducing the number of iterations required in step 8.

[0196] For pedicle screw insertion, it may be possible to identify the pedicle entry point in an AP view, where the pedicle axis is tilted, for example, at 10–45° relative to the viewing direction. With such an imaging orientation, the drill does not obstruct the view, and there is no need to acquire a second image from another direction after positioning the instrument tip over the anatomical entry point and angulating the instrument axis with the target trajectory. Because this procedure often requires a k-wire or Jamshidi needle, the instrument may be anchored in the bone with its axis already aligned with the desired target trajectory.

[0197] If desired, the user may then acquire one or more x-ray images from other viewing angles, which the system may use to perform the image registration described above to improve accuracy. Furthermore, if desired, the drill angle may be further optimized based on the additional information (possibly without backing out the drill), and the drill may proceed.

[0198] Due to the fact that in this first AP view the other pedicles have a mirrored inclination relative to the first pedicle (see Figure 15 showing two Jamshidi needles 15.JN and 15.JN2), it is possible to repeat the above procedure for other pedicles of the same vertebra and utilize the already inserted Jamshidi needle of the first pedicle (by using the projection in that X-ray image) to allow for more robust registration of the images.

[0199] Flowcharts in Figures 24 to 26 Figure 24 shows a general flowchart that covers all of the procedures shown above in the sections "Example of a Potential Processing Workflow for a Distal Fixation Procedure," "Example of a Potential Processing Workflow for Sacroiliac Joint (SI) or Pedicle Screw Placement," and "Example of a Potential Processing Workflow for Determining 3D Position and Orientation Between Instruments and Anatomical Structures Based on Image Registration for Accuracy." There are two possible implementations: a rapid implementation shown in Figure 25 that is applicable for the procedures shown above in the sections "Example of a Potential Processing Workflow for a Distal Fixation Procedure" and "Example of a Potential Processing Workflow for Sacroiliac Joint (SI) or Pedicle Screw Placement," and an improved accuracy implementation shown in Figure 26 that is applicable for the procedures shown above in the section "Example of a Potential Processing Workflow for Determining 3D Position and Orientation Between Instruments and Anatomical Structures Based on Image Registration for Accuracy."

[0200] It will be understood by those skilled in the art that not all steps must be performed, and in fact additional substeps not mentioned may be performed depending on the context of the specific application of the teachings provided herein.

[0201] The steps in Figures 24, 25 and 26 are as follows.

[0202] S10: Generate and load a 3D model. S11: Preoperative planning (optional). S12: Load the entire 3D model. S13: Intraoperative automatic determination of one or more target trajectories / one or more planes and, if applicable, target points (e.g., for anatomical structures).

[0203] S20: Supports C-arm adjustment. S21: Obtain an X-ray image. S22: Support for reaching a specific viewing direction to a target object, e.g., a circular hole (potentially supported by nail localization), or a true AP / ML view of an anatomical structure (potentially supported by DNN), by providing, for example, the rotation angle (including direction) around the C-axis, the rotation around the propeller axis, etc. S23: In case of ambiguity, the system provides only the value of the rotation angle without including the direction. S24: If the viewing direction is not close enough to the desired viewing direction, the user follows the system instructions and continues with S21. If a corresponding 3D model of the anatomical structure (e.g., a CT-scan, i.e., deterministic) is available, in this case the target trajectory can be obtained from the current viewing direction and the 3D model, and the desired viewing direction may differ from the target trajectory, knowing that the tip of the opening instrument will be placed on the anatomical structure. Example: distal fixation, where the opening instrument is placed on the femur and a 3D model of the femur is available. After locating the target object (S33), the system calculates the intersection of the target trajectory of the nail model with the surface of the femur model. Using the 3D model of the anatomical structure, the system provides adjustment instructions for the opening instrument tip (S37).

[0204] S30: Supports mouth opening device positioning. S31: Positioning of mouth opening devices. S32: An X-ray image is acquired. S33: Localization of target objects / structures. S34: If the target trajectory is well aligned with the viewing direction, the target point is directly visible in the 2D X-ray image (one DoF for undefined instrument tip position). Go to S36. S35: The system displays the 3D surface intersection of the anatomy (including the distal fixation) and the target trajectory (all DoFs for the defined instrument tip position) superimposed on the 2D X-ray image. S36: 2D matching of mouth opening devices. S37: The system provides instructions to the user to assist in adjusting the tip of the mouth tool. S38: If the position is not reached with sufficient accuracy, the user follows the system instructions and continues with S32.

[0205] S40: Determine the 3D position and orientation between the aperture instrument and the target object to align the aperture instrument with the target trajectory. S41e: Fixation of required opening appliances. S411e: Fixation of the aperture device. If S44 is intended to be applied, the system provides support for fixing the aperture device to the target object at an angle that ensures that the angle between the aperture device and all special viewing directions is less than 65°. Two angle values ​​are provided. If the user acquires another image from the same viewing direction, the fixation angle of the aperture device is confirmed by the system. S412e: Acquire an X-ray image that does not include changes in the relative position between the C-arm and the anatomical structure. S413e: Image difference analysis to determine penetration depth of opening instruments. S414e: Determine the 3D position and 3D orientation of the mouth appliance relative to the anatomical structure. S41q: Does not fixate mouth opening devices. Does not assist user in aiming trajectory. S42: The system calculates and displays the adjustment value for the C-arm rotation to reach a 25° angle between the viewing direction and the target trajectory. In case of ambiguity, the system provides only the value of the rotation angle without including the direction. S43: The user positions the C-arm according to the displayed adjustments and acquires the X-ray image. If the viewing direction is not close enough to the desired viewing direction from S42, go to S42. S44: Calculate the 3D position and 3D orientation between the target object and the aperture tool for final aperture tool adjustment instructions. S441e: In case of iterative optimization, 3D localization of the aperture device and calculation of the transformation matrix between the 3D position and 3D orientation of the aperture device between the current image and the previous special viewing direction. S442e: Potentially improved 3D orientations and 3D positions of the anatomical structures based on (i) the above transformation matrix (S441e), (ii) all previous 3D orientations and 3D positions of the anatomical structures along with the current orthodontic device position, and (iii) the current 3D orientations and 3D positions of the anatomical structures (either iteratively or jointly optimized). S441q: For iterative optimization, localize the target object. Based on the localization of the target object and the a priori information that the tip of the opening tool is positioned on the target trajectory / plane, determine the 3D position and 3D orientation between the opening tool and the target object. Proceed to 443. S442q: Joint optimization of 3D orientation and 3D position of a target object relative to an aperture instrument. S443: Confirm and correct a priori information (position of tip of opening instrument relative to target object) if, for example, distal-proximal deviation of nail can be confirmed and corrected. S444e: If the accuracy of the 3D position and 3D orientation between the anatomical structure and the mouth instrument needs further improvement, the system calculates and displays adjustment values ​​for the C-arm rotation to reach a more specific viewing direction. S445e: The user positions the C-arm according to the displayed adjustments and acquires the X-ray image. If the viewing direction is not close enough to the desired viewing direction, proceed to S44. S45: The user moves the opening tool by the provided adjustment to align the opening tool with the target trajectory. S451: As the 3D model of the target object provides the target trajectory, the system provides angles (two angles including a direction) for adjusting the direction of the aperture instrument to align the instrument with the target trajectory, obtained from the 3D position and 3D orientation of the instrument determined above relative to the target object. S452e: If the opening instrument is still fixed in the first position (S41e), the user retracts the opening instrument until its tip is positioned on the target trajectory, then aligns the opening instrument based on the system output and acquires an X-ray image (alternatively, use a second opening instrument to aim at the target trajectory, in which case proceed to S44). S453e: The system compares the images (e.g., by image difference analysis). If the images are locally close enough (for the target object), proceed to S44. S454: If the alignment of the aperture instrument with the target trajectory is not close enough, the user positions the aperture instrument based on the system output, acquires an X-ray image, and continues with S44. S455: If the remaining alignment instructions provide a sufficiently small value, the system displays information about how far the drill is. The user may align the drilling instrument based on the alignment instructions, and the user decides whether to acquire another X-ray image. S46: Drilling holes. S461: The user drills a hole. S462: If at any time the user wishes to verify the drill direction or drill depth, the user acquires a new X-ray image and continues with S44.

[0206] It should be noted that if the system can provide instructions on how to adjust the orientation (and possibly position) of the instrument already present in the first X-ray image, these can be used to reach a temporary alignment of the instrument with the target trajectory. After this alignment, if the instrument tilt is already within the required angular range and neither the power tool nor the surgeon's hands obstruct the view, another image from a different imaging direction may not be required, and if necessary, an X-ray image from the same imaging direction may be used to confirm the applied instructions (regarding the instrument orientation and possibly position). Another X-ray image may not be required if (i) no correction or only a very small correction is required, or (ii) a device is used that ensures sufficiently accurate application of the given instructions. Such a device may be a manual device or a robot.

[0207] As described in the "Method for Positioning a Target Object / Structure in the C-Arm's Field of View from a Desired Viewing Direction" section, perfect alignment of the C-arm in the direction of the target trajectory may not be required, especially if the instrument is positioned on the target object and the target point is identifiable in the current X-ray. It may be possible that all the information necessary to calculate the relative 3D position and 3D orientation is already available in the very first X-ray image. Thus, depending on the setup, the first X-ray image may be sufficient to perform the entire distal fixation procedure. An example of such a setup is the employment of a robot holding the instrument at a given point that already takes into account the required tilt. If it is already possible to identify both the target trajectory and the required starting point in the first acquired X-ray image and determine the 3D position and 3D orientation between the instrument and the target object, the robot may translate and rotate the instrument as needed and drill the hole. In the general case, if the target trajectory and resulting starting point for drilling cannot be identified based on the first X-ray, another X-ray may be acquired from a different, preferred viewing direction. Based on the localization of the target object, both X-ray images may be registered, which may allow the 3D position and 3D orientation of the instrument to be calculated relative to the target object and thus also relative to the target trajectory. Thus, the entire repositioning (translation and rotation) including drilling may be performed by the robot. This procedure is not limited to drilling.

[0208] Reduction support When reducing a proximal femur fracture anatomically, it may occur that the reduction appears correct on both the A-P X-ray (e.g., Adams' arch appears intact) and the lateral image, yet a residual dorsal gap is present. For this reason, it may be advisable to obtain a true ML X-ray, as this image has the greatest probability of showing such a gap. However, it may still occur that even a true ML image does not reveal such an incorrect reduction.

[0209] Furthermore, the remaining dorsal gap, which is not visible on x-ray, has limited degrees of freedom, which means that it must be possible to correct an inaccurate reduction by rotating the medial fragment about the axis defined by the main fracture line.

[0210] Such correction may be achieved by the following procedure, which is shown here for the case of two fragments.

[0211] 1. The system loads a 3D model (typically obtained using a pre-operative CT scan) showing segmented 3D bone fragments. 2. The surgeon obtains an A X-ray image. 3. The system may optionally detect fracture lines as a reference. 4. The system may optionally determine a line approximating the major fracture line. 5. The surgeon reduces the fracture until it appears correct on the APR radiograph. 6. The surgeon rotates the C-arm to the ML position and acquires an X-ray image. 7. The system may determine the relative 3D position and 3D orientation between two bone fragments, thus assisting the surgeon in assessing the reduction and potentially determining the correct reduction.

[0212] For step 7, the system may use the a priori information that bone fragments touch each other along the anterior fracture line, which is actually defined in 3D and therefore its detection in step 3 above is merely arbitrary.

[0213] In this scenario, the two objects (fragments) are in contact not just at a single point, but also along a one-dimensional structure that approximates a line in 3D space. Therefore, there is only one degree of freedom that is undefined (i.e., rotation about the anterior fracture line). Therefore, using the ideas presented in the previous section, it is possible to determine the relative 3D position and 3D rotation between the objects.

[0214] Greater accuracy can be obtained by combining this method with other previously described techniques (i.e., registration of numerous x-ray images with or without additional instruments / implants (e.g., nails) in the x-ray images). This may not even require any additional effort if performed after inserting the nail, which may be used to help record both images.

[0215] It should be noted that this kind of a priori information exists in some reduction scenarios in orthopedic trauma, for example in determining the varus / valgus position of the fragment. A further example is the scenario where it is known (e.g., based on an X-ray), i.e. The fragments are in contact with each other (the least restrictive kind of a priori information) If fragments are touching each other, it may be sufficient to know that their position is in one of several possibilities; for example, if the reduction looks correct on the A-P X-ray, it may be assumed that the fragments are touching along the fracture lines in either the dorsal or ventral direction; in a more extreme scenario, the dorsal fracture line of one fragment may be touching the ventral fracture line of another fragment; the algorithm may then evaluate all these possibilities and select the one that provides the best 3D match. How the pieces touch each other (in 1D structures such as points, lines, etc. or 2D structures such as planes, etc.) It may also be related to.

[0216] Note that the procedure shown above may also be applied to more than two fragments, and may be used to relate bone fragments A and C if there is a known relationship between bone fragments A and B and a known relationship between bone fragments B and C.

[0217] The system may also automatically determine whether any detected anatomical gaps between bone fragments are within normal limits and whether any protruding bone fragments are statistically significant because they deviate significantly from the fitted statistical shape model.

[0218] Another example in which a surgeon may incorrectly believe a reduction is correct based on what is seen in an x-ray image is the multi-fragment scenario of the proximal tibia. FIG. 27 shows an axial view of the proximal end of the tibia, including fragments A-E. Here, fragments labeled A-D are already anatomically reduced, but fragment E is submerged, i.e., displaced distally compared to the correct anatomical reduction. Such a situation may be difficult for a surgeon to determine because the x-ray (in either the AP or ML direction) shows many other fracture lines and also lines corresponding to regular anatomical structures (e.g., the fibula). The present invention may be able to detect such a scenario by accurately determining the relative 3D positions and orientations between all fragments. This may be possible because the system may use a priori information that all fragments are anatomically reduced, except possibly for the medial fragment (such as fragment E in FIG. 27), which may have been displaced distally. Here, the free parameters of the system are the proximal / distal positions of the fragments located within the bone.

Claims

1. 1. A system for processing x-ray images in computer-assisted surgery, the system comprising a processor and a software program product, the software program product, when executed by the processor, performing: receiving an X-ray image, the X-ray image being a projection image of a first object and a second object, the robot being configured to hold the second object and adjust a position and / or orientation of the second object; classifying the first object to receive a 3D model of the first object, determining geometric aspects of the first object, and identifying the geometric aspects with respect to the 3D model of the first object; classifying the second object to receive a 3D model of the second object, selecting points of the second object, and identifying the points in the 3D model of the second object; determining a relative 3D position and 3D orientation between the first object and the second object based on (i) the X-ray image, (ii) a 3D model of the first object, and (iii) a 3D model of the second object; The system is made to do this.

2. 10. The system of claim 1, wherein the system comprises the robot, and the software program product, when executed by the processing device, causes the robot to perform a bone treatment.

3. 3. The system of claim 2, wherein the bone treatment comprises at least one from the group consisting of inserting a screw, inserting a k-wire, drilling, making an incision, and reducing a fracture.

4. The system of claim 1 , further comprising a C-arm based X-ray imaging device for generating the X-ray image.

5. 5. The system of claim 4, wherein the C-arm based X-ray imaging device includes a robotic C-arm, and wherein the software program product, when executed by the processing device, causes the system to adjust the robotic C-arm.

6. 5. The system of claim 4, wherein the C-arm based X-ray imaging device includes a robotic C-arm, and when the software program product is executed by the processing device, the system is caused to acquire a new X-ray image with the robotic C-arm.

7. 7. The system of claim 1, wherein the geometric aspect of the first object is an aspect from the group consisting of a plane, a line, a point, an axis, a trajectory, an outline, a curvature, a center point, a 1-dimensional manifold, and a 2-dimensional manifold.

8. The system of claim 1 , wherein selecting a point on the second object comprises selecting a plurality of points on the second object.

9. 2. The system of claim 1, wherein the geometric side of the first object is a plane or a line, and the X-ray image is generated with an imaging direction tilted with respect to the geometric side at an angle within a range between 10° and 65°.

10. The system of claim 9 , wherein the angle is in the range between 20° and 30°.

11. 11. The system of claim 1, wherein determining the relative 3D position and 3D orientation between the first object and the second object is further based on knowledge of spatial relationships between points of the second object and geometric aspects of the first object.

12. 12. The system of claim 1, wherein the system is further configured to determine deviations of a 3D position and a 3D orientation of the second object from an intended spatial relationship of the second object relative to the first object.

13. 13. The system of claim 1, wherein the first object is a side of an anatomical structure or a first implant, and the second object is a side of an anatomical structure, an instrument, or a second implant.

14. 14. The system of claim 1, wherein the point of the second object is a distal tip of the object, and the 3D position information of the distal tip is a contact point of the distal tip with the surface of the first object.

15. 15. The system of claim 1, further comprising a device for providing information to a user, the information including at least one information from the group consisting of an X-ray image and instructions regarding a course of treatment.

Images