Determining the relative positions of objects in medical images based on artificial intelligence

Abstract

Methods and systems are provided that allow for classification of first and second objects in an X-ray projection image by matching models to match objects in the X-ray image, determining the respective representations and locations of the two objects, and determining the spatial relationship of the classified objects. Such methods and systems utilize artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer-assisted surgery. In particular, the present invention relates to an apparatus and method for determining a 3D representation and a relative 3D position and a relative 3D orientation of an object based on X-ray projection images. The method can be implemented as a computer program executable on a processing unit of the apparatus. Background Art

[0002] In the case of a fracture, the bone fragment can be stabilized by an implant such as an intramedullary nail, which can be inserted into the medullary canal of the bone, or a bone plate, which can be fixed to the surface of the bone as a support for fracture healing. The surgical procedure for implanting such an implant can be minimally invasive and may require repeated acquisition of X-ray images to enable the surgeon to correctly place the implant. The implant can also be connected to one or more sub-implants, such as screws or blades.

[0003] A particularly difficult step in this process is distal locking. The main difficulty in performing distal locking with long nails is the bending and twisting of the nail as the nail follows the medullary canal to a certain extent. This prevents the simple static mechanical locking procedure adopted in the case of short nails. Freehand distal locking is challenging and time-consuming, and may require the acquisition of many X-ray images. Therefore, some manufacturers provide flexible mechanical solutions that are suitable for the bending of the nail (hereinafter referred to as "long aiming devices"). Although the long aiming device simplifies the operation process, its application is still not direct because the X-ray image showing the long aiming device must be correctly interpreted and the C-arm position adjusted accordingly. Only after the C-arm is correctly adjusted can the long aiming device be appropriately adjusted.

[0004] Blau et al. (EP 2801320 A1) proposed a concept in which a reference body with metal markings at a fixed and known position relative to a long aiming device is detected and the imaging direction on the reference body is determined. Based on this, the system can give instructions on how to adjust the C-arm imaging device. A disadvantage of this system is that the X-ray image must contain the reference body. In order to adjust the long aiming device in the case of an antegrade femoral nail with a locking hole in the transverse direction, Blau et al. (US2013 / 0211386A1) used a reference body to determine the bending of the nail in the ML direction and the AP direction, respectively.

[0005] The latest product is described in a video available online: https: / / otaonline.org / industry-partners / stryker / multimedia / 16905832 / adapt-for-gamma3. In addition to a reference object at a fixed position to a long aiming device, this product also requires a reference body that is attached directly to the X-ray image intensifier.

[0006] For example, US 2014 / 0343572 A1 describes a method for computer-aided determination of correction parameter values ​​for positioning a medical device relative to a target structure or a reference base. US 2009 / 0209851 A1 describes a computer-assisted surgery system that processes two or more two-dimensional images of a region of interest taken from different angles to generate three-dimensional information related to the region of interest. WO 2008155772 A1 describes a system for measuring the true size and orientation of an object in a two-dimensional image. Summary of the invention

[0007] It is preferable to work without a reference body as this simplifies product development (e.g., if a new implant is being used), is more cost-effective, allows a workflow that more closely resembles a typical workflow in an operating room, and eliminates the additional uncertainty that applying a reference body's mechanical interface would introduce.

[0008] As described herein, the present invention proposes combining knowledge about the X-ray image generation process with artificial intelligence (in the form of so-called deep deformation and / or using neural networks) rather than any reference body to provide the information required when performing, for example, fracture treatment. Therefore, the purpose of the present invention can be seen as providing a 3D representation and relative 3D position and relative 3D orientation device and / or method that enables multiple objects to be at least partially visible in an X-ray projection image. Here, the object can be any object visible in the X-ray image, for example, an anatomical structure, an implant, a surgical tool and / or a part of an implant system.

[0009] The term "3D representation" may refer to a full or partial description of a 3D volume or 3D surface, and it may also refer to selected geometric aspects, such as radii, axes, planes, etc. The present invention may allow for determination of complete 3D information about a 3D surface or volume of an object, but the present application also contemplates disclosing methods for determining only selected geometric aspects.

[0010] Throughout this application, the terms "positioning" and "positioning position" refer to determining the 3D orientation of an object and determining the 2D spatial position of the projection of the object on the image plane. On the other hand, the imaging depth (i.e., the distance of the object from the image plane) is estimated based on prior information related to the typical community of objects in the operating room (e.g., the relative positions of implants, patients, and imaging devices). In the context of the present invention, this estimated imaging depth is sufficient. However, in some cases, the imaging depth can still be determined more accurately, as discussed further below.

[0011] The present invention can be used to achieve all the purposes of invention US 2013 / 0211386 A1 without any reference body. Using the technology described in the present invention, the long aiming device can be positioned together with the locking sleeve configuration. Based on this, instructions can be given on how to adjust the X-ray imaging device. The imaging depth of the long aiming device can be estimated based on geometric aspects of typical distal locking situations in the operating room. Then, the distal locking hole can be detected and its 3D position and 3D orientation relative to the locking sleeve configuration and / or the long aiming device can be determined.

[0012] Using known geometric aspects of the long aiming device and the relative position and orientation of the unbent nail, the actual position of the nail is compared to the projected position of the unbent nail. This allows the relative 3D position and 3D orientation between the target distal locking hole of the (possibly bent) implanted nail and the long aiming device to be calculated. This takes into account the fact that the proximal tip of the nail is held in place by the long aiming device. Therefore, the possible position of the tip of the bent nail describes the base of a cone with the apex at the proximal tip of the nail. The nail may also be subject to torsion.

[0013] Even if it is not visible or only partially visible in the X-ray image, a 3D representation and positioning of related objects such as an anatomical structure, an implant, a surgical tool and / or a part of an implant system can be provided. As an example, even if the projection image cannot fully depict the femur, it can be fully reconstructed.

[0014] It should be noted that the image data of the processed X-ray image may be received directly from an imaging device, such as from a C-arm based 2D X-ray device, or alternatively from a database. Aspects of the present invention may also be used to process medical images acquired using other imaging modalities such as ultrasound or magnetic resonance imaging.

[0015] The system proposed according to the present invention comprises at least one processing unit, which is configured to execute a computer program product including a set of instructions, which causes the device to (i) receive an X-ray projection image whose characteristics depend on imaging parameters, (ii) classify at least one object in the X-ray projection image, (iii) receive a model of the classified object, and (iv) determine a 3D representation of the classified object and position the classified object relative to the coordinate system by matching the virtual projection of the model with the actual projection image of the classified object. The process can take into account the characteristics of the X-ray imaging method. In particular, it can be considered that the intercept theorem applies, which will be discussed in the following examples.

[0016] The X-ray projection image can represent an anatomical structure of interest, in particular a bone. The bone can be, for example, a bone of the hand or foot, but can in particular be a long bone of the lower limb (e.g., femur and tibia) and a long bone of the upper limb (e.g., humerus). The image can also include an artificial object, such as a bone implant that has been inserted into or attached to the imaged anatomical structure of interest.

[0017] In the context of the present invention, a distinction will be made between "object" and "model". The term "object" will be used for real objects, for example, for a bone or a part of a bone or another anatomical structure, or for an implant, such as an intramedullary nail, a bone plate or a bone screw, or for a surgical tool, such as a sleeve, a Kirschner wire, a scalpel, a drill bit or an aiming device that can be connected to an implant. "Object" can also only describe a part of a real object (for example, a part of a bone), or it can be a collection of real objects, and is therefore composed of sub-objects. For example, an implant and an aiming device assembly can be considered to be composed of a sub-object "aiming device" and a sub-object "nail", wherein the object "nail" is subjected to bending while the object "aiming device" is not subjected to bending. As another example, the proximal portion of a nail can be considered to be an object, and one of the locking holes of the distal portion of the nail can be considered to be another object.

[0018] On the other hand, the term "model" will be used for a virtual representation of an object. For example, a data set defining the shape and size of an implant can constitute a model of the implant. As another example, a 3D representation of an anatomical structure, such as generated during a diagnostic process, can be considered a model of a real anatomical object. It should be noted that a "model" can describe a specific object, such as a specific nail, or can describe a class of objects with some variability, such as a femur. In the latter case, such objects can be described, for example, by a statistical shape or appearance model. Then, the purpose of the present invention can be to find a 3D representation of a specific instance from a class of objects depicted in an acquired X-ray image. For example, the purpose can be to find a 3D representation of a femur depicted in an acquired X-ray image based on a general statistical shape model of the femur. It is also possible to use a set of discrete deterministic possible models, and then the system will select which of these models best describes the object in the image. For example, there may be several nails in the database, and then the algorithm will identify which nail is depicted in the image (if the user does not provide this information in advance).

[0019] Since a model is actually a set of computer data, it is easy to extract specific information from this data, such as the geometry and / or dimensions of the virtually represented object.

[0020] The model may include more than one portion of the imaged object, at least one portion of which is not visible in the X-ray projection image. For example, a model of an implant may include a screw intended to be used with the implant, but only the implant has been introduced into the anatomical structure and therefore only the implant is visible in the X-ray projection image.

[0021] Also note that the model may not be a complete 3D model of the real object, as it only describes certain geometry of the object, e.g. a femur is approximated as a sphere in 3D and as a circle in a 2D projection image.

[0022] The appearance of the object in the projection image may be affected by the X-ray imaging process. For example, imaging parameters such as imaging direction relative to gravity, scaling, radiation intensity and / or the presence of magnetic fields may affect the appearance of the object in the projection image. These or additional imaging parameters may cause characteristic changes in the projection image, such as distortion of the projected object due to a pincushion effect, mechanical bending, curvature, noise and / or distortion of the C-arm imaging device depending on the imaging direction. Here, those changes are represented as image features.

[0023] It will be appreciated that those image features can be determined with sufficient accuracy in the projected image. For example, the location of structures shown in the edge regions of the image are more affected by the pincushion effect than structures in the center of the image. Therefore, the features of the pincushion effect can be determined with sufficient accuracy based on structures of known shape from the edge regions to the center region. Image features determined for a region in the 2D X-ray can be extrapolated to the entire image.

[0024] The effect of the Earth's magnetic field on image distortion depends on whether the C-arm's image intensifier is positioned horizontally or vertically relative to gravity (due to the effect of the magnetic field on the electron beam inside the image intensifier, see M. "Fluoroskopiebasierte Navigation zur intraoperativen Eingriffe,” dissertation, TU Munich, 2004). If the patient position (e.g., lateral, left lateral, prone, or supine) is known (e.g., provided as input by the user), and after the system has calculated the imaging orientation (e.g., using Blau's invention filed as a patent application on August 23, 2018), it can be determined whether the C-arm is positioned horizontally or vertically with respect to gravity. This provides prior information about the effect of the Earth's magnetic field on the distortion of the X-ray image. Another example of prior information that the system can use is the bending of the C-arm imaging device due to gravity.

[0025] Another method of utilizing the C-arm position may be to obtain initial measurements of image distortion for representative views (e.g., AP and ML) with corresponding C-arm positions, which may be performed before the patient is turned on and then saved by the system. If the position of the C-arm is known (determined using the method in the previous paragraph), the corresponding previously determined and saved image features may be applied to the current image and object matching.

[0026] Taking into account the characteristics of the image and the object and the effects of X-ray attenuation, absorption and deflection, the virtual projection of the model may be deformed and / or distorted, just as the object is deformed and / or distorted in the X-ray projection image. Such a virtual projection can then be matched with the projection seen in the X-ray image. It will be understood that the matching of the object in the X-ray projection image with the model can include the adaptation of the image features of the X-ray projection image to the image features of the virtual projection of the model and / or the adaptation of the image features of the virtual projection of the model to the image features of the X-ray projection image. It should also be understood that by minimizing the distance in 3D, matching in the 3D projection volume is also possible.

[0027] The physical size of an object is related to its projected size in an X-ray image by the intercept theorem (also called the fundamental proportion theorem) because the X-ray beam originates from the X-ray source (focal point) and is detected by the X-ray detector of the image in the plane. In the context of the present invention, an exact imaging depth (i.e., the distance of the object to the image plane) is generally not required. However, if the object is large enough, the imaging depth can be determined by the intercept theorem, and the larger the object, the more accurate the determination will be. Alternatively, if the size of the X-ray detector and the distance between the image plane and the focal point are known, the imaging depth can also be determined. If a more accurate imaging depth is required for certain applications, the size of the X-ray detector and the distance between the image plane and the focal point can be provided by user input or a database. The former can also be determined by a calibration step by placing a known object (e.g., a nail, a Kirschner wire, etc.) directly on the image intensifier / X-ray detector.

[0028] According to one embodiment, a deep neural network (DNN) can be used for classification of objects in an X-ray projection image (e.g., the proximal portion of the femur, the distal portion of the femur, the proximal portion of the nail, or the distal portion of the nail, etc.). It should be noted that the DNN can classify the object without knowing its position (e.g., see Krizhevsky, A., Sutskever, I. and Hinton, GE "ImageNet, ImageNet classification with deepconvolutional neural networks". In NIPS, pp. 1106–1114, 2012). Neural networks can also be used to roughly classify the imaging direction (e.g., AP vs. ML, see the following article: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jürgen Schmidt: "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge", 2018, available online at). This classification of objects and imaging directions can be used to select an appropriate model for subsequent processing steps.

[0029] According to an embodiment, the contour of the classified object can be detected in an X-ray image. For objects with variable shapes (such as anatomical structures), this can be done by using the "deep deformation" method described in the paper (2018) of Pries et al. cited above. The paper proposes a method based on a deep neural network to detect bone structures in fluorescent X-ray images. The technology specifically addresses the challenges in the automatic processing of fluoroscopic X-rays, namely their low quality and the fact that there are usually only a few data sets available for training neural networks. The technology incorporates high-level information about the object in the form of a statistical shape model. The technology includes a two-stage method (called deep deformation), in which in the first stage, the neural segmentation network detects the contour (outline) of the bone or other object, and then in the second stage, the statistical shape model is fitted to the contour using a variant of the active shape model algorithm (but other algorithms can also be used in the second stage). This combination enables the technology to mark points on the contour of the object. For example, in femoral segmentation, the technology will be able to determine which points on the contour in the 2D X-ray projection image correspond to the lesser trochanter area, which points correspond to the femoral neck area, etc. Images described by deterministic models (e.g., PEGs) can also be detected by deep morphing or, as in the first stage of deep morphing, by a neural segmentation network.

[0030] In another step, the virtual projection of the model can then be adjusted to match the appearance of the object in the X-ray projection image, taking into account the features of the image and / or the object and the effects of X-ray attenuation, absorption and deflection. According to one embodiment, for an object described by a deterministic model, the matching can be performed along the lines described in the following papers: Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps"; Laugier C. (eds) "Geometric Reasoning for Perception and Action"; GRPA 1991. Lecture Notes in Computer Science, vol. 708. Springer, Berlin, Heidelberg. In this method, the effects of image features and object features (especially bending and / or twisting of implanted nails) and X-ray attenuation, absorption and deflection can be addressed by introducing additional degrees of freedom in the parameter vector or by using a suitably adjusted model. When simulating bending of an implanted nail, it may be considered that the possible position of the tip of the bent nail describes the base of a cone with its apex at the proximal end of the nail and possible rotational axis (torsion) about the nail axis.

[0031] According to one embodiment, for objects described by a statistical shape or appearance model, the matching of the virtual projection with the actual projection can be carried out along the lines of the following paper: V. Blanz, T. Vetter (2003) "Face Recognition Based on Fitting a 3D Morphable Model", IEEE Transactions on Pattern Analysis and Machine Intelligence. In this paper, a statistical deformable 3D model is fitted to a 2D image. To this end, the statistical model parameters for contour and appearance and the camera and pose parameters for perspective projection are determined. It needs to be initialized with key points, which can be provided by the results of the deep deformation step. Another method can be in accordance with the following paper: L. Gu and T. Kanade, "3D alignment of face in a single image", in Proc. of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. This is a variation of the active appearance model for 2D-3D matching, which also determines the position of the camera and the statistical model parameters.

[0032] The neural network can be trained based on a plurality of data to be compared with the data to be applied on the neural network. In the case of evaluating bone structures in an image, the neural network should be trained based on a large number of X-ray images of the bone of interest. It will be appreciated that the neural network can also be trained based on simulated X-ray images. For example, simulated X-ray images can be generated from 3D CT data, as described in the appendix of the following paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jürgen Schmidt: Detecting bone structures in fluoroscopic X-ray images with prior knowledge, available online at https: / / arxiv.org / abs / 1808.04441. The realism of the simulated X-ray images can be enhanced by using so-called generative adversarial networks (GANs). In this approach, a fully convolutional neural network can receive simulated images as its input and then return a refined image. The refinement is learned by an adversarial loss function that rewards realistic refinement. This is described in the following paper: Ashish Shrivastav, Tomas Pfister, Afterl Tuzel, Sosh Susskind, Wenda Wang, Russ Webb: “Learning from Simulated and Unsupervised Images through Adversarial Training”, available online.

[0033] According to one embodiment, more than one neural network may be used, wherein each neural network may be specifically trained for the sub-steps necessary to achieve the desired solution. For example, a first neural network may be trained to evaluate X-ray image data in order to classify anatomical structures in a 2D projection image, while a second neural network may be trained to detect the location of the structure in the 2D projection image. A third network may be trained to determine the 3D position of the structure relative to a coordinate system. Neural networks may also be used in conjunction with other algorithms, including but not limited to active shape models. It should be noted that the neural network may also learn to locate an object or determine an imaging orientation without first detecting the outline of an object in a 2D X-ray image. It should also be noted that neural networks may also be used for other tasks, such as determining one or more image features, such as a pincushion effect.

[0034] According to one embodiment, objects can also be manually classified and / or identified in the X-ray projection images. Such classification or identification can be supported by the device by automatically referencing structures identified by the device.

[0035] According to one embodiment, the system can calculate the object's (e.g., axis, contour, curvature, center point) and the object's dimensions (e.g., length, radius or diameter, distance). This can be achieved due to the correspondence between the model and the virtual projection that has been matched to the projection seen in the X-ray image.

[0036] When the X-ray projection image is displayed, the geometry and / or size may be displayed as an overlay in the projection image. Alternatively and / or additionally, at least a portion of the model may be shown in the X-ray image as, for example, a transparent visualization or 3D rendering, which may aid the user in identifying structural aspects of the model and, therefore, of the imaged object.

[0037] It will be appreciated that the size may also define a scale suitable for measuring the size of objects visible in the X-ray projection image and the size between objects visible in the X-ray projection image. For example, the diameter of a matching object (referred to as "object A") indirectly provided by the model data may be used to determine the size of other shown objects and potentially unknown objects, at least for those objects that can be assumed to be located at a sufficiently close imaging depth. If the imaging depth of object A is known (e.g., because object A is large enough or because the size of the X-ray detector and the distance between the image plane and the focal point are known), and if there is information about the difference in imaging depth between object A and object B (e.g., based on anatomical knowledge), the size of a different object (referred to as object "B") at a different imaging depth may be calculated based on the truncation theorem.

[0038] The present invention provides 3D reconstruction and localization of objects whose shape and appearance have a certain variability. Such objects can be described, for example, by 3D statistical shape or appearance models. This can be done based on a single X-ray image or multiple X-ray images.

[0039] Based on an image of the anatomical object, the model is deformed in such a way that its virtual projection matches the actual projection of the object in the X-ray image. This enables the calculation of the imaging direction (describing the direction in which the X-ray beam passes through the object). As an additional plausibility check, the calculated imaging direction can then be compared with an imaging direction of the same object, which is determined based on the method using Blau's invention filed as a patent application on August 23, 2018, which takes the entire X-ray image (not just the specific object) into account. Since the two imaging directions should not be different, a plausibility check can be performed.

[0040] If multiple X-ray images are acquired, information from them can be fused (or recorded) to improve the accuracy of the 3D reconstruction and / or determination of spatial position or orientation. Preferably, the X-ray images are acquired from different imaging directions, as this can help disambiguate. The more different the imaging directions (e.g., AP and ML images) are, the more helpful the additional images are in terms of determining 3D information.

[0041] For the registration of a pair of X-ray images showing the same object (referred to as "object C") from two different imaging directions, the accuracy of the registration process can be increased by considering the 3D angle between the imaging directions (e.g., represented by Euler angles or other suitable representations). One way to determine this angle is to use the invention of Blau, filed as a patent application on August 23, 2018, to determine the imaging direction for each X-ray image and calculate their difference. Another way can utilize another object (referred to as "object D") that is also shown in both images, the model of which is deterministic (e.g., a nail connected to a sighting device). By matching the virtual projection of object D with its actual projection in each X-ray image, the imaging direction of object D can be determined. This requires that (i) there is no movement of object C relative to object D between the two X-ray images acquired, and (ii) in the case of movement, it can be determined as discussed later. This process can obviously be extended to the registration of more than two images. This makes it possible to determine the relative 3D position and 3D orientation between multiple objects depicted in the same X-ray image even if the imaging depth has not yet been determined.

[0042] The appearance of an object in an X-ray image depends, among other things, on the attenuation, absorption and deflection of the X-ray radiation, which in turn depends on the material of the object. The more material the X-ray beam has to pass through, the less X-ray radiation is received by the X-ray detector. This not only affects the appearance of the object within its contour, but can also change the shape of the contour itself in the X-ray projection image, especially in areas where the object is thin.

[0043] The strength of this effect also depends on the X-ray intensity and the amount of tissue surrounding the subject that the X-ray beam has to pass through. The latter depends on the patient's body mass index and the imaging direction (e.g. using the invention of Blau filed as a patent application on August 23, 2018). The amount of soft tissue surrounding the subject can be derived from a database that takes into account, for example, race, sex, body mass index, age.

[0044] Object features can also be determined. Object features can be, for example, bending and twisting of an intramedullary nail or fractures of a bone. These object features can be determined automatically, manually or semi-automatically, wherein the device can support, for example, manual determination by instructions on structures that are not expected when only image features are considered (as described above).

[0045] On the other hand, a priori known object features can be used to increase the accuracy of the positioning and determination of the 3D representation. An example is the shaking of the connection between the implant and the aiming device, which can be described by the model. It is understood that when object features are used in the calculation of the positioning and determination of the 3D representation, some features may be highly weighted compared to other features.

[0046] Note that the processing unit can be implemented by only one processor that performs all steps of the process, or by a group or multiple processors that are not necessarily located in the same place. For example, cloud computing allows the processor to be placed anywhere. For example, the processing unit can be divided into (i) a first sub-processor on which a first neural network is implemented to evaluate image data, the image data including, for example, the classification of anatomical structures of bone surfaces, (ii) a second sub-processor on which a second neural network is implemented, dedicated to determining the imaging direction of the classified anatomical structures, and (iii) another processor for controlling a monitor to visualize the results. One of these processors or another processor can also control, for example, the movement of a C-arm of an X-ray imaging device.

[0047] According to one embodiment, the device may further comprise storage means providing a database for storing, for example, X-ray images. It will be appreciated that such storage means may also be provided in a network to which the system may be connected and data relating to the neural network may be received via the network.

[0048] Furthermore, the apparatus may comprise an imaging unit for generating at least one 2D X-ray image, wherein the imaging unit may be capable of generating images from different directions.

[0049] The device may further include an input device for manually determining or selecting the position or part of an object in the X-ray image (e.g. a bone contour), for example for measuring a distance in the image. Such an input device may be, for example, a computer keyboard, a computer mouse or a touch screen to control a pointing device such as a cursor on a monitor screen, which may also be included in the device.

[0050] The computer program can preferably be loaded into a random access memory of a data processor. Therefore, a data processor or a processing unit of a system according to an embodiment can be equipped to perform at least a portion of the described process. In addition, the present invention relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program can be stored. However, the computer program can also be presented through a network such as the World Wide Web and can be downloaded from such a network into a random access memory of a data processor. In addition, the computer program can also be executed on a cloud-based processor, and the result is presented through the network.

[0051] It must be noted that embodiments are described with reference to different subject matters. Specifically, some embodiments are described with reference to method type claims (computer programs), while other embodiments are described with reference to device type claims (systems / devices). However, a person skilled in the art will deduce from the above and following description that, unless otherwise stated, any combination of features belonging to one subject matter and any combination between features relating to different subject matters are deemed to be disclosed in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The aspects defined above as well as further aspects, features and advantages of the invention can also be derived from the embodiments to be described hereinafter and are explained with reference to the embodiments shown in the drawings, but the invention is not limited to these embodiments.

[0053] Figure 1 An example of 3D registration of AP and ML images is shown.

[0054] Figure 2 An example of 3D registration of AP and ML images is shown and the impact of an incorrectly estimated C-arm width is illustrated.

[0055] Figure 3 Comparison Figure 1 and Figure 2 situation.

[0056] Figure 4 An example for 3D registration of an AP image and an ML image is shown, and the effect of scaling is illustrated.

[0057] Figure 5 Comparison Figure 1 and Figure 4 situation.

[0058] Figure 6 An example of 3D registration of AP and ML images is shown, and the effect of the X-ray receptor size is illustrated.

[0059] Figure 7 An undistorted proximal AP X-ray of the femur with the nail and aiming device is shown.

[0060] Figure 8 A distorted proximal AP X-ray of the femur with the nail and aiming device is shown.

[0061] Fig. 9 Comparison Figure 7 and Figure 8 .

[0062] Fig.10 An undistorted proximal ML X-ray of the femur with the nail and aiming device is shown.

[0063] Fig.11 A deformed proximal ML X-ray of the femur with the nail and aiming device is shown.

[0064] Fig.12 Comparison Fig.10 and Fig.11 .

[0065] Fig.13 The effect of neglecting distortion on the positioning of the nail and the aiming device is shown.

[0066] Fig.14 An incorrect 2D match of the nail and the aiming device outline is shown.

[0067] Fig.15 yes Fig.14 A magnified view of a portion of .

[0068] Fig.16 yes Fig.14 A magnified view of a portion of .

[0069] Fig.17 An example for 3D registration of an AP image and an ML image is shown, and the effect of receiving a PA image instead of an AP image is shown.

[0070] Fig.18 The effect of receiving a PA image instead of an AP image in the projection plane is shown.

[0071] Fig.19 The effect of receiving a LM image instead of a ML image in the projection plane is shown.

[0072] Fig. 20 An example of a possible workflow is shown.

[0073] Fig.21 The effect of using an incorrect focal point on the distal locking support is shown.

[0074] Fig. 22 The effect of receiving a PA image instead of an AP image on the distal locking support is shown.

[0075] Fig.23 A scheme for reducing evaluations is shown.

[0076] Unless otherwise specified, the same reference numerals and characters are used in all drawings to represent the features, elements, components or parts of the illustrated embodiments. In addition, although the present disclosure will now be described in detail with reference to the drawings, the present disclosure is completed in conjunction with the illustrative embodiments and is not limited to the specific embodiments shown in the drawings. DETAILED DESCRIPTION

[0077] 3D reconstruction of anatomical objects from one image

[0078] The above-referenced paper on Deep Morphing by Pries et al. (2018) proposes a method that enables a system to detect the contour / outline of a bone (in a 2D projection image) as well as landmarks on the outline. For example, in femoral segmentation, the technique is able to determine which points on the contour in a 2D X-ray projection image correspond to the lesser trochanter, which points correspond to the femoral neck, etc. Given a 3D statistical shape or appearance model of the same anatomy, it is then possible to deform the model so that its virtual projection matches the actual projection in the X-ray image, resulting in a 3D reconstruction of the anatomy and allowing the object to be localized and oriented for imaging.

[0079] Being able to provide a 3D reconstruction of an anatomical structure based on only one X-ray image is advantageous over prior art techniques, which require acquiring at least two images (usually AP and ML images) from different viewing directions. In addition, the present invention is also able to determine the 3D orientation of an implant or instrument relative to an anatomical structure based on one X-ray image.

[0080] Thus, two objects, at least one of which is an anatomical object, can be localized based on one X-ray image depicting the two objects. This means that the only remaining uncertainty is the unknown difference in imaging depth between the two objects. This allows solving surgical procedures where it is not possible to acquire a second image from a different imaging direction, e.g. because a bone fracture is not stable enough.

[0081] The accuracy of the 3D reconstruction of anatomical objects can be improved by using the necessary prior information. This prior information can be the size or gender of the patient, or it can be more anatomically specific information, such as geometric information about the anatomical object to be reconstructed. For the example of 3D reconstruction of the proximal femur based on ML images, this information can include the length of the femoral neck or the CCD angle. Such information greatly reduces ambiguity in the 3D reconstruction. However, since such information may not be determined with sufficient accuracy in typical ML images, this information can be extracted from AP images that are routinely acquired earlier during proximal femoral surgery. The more information used from the earlier images, the higher the accuracy of the 3D reconstruction. Another way to describe this process is to say that based on the AP image, a 3D reconstruction of the proximal femur can be performed using typical residual uncertainties (e.g., the width of the femoral neck along the AP direction), and this 3D reconstruction serves as prior information or as a starting point for 3D reconstruction based on later ML images.

[0082] Determine the relative 3D position and 3D orientation between objects without determining the imaging depth of any object

[0083] As discussed, it is usually impossible to accurately determine the imaging depth of a local object. However, if there is prior information about the 3D distance between two objects, the 3D orientation and 3D position of one object relative to the other can be determined.

[0084] Example: If two objects are known to be touching each other in physical 3D space, the relative 3D orientation and relative 3D position between the objects can be determined. This can be calculated by positioning each object and then virtually moving the model of one of the objects along a vector from the focus through the center of that object until it touches the model of the other object in virtual 3D space. If the area where the objects are touching each other is small enough, this will work even if their projections in the image do not overlap. In the special case where the two projections of the two objects do not overlap in the X-ray image, there is no need to virtually move the models, since both models are already at the same imaging depth.

[0085] The registration process of two or more X-ray images from different directions

[0086] Depending on the shape of the bone, a 3D reconstruction based on only one image may suffer from residual ambiguities or matching errors. This can be mitigated by acquiring a series of images, possibly from different viewing directions, which may be routinely acquired during surgery anyway (e.g., when repositioning a fracture). In general, additional images from different imaging directions are more useful, and the more different the imaging directions (e.g., AP images and ML images), the more helpful the additional images are in terms of determination of 3D information. However, even adding images from only slightly different perspectives - which may be more easily acquired during surgery rather than changing to a completely different view (AP to ML or vice versa) - can be beneficial.

[0087] Finally, another approach to improve the accuracy of the matching process could be to use preoperative imaging to generate information about the 3D shape of the patient's specific bone, rather than using general statistical models that describe bone variability.

[0088] The present invention allows registration of multiple X-ray images of at least one common object acquired from different directions. This is important because 3D registration allows determination of the relative 3D position of multiple objects without the need to determine the imaging depth.

[0089] For 3D reconstruction of an object of variable shape (typically an anatomical structure described, for example, by a statistical shape or appearance model, referred to in this section as "object F") based on two or more X-ray images, the process outlined above for one image can be extended to two or more images. That is, deep deformation can be used to detect the contour of object F and the points on the contour marked in each 2D X-ray image. Given a 3D statistical shape model of object F, the model can then be deformed in such a way that its virtual projection simultaneously matches as closely as possible the actual projection of object F in two or more X-ray images. This process does not require a priori information about the imaging direction, because it implicitly determines the imaging direction of each X-ray image.

[0090] As an alternative to registering a pair of X-ray images acquired from two different imaging directions, the accuracy of the registration process can be improved by taking into account the 3D angle between the imaging directions, which can be determined using two different procedures. The more accurately this angle can be determined, the more accurate the 3D registration can be.

[0091] One way to determine this angle is to use Blau's invention, filed as a patent application on August 23, 2018, to determine the imaging direction of each X-ray image and calculate their difference. Another way can be to use another object in the X-ray image called "object G" whose model is deterministic (e.g., a nail connected to a sighting device). By matching the virtual projection of object G with its actual projection in each X-ray image, the imaging direction of object G can be determined.

[0092] If the 3D angle between the imaging directions calculated from object G differs significantly from the 3D angle between the imaging directions of object F, this may indicate, for example, that the position of object G relative to object F has changed between the acquisition of the images (e.g., there has been a rotation about the nail axis). Such movement can be corrected automatically; if this is not possible, then a 3D registration should not be attempted. If the angles determined by the two processes are close enough, the results can be weighted and averaged (the weighting depends on, for example, image quality, detection quality, visibility, 3D reconstruction uncertainty), and then a 3D registration can be performed. This process can obviously be extended to register more than two images.

[0093] In the following, examples are given to illustrate the influence of the C-arm width, the size of the image detector, scaling, etc. on the 3D registration. It is shown that in all these examples, it is not necessary to determine the imaging depth.

[0094] Effect of C-arm width: Figure 1 The left femur (denoted as LF) and the nail implant with the aiming device (denoted as NAD) are depicted. In addition, it also shows the AP X-ray image (denoted as 1.AP) and the ML X-ray image (denoted as 1.ML) and their corresponding focal points (denoted as 1.FP.AP and 1.FP.ML). The 3D sphere approximates the femur (denoted as FH) and the white dashed circle is its 2D approximate projection in the image (denoted as 1.FH.AP and 1.FH.ML). The width of the C-arm (defined here as the distance between the focal point and the image plane) is 1000 mm. The cone represents the portion of the X-ray beam that passes through the femur. Note that in this application, we follow the convention to refer to images acquired in the posterior-anterior direction as "AP" images and images acquired in the anterior-posterior direction as "PA" images. Similarly, we refer to images acquired in the lateral-medial direction as "ML" images and images acquired in the medial-lateral direction as "LM" images.

[0095] exist Figure 2In the image, the C-arm width is incorrectly estimated to be 900 mm instead of the true 1000 mm. Therefore, all objects in the image, including the femur (FH), appear smaller than they should be in the X-ray image. Therefore, it appears that the objects have moved toward the AP image plane (denoted as 2.AP) and the ML image plane (denoted as 2.ML). The corresponding focal spots are denoted as 2.FP.AP and 2.FP.ML. Figure 1 In contrast, the 3D reconstruction of the femur (FH) based on the 2D projections of the approximated femur (white circles 2.FH.AP and 2.FH.ML) remains unchanged. The only parameter that changes is the apparent imaging depth. However, the imaging depth is not relevant in this case because the relative 3D position of the femur and the nail does not change.

[0096] To illustrate Figure 1 and Figure 2 The only difference between them is the apparent imaging depth, Figure 3 Both cases are shown.

[0097] Effect of scaling: If one of the images is captured with a scaling factor, the object will appear larger than it would without scaling. Figure 4 , the AP image (denoted as 4.AP) was captured with a zoom factor of 1.5. Therefore, all objects in the image, including the femur (FH), appear to have moved toward the focal point in AP (denoted as 4.FP.AP). As mentioned earlier, the 3D reconstruction of the femur (FH) based on the 2D projections of the approximate femur (dashed white circles 4.FH.AP and 4.FH.ML) is similar to Figure 1 The comparison remains unchanged. The only parameter that changes is the apparent imaging depth. However, the imaging depth is not relevant in this case because the relative 3D position of the femur and nail has not changed. Similar comments apply when both images have zoom. Figure 5 Compared with the scaling (such as Figure 4 as shown) and without scaling (as shown Figure 1 shown).

[0098] Effect of X-ray detector size: If the assumed size of the X-ray detector is 12" instead of the true 9", the object appears larger in the image and it appears that the object has moved towards the focal point in both images. Figure 6 , where:

[0099] 6.AP.9 refers to the AP image with a 9" X-ray detector, and its focus is indicated as 6.FP.AP.9.

[0100] 6.AP.12 refers to the AP image with a 12" X-ray detector, and its focus is indicated as 6.FP.AP.12.

[0101] 6.ML.9 refers to the ML image with a 9" X-ray detector, with the focal point indicated as 6.FP.ML.9.

[0102] 6.ML.12 refers to the ML image with a 12" X-ray detector, and its focus is indicated as 6.FP.ML.12.

[0103] The effect is equivalent to the scaling factor applied to both images. Therefore, the same conclusions can be drawn as in the scaling case.

[0104] Effect of gravity on ML images: (2004) paper, when capturing ML images using a C-arm based X-ray imaging device, the focus is moved 8 mm towards the ground. Given a C-arm width of 1000 mm, this corresponds to a C-arm rotation of approximately 1 degree. This rotation results in a vertical shift and a small rotation of the objects depicted in the image. Since the shift and rotation are detected during scene reconstruction (based on the estimated position of the implant), ignoring this effect will result in a small error relative to the reconstructed femoral center. Moreover, the effect of gravity can be taken into account during reconstruction to avoid this error.

[0105] Measuring characteristics of categorical objects

[0106] The present invention does not require a priori calibration. If there is a known object in the image that is located near the structure to be measured (at a similar depth to it), the measurement can be made in millimeters. Since the known object has a known size, it can be used to calibrate the measurement. This is similar to the procedure proposed by Baumgaertner et al. to determine the TAD value (see Baumgaertner MR, Curtin SL, Lindskog DM, Keggi JM: "The value of the tip-apex distance in predicting failure of fixation of peritrochanteric fractures of the hip". J Bone Joint Surg Am. 1995, 77: 1058-1064.).

[0107] Example 1: The nail has been inserted and an AP image is available. The nail has been identified and located. Since the nail is located in the middle of the shaft and therefore at a similar imaging depth to the lateral cortex of the depicted shaft, the known nail geometry can be used for calibration. This enables a ratio to be provided to determine the distance between the nail axis and the lateral cortex of the shaft.

[0108] Example 2: If the imaging depth of object A is known (e.g., because object A is large enough or because the size of the X-ray detector and the distance between the image plane and the focal point are known), and if there is information about the difference in imaging depth between object A and object B (e.g., based on anatomical knowledge), then the size of a different object (called “object B”) at a different imaging depth can even be calculated based on the intercept theorem.

[0109] Taking intramedullary nail as an example to deal with image distortion

[0110] Generally, there are two ways to deal with image distortion:

[0111] 1. To weaken the areas of the X-ray image that are known to be severely distorted (such as the borders of the image) and focus more on areas that are less affected by the distortion,

[0112] 2. Determine the distortion and account for it.

[0113] These situations will now be illustrated with the example of an AP image of a femur with a nail inserted.

[0114] About 1. Fig. 9 The following letters are used in the markings. The solid lines are the outlines of the nail and the aiming device as seen in the distorted X-ray image. The white dashed lines represent the imaginary outlines of the nail and the aiming device when they are shown without distortion in the image.

[0115] 9.D: Distal end of intramedullary nail

[0116] 9.C: Central portion of the nail, including the hole for the cervical screw

[0117] 9.P: Proximal end of intramedullary nail

[0118] 9.A: Aiming device

[0119] Typically, 9.D is located in a more distorted area of ​​the X-ray image. Furthermore, the exact location of 9.D is not important when predicting the trajectory of a screw inserted through the hole at 9.C. Therefore, the locations of 9.C and 9.P may have a higher weight than the location of 9.D in the prediction of the screw trajectory, where the exact weight can be determined based on their visibility and the reliability of detection. It may also be justified to have a higher weight on 9.C and 9.P because these areas are closer to the target area (screw hole and femur). Moreover, the appearance of 9.C carries information about the rotation of the nail around its axis.

[0120] Regarding 2. Distortion in an image can be determined by the following factors:

[0121] a) earlier surgery (which can be known for a dedicated C-arm),

[0122] b) Pre-operative calibration: A known object (e.g., pins, K-wires, etc.) can be placed directly on the image intensifier / X-ray detector at a known distance from the image plane. This can also be used to determine the size of the X-ray detector and the distance between the focal point and the image plane.

[0123] c) images acquired earlier (which can be learned by algorithms during surgery),

[0124] d) A database with typical distorting effects (e.g. typical pincushion effect of typical C-arm positions, earth's magnetic field). The device can use the knowledge that digital X-ray machines do not distort.

[0125] If such information is available, it can be used when matching the virtual projection of the model with the projection in the X-ray image.The distortion can be applied to the entire image, or specifically to the shape being matched.

[0126] Due to the effect of the earth's magnetic field, the distortion depends strongly on the position of the C-arm imaging device in physical space. In a typical operating room setting, if the position of the patient is known (e.g., prone or supine), the position of the C-arm imaging device can be obtained based on the imaging direction. The imaging direction can in turn be determined, for example, using the invention of Blau filed as a patent application on August 23, 2018.

[0127] In the following, an example of the pincushion effect in the case of AP images of the proximal femoral nail is discussed. Figure 7 Depicted is a typical AP image without distortion, as can be seen from the dashed rectangular grid.

[0128] Figure 8 shows the same scene with simulated pincushion effect distortion. (2004) paper, this is the type of distortion that has the greatest impact on AP images. The distortion can be seen in the dotted grid of the distortion. It is clear that pixels near the image border are more affected than pixels in the center of the image.

[0129] Fig. 9 Again, it is shown that Figure 8 The same distorted image, but without the grid. The solid lines are the outlines of the nail and the aiming device as seen in the X-ray image (indicated as 9.NAD). The white dashed lines represent the imaginary outlines of the nail and the aiming device if they were shown undistorted in the image.

[0130] In the following, an example of sinusoidal distortion in the case of ML images of the proximal femoral nail is discussed. (2004), in addition to the pincushion effect, ML images are also subject to sinusoidal distortion (due to the influence of the Earth's magnetic field). The pincushion effect can be determined based on the AP image, and then a correction can be applied to the ML image. Assuming that the correction is correct, only the sinusoidal distortion will remain in the ML image.

[0131] Fig.10 Depicted is an ML image without distortion as can be seen from the dashed rectangular grid. Fig.11 The same scene is shown with sinusoidal distortion as can be seen from the distorted grid.

[0132] Fig.12 Again, it is shown that Fig.11 The same distorted image in , but without the grid. The solid lines are the outlines of the nail and aiming device as seen in the X-ray image (denoted by 12.NAD). The white dashed lines represent the hypothetical outlines of the nail and aiming device if they were shown undistorted in the image.

[0133] Fig.13 The results of positioning the nail and aiming device based on a distorted X-ray AP image of the proximal femur are shown. X-ray projection image 13. XRAY and Fig. 9 . If the distortion is not taken into account, this will result in the nail and the aiming device being positioned in the wrong position indicated by 13.NAD.IC, while the correct position should be at the position indicated by 13.NAD.C. Although an incorrect determination of the imaging depth is not relevant in the context of the present invention, an incorrect 3D orientation is problematic on the other hand. In the case shown, there is an error of about 4.5 degrees relative to the nail axis.

[0134] However, Fig.14 Shows the result Fig.13 The matching of the 2D contour (solid white line) in the corresponding X-ray image of the incorrect positioning in itself is only a poor match of the true contour. Fig.15 and Fig.16 This can be seen more clearly when zooming in on the image. Fig.15 In , there are some obvious gaps between the black nails and the determined outline (indicated as the white solid line 15.O), some of which are indicated by 15.GAP. Fig.16In the figure, there is also a clearly visible gap between the nail presented in black and the determined contour (represented as a solid white line 16.O), especially at the nail hole, as shown by 16.GAP. The gap 16.GAP can be explained as being caused by an incorrectly determined nail rotation. By combining the two methods described above (focusing on the area with less distortion around the nail hole and taking into account the type of expected distortion, especially in the image borders where the aiming device is located), the positioning accuracy can be significantly improved. The present invention allows an interaction between the unfolding and matching processes, leading to more accurate results.

[0135] Handles the position swapping of X-ray source and receiver

[0136] Because the X-ray imaging device allows mirroring of the image, and the present invention does not use a calibration reference body attached to the image detector during the entire procedure, it may not be possible to detect the position swap of the X-ray source and the receiver even if the treatment side (left bone or right bone) is known. The user may be asked to provide information on whether the mirroring function is enabled. If this is not desired, the situation of how a correct 3D reconstruction of the scene can still be performed is discussed below.

[0137] In the case of proximal AP images of the femur or humerus, the positional exchange of the X-ray source and receiver could not be detected, but due to the symmetry of the proximal femur or humerus, this exchange had no significant effect on the correct 3D reconstruction. Fig.17 Shows Figure 1 The scene in which the original imaging direction is AP with focus 17.FP.AP, resulting in the AP image denoted as 17.AP. In addition, Fig.17 Also shown is a PA image, denoted 17.PA, which is an approximate mirror image of 17.AP (note that in the figure, image 17.AP is shown from the front, while image 17.PA is shown from the back). If the focus of image 17.PA is at position 17.FP.PA, then the image is generated by the C-arm. In the 3D scene, the PA configuration is obtained by mirroring the AP configuration relative to the plane defined by the femoral axis and the femoral center. It can be seen that both the AP and PA images will result in the same 3D reconstruction of the femur (FH).

[0138] The reason is that when the nail implant is kept at a fixed distance from the imaging plane, the nail implant looks very similar in both the AP direction and the mirrored PA direction. In addition, the proximal femur has a certain symmetry. For example, if a mirror image is received, it can be concluded that the other side (e.g., the right side instead of the left side) is being treated if the current C-arm orientation is maintained. However, the projected 2D contours of both bones (i.e., left and right) are very similar. Fig.1818.PROJ. This discussion is shown. It shows a typical left femur AP imaging situation. It also depicts the right femur at the same imaging depth. For the left and right bones, the contours of the nail implant in the 2D projection plane (denoted as 18.PROJ) will be essentially the same. In addition, by comparing the dotted and solid lines in plane 18.PROJ, it can be seen that the 2D bone contours in the 2D projection plane are only slightly different.

[0139] Since in typical ML images the distance from the pin to the image plane is smaller than the distance from the aiming device to the image plane, these assumptions no longer hold when switching from ML to LM with an additional mirror. Fig.19 Depicted are the contours of the aiming device and the nail in the projection plane (denoted 19.PROJ) of the ML image (thick solid line) and the corresponding mirrored LM image (thick dashed line). As can be seen, these contours are clearly different. Fig.19 The femur contours of the ML view (thin solid line) and the mirrored LM view (thin dashed line) in the projection plane 19.PROJ are also depicted. Within the field of view of the X-ray receiver (rectangle denoted 19.XRAY), the contours of the bones are almost identical. However, the contours of the aiming device and the nails have sufficient differences within the rectangle 19.XRAY that they can be easily distinguished between the ML image and the mirrored LM image.

[0140] The benefits of being able to determine imaging orientation based on different methods

[0141] As previously described, the present invention, and the use of the invention of Blau, filed as a patent application on August 23, 2018, allows the determination of the imaging direction based on independent methods that utilize different information. This can be used to (i) cross-validate the results obtained by different methods and / or (ii) improve the accuracy of the calculated imaging direction by appropriately fusing the results obtained by different methods. Next, an example showing the benefits of this approach is shown.

[0142] Example 1: A key question that may arise when processing a series of X-ray images is whether there is movement of an object relative to another object that is also depicted in the X-ray image, such as a nail relative to the anatomical structure. If this movement exists, it may be necessary to calculate it. A complicating factor is that when an object moves in the X-ray projection image, it may be due only to the movement of the C-arm imaging device. The present invention allows distinguishing between the movement of the C-arm imaging device and the movement of the object. In the following, it is explained how the movement of the nail relative to the patient can be detected. The present invention solves this problem by determining the viewing direction on the anatomical structure (for example, using the invention patented by Blau on August 23, 2018) and the position and direction of the nail. It is thus possible to determine at which angle the C-arm observes the nail and at which angle the C-arm observes the anatomical structure. Based on two consecutive X-ray images acquired in the AP direction, it is possible to detect whether the nail has rotated about its axis, even if there is a (slight) C-arm rotation between the images. This detection is important because it may happen during a surgical procedure that the nail may inadvertently rotate when adjusting the entry depth of the nail.

[0143] Example 2: Existing systems require that when performing 3D registration, the implant does not move relative to the anatomical structure when switching between the AP view and the ML view. Ignoring this requirement may result in incorrect 3D registration. This example cites an example of a potential processing workflow. In this processing workflow, the 3D angle between two viewing directions is calculated based on two methods (see steps 3, 12, and 13), one of which is based on the anatomical structure (using the invention of Blau filed as a patent application on August 23, 2018), and one method uses the implant component as a reference. If both calculations are considered correct (which can be determined based on the viewing direction, visibility of the anatomical structure, image quality, etc.), but the difference exceeds a threshold, the system concludes that there is movement of the nail relative to the anatomical structure between the acquisition of the two images, and therefore notifies the user. Movement can also be automatically corrected (see Example 1).

[0144] If the 3D registration process is successful, it can lead to higher accuracy in localization and 3D reconstruction. For example, it can help resolve ambiguities that may exist when only a single image is processed.

[0145] Example of a potential processing workflow for a remote locking process

[0146] Fig. 20The flowchart in the Figure illustrates the principles of the steps performed according to the embodiments of the disclosed invention. It will be understood that the steps described are main steps, wherein these main steps can be distinguished or divided into several sub-steps. In addition, there may be sub-steps between these main steps. It should also be understood that only a part of the entire method may constitute the present invention, that is, steps may be omitted.

[0147] This workflow involves a long antegrade femoral nail from the side locking process. Similar steps can be used for retrograde nails, tibia or humerus, and / or locking from other sides (e.g., front side).

[0148] 1. Requires assembly and installation of long sight device. Insert locking sleeve assembly.

[0149] 2. Obtain a transverse X-ray image of the distal femur. In the prior art, this step requires a specific viewing angle. The present invention does not require a specific imaging direction, as long as the long aiming device can be seen in the X-ray.

[0150] 3. Positioning the long aiming device. If the C-arm is not positioned properly (e.g., the sleeve is not sufficiently visible), the device can calculate the necessary adjustments (because the system knows where the sleeve is positioned relative to the long aiming device) and give appropriate instructions on how to achieve that adjustment (rotation and translation of the C-arm).

[0151] 4. The device determines the position and contour of the desired unbent nail in the X-ray projection image (taking into account the intercept theorem).

[0152] 5. The actual position of the nail is determined by matching the virtual projection of the nail model with the appearance of the nail in the X-ray image. This matching may take into account the following factors:

[0153] a The expected position and contour of the unbent nail in the X-ray projection image,

[0154] b. The object properties of the nail, especially possible bending in AP and ML directions, the thickness and material of the nail, the play in the connection between the nail and the aiming device. For example, a thicker nail may bend less and may have negligible torsion,

[0155] c. Image characteristics and X-ray attenuation, absorption and deflection.

[0156] This enables calculation of the relative 3D position between the distal portion of the nail and the long aiming device.

[0157] 6. Determine the 3D position of the axis of the target nail hole (which may have a cylindrical shape) relative to the axis of the sleeve (determined by the position of the long sighting device and the sleeve). The device determines the distance between the sleeve axis and the nail axis. Depending on the type of long sighting device and the hole being aimed at (which may be known a priori), the system calculates how the long sighting device needs to be adjusted to compensate for this distance.

[0158] 7. Adjust the long aiming device.

[0159] 8. Repeat steps 2 to 7 until the distance between the hole and the sleeve axis is small enough.

[0160] 9. The surgeon can now perform distal locking.

[0161] The distance from the image detector to the focal point may not be available in practice. In the following example, it is shown that, assuming the target hole is close enough to the image center (ensured by giving appropriate C-arm positioning instructions), the error introduced by estimating the imaging depth of the aiming device alone is small enough.

[0162] For the example below, the long aiming device is located 180 mm (towards the focus) from the nail. Therefore, in the telescopic ML image, the distance of the long aiming device from the image plane is different from the distance of the nail from the image plane. Since the relative position of the long aiming device and the nail is known, the nail outline can be correctly projected onto the image plane.

[0163] If the focus is estimated incorrectly, for example 1100mm instead of 1000mm (which is the worst case), the error will be small for points near the center of the image. Fig.21 , where the correct focus is indicated as 21.FP.C and the incorrect focus is indicated as 21.FP.IC. The corresponding nail outline in the 2D projection image is a solid line in the incorrect case, which is difficult to distinguish from the dashed line in the correct case, and the corresponding nail holes are indicated by 21.NH.C (the correct nail hole is indicated by "x") and 21.NH.IC (the incorrect nail hole is indicated by a circle). For a given cluster, the height of the nail hole is 220 mm, and its 2D projection is 15 mm from the center of the image. The incorrect nail hole position deviates from the correct nail hole by only 0.65 mm, which can be ignored for distal locking.

[0164] It should also be noted that swapping the positions of the X-ray source and receiver does not present a problem, as this can be addressed as discussed in the "Handling X-ray Source and Receiver Position Swaps" section. If the C-arm is rotated 180 degrees so that the focal point and image plane swap their positions, the distance from the long aiming device to the image plane will be smaller than the distance from the nail implant to the image plane. Fig. 22As shown, the profile of the nail with a rotated C-arm (indicated by the dashed line) will deviate greatly from the profile of the nail with an unrotated C-arm (indicated by the solid line). Therefore, the effect of the position swap of the X-ray source and the receiver is easily detected.

[0165] Distal locking can also be performed freehand, ie without the support of a long aiming device. For the following exemplary method, it is assumed that the nail has a plurality of locking holes, the axes of which may or may not be located in the front plane.

[0166] After the nail is inserted into the long bone, the nail (e.g., femoral nail) is first fixed in the bone at the insertion end. The fixation or locking nail at the other end usually requires the C-arm of the imaging system to be moved to that end.

[0167] As described above, the present invention provides a system capable of determining objects and their geometric shapes based on X-ray images, wherein the objects may be bones or implants, or tools such as surgical knives, alignment devices or drills.

[0168] 1. Typically, an incision will be made in the skin, for example with the aid of a scalpel, and then a hole will be drilled in the bone. If the hole is aligned with the through hole in the nail, the locking screw can be correctly inserted. The appropriate position of the incision and the position and orientation of the drill can be found based on information determined and calculated by processing the X-ray image data. Basically, the required position and orientation of an object such as a scalpel or a drill can be determined based on the tilted lateral image. For this purpose, the system can utilize additional information from the AP (Anterior to Posterior) image acquired first.

[0169] 2. The surgeon can move the C-arm to the distal AP position and an X-ray image can be acquired. If the orientation of the C-arm is not adequate for system guidance, the system can instruct the surgeon on how to adjust the orientation of the C-arm. This can be done based on the viewing direction on the implant or anatomical structure (e.g., which can be determined based on the X-ray image, for example, using the invention in the patent application filed by Blau on August 23, 2018).

[0170] 3. The system can detect the nail and distal locking holes in the AP image, and can measure the distance from the nail axis to the lateral surface of the bone perpendicular to the nail axis, or measure the distance from at least one hole to the lateral surface of the bone. It can also measure the distance from at least one hole to the skin surface. The system can also assist the surgeon in applying marks on the skin at the height of each hole. In order to determine the required length of the (e.g., bicortical) locking screw for subsequent steps, the system can also measure the distance between the nail axis and the medial bone surface accordingly. Such measured distances can be used as prior information when processing subsequent transverse images (true ML and / or oblique transverse images).

[0171] 4. Alternatively or additionally, further improvements can be achieved by determining the insertion axis of the locking screw.

[0172] The surgeon can move the C-arm to the distal true ML position and acquire an X-ray image. For example, if the distal locking hole appears as a perfect circle in the acquired X-ray projection image, the C-arm is probably in a true lateral position. If necessary, the system can support the surgeon by providing precise instructions for any required C-arm adjustments. Thus, the desired position and orientation of the C-arm can be achieved more efficiently and with fewer iterations.

[0173] 5. Additionally or alternatively, the system may match a statistical 3D model of the bone to determine the 3D position of the nail relative to the 3D position of the bone, thereby enabling determination of the required locking screw length in 3D space.

[0174] Moreover, when processing subsequent oblique lateral images, the information obtained from the real ML images can be used as prior information.

[0175] 6. The system assists the surgeon in making the first incision for inserting the distal locking screw. Once the scalpel is placed at the assumed incision location, an X-ray (true or oblique lateral) is acquired, and the system instructs the surgeon how to adjust the position and orientation of the scalpel to begin inserting the locking screw at the appropriate entry point. The surgeon then makes the cut.

[0176] The alternative method described below is to distinguish between holes whose axes are located in the front plane and holes whose axes are not located in the front plane. If the hole axis is located in the front plane, the system will assist the surgeon in moving the C-arm so that the central X-ray beam is parallel to the front plane. In the more common case (if the hole axis is not in the front plane), the system will assist the surgeon in moving the C-arm so that the 2D projection of the hole has the maximum diameter in the direction perpendicular to the nail axis.

[0177] This can be done based on the viewing direction on the implant or anatomical structure (e.g., can be determined based on an X-ray image, for example, using the invention in the patent application filed by Blau on August 23, 2018). For each locking hole, the surgeon makes an incision so that the scalpel is located on the mark applied in step 3 above and simultaneously matches the nail axis in the X-ray projection.

[0178] 7. With the C-arm in the lateral position, the surgeon positions the drill (with or without sleeve) so that in the acquired X-ray image, the drill tip is located on the central axis of the locking hole, i.e., at the center of the distal locking hole when viewed along this axis. The system can detect the tip of the drill and the locking hole, can calculate their relative position, and provide guidance for repositioning the drill if necessary. Another X-ray image can be acquired to verify the position.

[0179] 8. The surgeon can adjust the orientation of the drill without changing the drill tip position. For example, the drill axis can be roughly parallel to the true transverse X-ray imaging direction. With the C-arm in an inclined transverse position, the system can verify the drilling direction by using the distance information from the drill tip to the locking hole (when it is on the bone surface) and the information that the drill tip is on the correct locking trajectory.

[0180] 9. The system can use information from earlier images to more accurately calculate the position and orientation of the drill bit (drill sleeve). For example, a first image can be acquired when the tip of the instrument is on the bone, and the system uses information about the distance between the lateral bone surface and the nail axis to calculate the relative position and orientation of the instrument and implant. A second image can be acquired during drilling in the bone, and the system can use information about the relative position and orientation of the previous image (such as trajectory) instead of using the distance between the bone surface and the implant.

[0181] Support and reset assessment

[0182] In any osteosynthesis procedure, adequate and correct reduction of the fracture is crucial to achieving a satisfactory clinical outcome. In general, a fracture can only heal satisfactorily if the reduction is performed correctly. Reduction of long bones in particular is often difficult to assess during surgery, especially with regard to the correct anteversion angle. An incorrect anteversion angle is often only noticed after the patient is able to stand again after the surgery. At this stage, an incorrect anteversion angle can cause significant discomfort to the patient, even if the fracture itself has healed appropriately. Indeed, an incorrect anteversion angle is one of the main reasons for revision surgery. Therefore, an adequate and correct anteversion angle is crucial to achieving a satisfactory clinical outcome, especially for osteosynthesis of the femur and tibia.

[0183] The prior art proposes different methods to determine the anteversion angle. In the case of femoral and cephalomedullarynail nails, one method is to determine by hand whether the knee is facing up toward the ceiling of the operating room and to subjectively judge whether the screw, which should intersect the nail axis and the center of the femur, is at an angle of approximately 10 degrees to the operating room floor. Blau et al. proposed another method (patent application US 2015 / 0265361 A1), in which two reference bodies with metal markers are used, one in the distal region of the femur and one in the proximal region of the femur, as well as two proximal X-ray images and one distal X-ray image, all depicting the respective reference bodies.

[0184] In the following, different methods of assessing whether the bone fragments are anatomically reduced are discussed. Note that these techniques do not require determination of 3D position or 3D orientation.

[0185] A.

[0186] One method is to measure the thickness of the bone fragments and the thickness of the cortex as they appear in the X-ray images. Fig.23 This is illustrated in the example of a femoral shaft fracture shown in FIG. In this figure, the proximal portion of the femur is indicated as 23.PF, while the distal portion is indicated as 23.DF. In the X-ray image, the proximal femur next to the fracture line appears to have a width of 23.PW, while the distal portion next to the fracture line appears to have a width of 23.DW. If these two widths are different, it can be concluded that the fragments are misaligned relative to each other. Moreover, the thickness of the medial cortex of the proximal fragment next to the fracture line (indicated as 23.PCM) must be sufficiently close to the thickness of the medial cortex of the distal fragment next to the fracture line (indicated as 23.DCM). Similarly, the thickness of the lateral cortex of the proximal fragment next to the fracture line (indicated as 23.PCL) must be sufficiently close to the thickness of the lateral cortex of the distal fragment next to the fracture line (indicated as 23.DCL).

[0187] In order for a fracture to be correctly reduced, all three conditions must be met simultaneously. If at least one of these conditions is violated, this is usually due to rotation about the diaphyseal axis, since the diaphyseal shaft of a long bone is not usually perfectly round. The more the cross section through the bone (perpendicular to the diaphyseal shaft) deviates from the disk, the more sensitive this method is to slight rotations about the diaphyseal axis.

[0188] The method can be performed manually or automatically. Therefore, no calibration images are required, as all measurements are performed relative to each other and at similar imaging depths. A neural network can decide whether the above conditions for correct repositioning are met.

[0189] B.

[0190] Furthermore, the fracture line is usually not straight and perpendicular to the axis of the diaphysis. Therefore, if there is no comminuted fracture, the assessment of the fracture line shape may provide key information about the correct reduction, because the shape of the proximal fracture line must match the shape of the distal fracture line. This fracture line can be determined using the invention of the patent application filed by Blau on November 26, 2018. The neural network can decide whether the above conditions for correct reduction are met.

[0191] C.

[0192] Another possibility to assess whether the reduction is anatomically correct is the visibility of other typical landmarks of the bone fragments. For the example of a femoral shaft fracture, the lesser trochanter is a particularly suitable landmark for assessing the rotational alignment of the proximal femoral fragment. This can be based on the visibility of the lesser trochanter in AP images, since it will disappear if the bone is rotated about the diaphyseal axis. For the distal fragment, in a true AP image, the patella will be centered between the condyles.

[0193] In the AP imaging orientation, the C-arm can be easily moved parallel to the axis of the diaphysis from a distal position to a proximal position and vice versa without rotating the C-arm. Taking a femoral shaft fracture as an example, after acquiring a true AP image of the knee, the C-arm can be moved to the hip. If the proximal femur then also appears in a typical position in the AP imaging orientation (e.g., based on the visibility of the lesser trochanter), it can be concluded that rotational alignment and, therefore, anatomical reduction of the femoral shaft fracture has been correctly achieved.

[0194] Combination method:

[0195] By combining the methods described in paragraphs A. to C. above, a better assessment of the reduction can be made. An algorithm can fuse the information obtained from these different methods and provide a comprehensive assessment of the quality of the reduction. This may involve weighting the different methods, where there may be different weighting schemes. For example, one method may always have a higher weight than the others, for example, method A may have a higher weight than method C because C is based on the typical appearance of the average anatomical structure, while A is patient-specific. The weighting can also be based on a confidence value that can be calculated by the system for the results of each method. Such weighting can be based on image quality (poorer quality means lower confidence values) or fracture type (unclear fracture lines mean lower confidence values).

[0196] As described in the previous paragraph, the fusion of information can be performed by a neural network trained on a sufficiently large sample size of correctly reduced fractures and incorrectly reduced fractures. After appropriate training, the DNN is able to classify whether the reduction is correct or incorrect and provide this information to the user.

[0197] In the following, exemplary embodiments related to the present invention are listed:

[0198] 1. A system for determining a 3D representation and positioning of an object in an X-ray projection image relative to a coordinate system, the system comprising a processing unit configured to execute a computer program product comprising a set of instructions causing the system to:

[0199] - receiving X-ray projection images,

[0200] - classify objects in X-ray projection images,

[0201] - a model that receives the object to be classified,

[0202] - determining a 3D representation of the classified object, and

[0203] - Positioning the classified object relative to the coordinate system by applying the model to match the classified object in the X-ray image.

[0204] 2. A system for determining a 3D representation and a position of a first object relative to a second object in an X-ray projection image, the system comprising a processing unit configured to execute a computer program product comprising a set of instructions causing the system to:

[0205] Receive X-ray projection images,

[0206] classifying a first object in the X-ray projection image,

[0207] receiving a first model of a first object,

[0208] determining at least a partial representation of the first object and a location of the first object by applying the first model to match the first object in the X-ray image,

[0209] classifying a second object in the X-ray projection image,

[0210] receiving a second model of a second object,

[0211] determining at least a partial representation of the second object and a location of the second object by applying the second model to match the second object in the X-ray projection image, and

[0212] Determine the spatial relationships of the objects being classified.

[0213] Hereinafter, when a "model" is mentioned, it refers to one model of Example 1 or one of the first model and the second model of Example 2. The same is true for an "object".

[0214] 3. A system according to example 1 or 2, wherein the matching of the model to the classified object takes into account image features of the X-ray projection image, wherein the image features depend on at least one imaging parameter and include one of a group consisting of a pillow effect, curvature, noise, distortion, and an X-ray imaging generation method.

[0215] 4. The system according to any one of Examples 1 to 3, wherein the matching of the model to the classified object takes into account the effect of at least one of the group consisting of X-ray attenuation, absorption and deflection.

[0216] 5. A system according to any one of Examples 1 to 4, wherein at least a portion of a contour of the classified object is detected in the X-ray projection image.

[0217] 6. A system according to any one of examples 1 to 5, wherein the system is also caused to locate a further object, wherein the object is even not visible or only partially visible in the X-ray image.

[0218] 7. The system of any one of examples 1 to 6, wherein the computer program further comprises a set of instructions that cause the system to transfer the geometric aspects from the model to the object.

[0219] 8. The system of example 7, wherein the computer program further comprises a set of instructions that cause the system to display the X-ray projection image together with those geometric aspects.

[0220] 9. A system according to Example 7, wherein the geometric aspect includes a size, wherein the size defines a scale suitable for measuring the size in the X-ray projection image.

[0221] 10. A system according to any one of Examples 3 to 9, wherein at least one of the imaging characteristics is determined by using an object of known shape for calibration, by automatically detecting the imaging direction based on information from a database.

[0222] 11. A system according to any one of Examples 5 to 10, wherein at least one of the following steps is assisted by user input: classifying objects in an X-ray projection image; detecting outlines of classified objects in an X-ray projection image; determining a 3D representation of the classified objects; determining a positioning of the classified objects; determining image features; determining object features; determining X-ray attenuation, absorption, and deflection.

[0223] 12. A system according to any one of Examples 5 to 11, wherein at least one of the following steps is automatically performed: classifying objects in an X-ray projection image; detecting outlines of classified objects in the X-ray projection image; determining a 3D representation of the classified objects; determining the positioning of the classified objects; determining image features; determining object features; determining X-ray attenuation, absorption, and deflection.

[0224] 13. A system according to Example 12, wherein at least one of the following steps is performed by a neural network: classifying objects in an X-ray projection image; detecting the outline of the classified object in the X-ray projection image; determining a 3D representation of the classified object; determining the positioning of the classified object; determining image features; determining object features; determining X-ray attenuation, absorption, and deflection.

[0225] 14. A system according to any one of Examples 1 to 13, wherein matching of an object in an X-ray projection image with a model comprises: adapting image features of the X-ray projection image with image features of a virtual projection of the model, and / or adapting image features of the virtual projection of the model with image features of the X-ray projection image.

[0226] 15. A system according to any one of Examples 1 to 14, wherein the matching of objects in the X-ray projection image takes into account object characteristics, the object characteristics including at least one of the group consisting of shaking of the mechanical interface, material and / or bending of the implant and tool, and fracture.

[0227] 16. A system according to any of Examples 1 to 15, wherein the model is a statistical shape or appearance model.

[0228] 17. A system according to any one of Examples 1 to 16, wherein an imaging direction of the X-ray projection image is tilted relative to an axis of the object.

[0229] 18. A system according to any one of Examples 1 to 17, wherein the computer program also includes a set of instructions for causing the system to determine an imaging direction of an X-ray projection image.

[0230] 19. A system according to any one of Examples 1 to 18, wherein the computer program further comprises a set of instructions for causing the system to receive information representing a target position of an object relative to a coordinate system and / or relative to another object, and to determine a deviation of the determined position of the classified object from the target position.

[0231] 20. A system according to any one of Examples 1 to 19, wherein the computer program further comprises a set of instructions for causing the system to determine at least one geometric aspect of the object and determine a relationship of the geometric aspect relative to a coordinate system and / or relative to a geometric aspect of another object.

[0232] 21. A system according to any one of Examples 1 to 20, wherein the computer program further includes a set of instructions for causing the system to receive another X-ray projection image of the same object and determine a change in the spatial relationship of the object with the coordinate system and / or another object based on a comparison of the spatial relationship determined by the first X-ray image with the spatial relationship determined by the other X-ray image.

[0233] 22. A system according to any one of Examples 1 to 21, wherein the computer program further includes an instruction set that causes the system to receive at least one additional X-ray projection image of the same object, wherein imaging parameters and / or imaging directions of the X-ray projection images in the X-ray projection images are different from each other, and to determine a three-dimensional aspect of the object by combining information from multiple projection images.

[0234] 23. A system according to any of Examples 1 to 22, wherein the computer program further comprises a set of instructions for causing the system to receive at least one additional X-ray projection image depicting the same object from a different imaging direction, wherein a 3D representation of the object is used to register the images.

[0235] 24. A system according to example 23, wherein the registration of the images is used to determine a 3D representation and positioning of at least one additional object at least partially depicted in all of those X-ray images.

[0236] 25. A system according to any one of Examples 1 to 24, wherein the matching of the classified object is based on the fact that the 3D position of the object is sufficiently close to the expected 3D position.

[0237] 26. A system according to any one of Examples 1 to 25, wherein the matching of the classified object is based on the fact that the 3D orientation of the object is sufficiently close to the expected 3D orientation.

[0238] 27. The system of any one of Examples 1 to 26, wherein a swap of roles of an X-ray source and an X-ray receiver can be detected.

[0239] 28. A system according to any of Examples 1 to 27, wherein the imaging depth is determined based on at least one of the sufficiently large objects in at least one of the received X-ray images.

[0240] 29. A system according to any of Examples 1 to 28, wherein the imaging depth is determined based on a previous calibration of a known object.

[0241] 30. A method for determining a 3D representation and positioning of an object in an X-ray projection image relative to a coordinate system, the method comprising the steps of:

[0242] - receiving X-ray projection images,

[0243] - classify objects in X-ray projection images,

[0244] - a model that receives the object to be classified,

[0245] - Matching the classified object in the X-ray image by applying the model to determine a 3D representation of the classified object and positioning the classified object relative to the coordinate system.

[0246] 31. A method according to Example 30, wherein the matching of the model with the classification object takes into account image features of the X-ray projection image, wherein the image features depend on at least one imaging parameter and include at least one of the group consisting of pillow effect, curvature, noise, distortion, and X-ray imaging generation method.

[0247] 32. A method according to any of Examples 30 or 31, wherein the matching of the model to the classified object takes into account the effects of at least one of the group consisting of X-ray attenuation, absorption, and deflection.

[0248] 33. The method according to any one of Examples 30 to 32 further includes the following step: detecting at least a portion of the contour of the classified object in the X-ray projection image.

[0249] 34. The method according to any one of Examples 30 to 33 further includes the following step: locating another object that is not visible or only partially visible in the X-ray image.

[0250] 35. The method according to any one of Examples 30 to 34, further comprising the step of transferring geometric aspects from the model to the object.

[0251] 36. The method of Example 35 further comprises the following steps: defining a scale based on geometric aspects to measure dimensions in the X-ray projection image.

[0252] 37. A method according to any one of Examples 30 to 36, wherein the matching of the object in the X-ray projection image with the model includes: adapting the image features of the X-ray projection image to the image features of the virtual projection of the model, and / or adapting the image features of the virtual projection of the model to the image features of the X-ray projection image.

[0253] 38. A method according to any one of Examples 30 to 37, wherein matching an object in an X-ray projection image with a model takes into account object characteristics, wherein the object characteristics include at least one of the group consisting of shaking of a mechanical interface, material and / or bending of the implant and tool, and fracture.

[0254] 39. The method according to any one of Examples 30 to 38 further includes the following steps: receiving another X-ray image and aligning the X-ray images taking into account the movement of the X-ray imaging device between the acquisition of the received first X-ray image and the second X-ray image.

[0255] 40. A method according to Example 39, wherein the amount of movement of the X-ray imaging device is measured by a sensor at the X-ray imaging device.

[0256] 41. The method according to any one of Examples 30 to 40 further includes the following steps: receiving another X-ray image and aligning the X-ray image taking into account the geometry of the model, wherein the model has portions that extend into one of the X-ray images respectively.

[0257] 42. The method according to any one of Examples 30 to 41 further includes the following steps: receiving another X-ray image; classifying another object in each received X-ray image, wherein different parts of the object extend into the received X-ray images respectively, and aligning the X-ray images based on the known geometry of the object.

Claims

1. A method for determining an arrangement of objects relative to each other in an X-ray projection image, comprising the following steps: Receive X-ray projection images, classifying a first object in the X-ray projection image, wherein the first object is a bone, receiving a first model of the first object, matching a first object in an X-ray image by applying the first model to determine at least a partial representation of the first object and a location of the first object, wherein the location of the first object is determined using a neural network, and wherein the location of the first object comprises determination of a 3D orientation of the first object and a 2D spatial position of a projection of the first object on an image plane, classifying a second object in the X-ray projection image, receiving a second model of the second object, matching a second object in the X-ray projection image by applying a second model to determine at least a partial representation of the second object and a location of the second object, wherein the location of the second object includes determination of a 3D orientation of the second object and a 2D spatial position of a projection of the second object on the image plane, The 3D spatial relationship of the classified objects is determined, wherein the determination of the 3D spatial relationship of the classified objects is based on information of a contact point between the two objects.

2. The method according to claim 1, wherein: The matching of at least one of the models with the classification object takes into account image features of the X-ray projection image, wherein the image features depend on at least one imaging parameter and include at least one of the group consisting of a pincushion effect, curvature, noise, distortion and an X-ray imaging generation method, and wherein the image features are determined in the X-ray projection image.

3. The method according to any one of claims 1 or 2, wherein: The matching of at least one of the objects in the X-ray projection image with the corresponding model takes into account object characteristics, which include at least one of the group consisting of shaking of the mechanical interface, bending of the implant and the tool, and bone fracture.

4. The method according to claim 1 or 2, wherein: A 3D representation of at least one of the classified objects is determined.

5. The method according to claim 1 or 2, further comprising receiving information about an expected 3D position and / or an expected 3D orientation of at least one of the classified objects, wherein The matching of at least one object takes into account the expected 3D position and / or the expected 3D orientation.

6. The method of claim 1 or 2, further comprising the step of transferring geometrical aspects from one of the first model and the second model to a corresponding object.

7. The method of claim 6, further comprising the step of defining a scale based on the geometrical aspects to measure dimensions relative to a coordinate system.

8. The method according to claim 1 or 2, wherein: The matching of the model to the classification object takes into account the influence of at least one of the group of X-ray attenuation, absorption and deflection.

9. The method according to claim 1 or 2, wherein: Determining the 3D spatial relationship of the classified objects includes determining the contact points between the classified objects.

10. The method according to claim 1 or 2, wherein: A swap of the positions of the X-ray source and the X-ray receiver is detectable.

11. A system for determining an arrangement of objects relative to each other in an X-ray projection image, the system comprising a processing unit configured to execute a computer program product comprising a set of instructions for causing the system to perform the method of any one of claims 1 to 10.

12. The system of claim 11, further comprising an X-ray imaging device.

13. The system according to any one of claims 11 and 12, further comprising a database.

14. A computer program product comprising a set of instructions which, when executed on a processing unit of a system, causes the system to perform the method of any one of claims 1 to 10.

Images