Determining relative 3D position and orientation between objects in 2D medical images

Abstract

To provide a system, a method, and a computer program for determining 3D representations, and relative 3D positions and relative 3D orientations between objects based on X-ray projection images.SOLUTION: A device receives an X-ray image, a projection image of a first object and second object, classifies the first and second objects, and receives respective 3D models of the objects. A geometrical aspect like an axis or a line is determined on the first object, and another geometrical aspect like a point is determined on the second object. Finally, a spatial relation between the first object and the second object is determined based on a 3D model of the first object, a 3D model of the second object, and information that the point of the second object is located on the geometrical aspect of the first object.SELECTED DRAWING: None

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer-assisted surgery. In particular, the present invention relates to an apparatus and method for determining 3D representations as well as relative 3D positions and relative 3D orientations between objects based on X-ray projection images. The method may be implemented as a computer program executable on a processing device of the apparatus.

Background Art

[0002] In orthopedic or orthopedic trauma surgery or spinal surgery, it is a common operation to aim a relatively thin instrument at a target object or target structure (as part of the target object). The target structure may be an anatomical structure (e.g., pedicle) or part of another instrument or implant (e.g., the distal fixation hole of a long antegrade intramedullary nail). Generally, the goal may be to determine the relative 3D position and relative 3D orientation between the instrument and the target object. Based on available intraoperative 2D imaging techniques, this can be difficult. It is particularly difficult when the exact geometric shape of the target object is unknown and / or the instrument is known but not uniquely locatable in 3D space based on 2D X-ray images.

[0003] A preoperative CT scan may be performed for a surgical procedure, which allows for a more accurate planning of the procedure. This is the case, for example, when performing surgery within a complex 3D structure, or when drilling holes or placing screws within a narrow anatomical structure or near important structures (e.g., spinal cord, nerves, aorta). Typical examples of such procedures are the placement of sacroiliac joints or pedicle screws. When the target structure is an instrument or implant, its 3D geometry is typically known. An example is a distal fixation procedure, where 3D models or 3D information regarding the target object (nail) and in particular the target structure "distal fixation hole" (cylinder) are available.

[0004] However, in order for a surgeon to utilize this 3D information and apply it to intraoperative 2D X-ray images, a high level of spatial awareness and imagination is required.

[0005] In some cases, for some procedures, it may be possible to determine the drilling direction with a drill by aligning it, for example, in a specific viewing direction of the imaging device (e.g., a true medial-lateral view for distal fixation procedures). Furthermore, it is generally not possible to guarantee that the drilling with the drill actually proceeds accurately along this direction. Next, an example of distal fixation of a long antegrade intramedullary nail will be given to illustrate this.

[0006] In the conventional distal fixation procedure of an antegrade nail, the surgeon moves the C-arm to a true lateral position (which means that the hole to be fixed appears completely circular in the X-ray image). This positioning is monotonous and time-consuming as it is repeated, sometimes taking several minutes, and typically requires the acquisition of 5 - 20 X-ray images with corresponding readjustments of the C-arm. A faster way to achieve this positioning is to use the fluoroscopy mode of the C-arm (which generates a continuous X-ray video stream), but this results in a higher X-ray dose.

[0007] Furthermore, to ensure high accuracy for distal fixation, not only must the hole appear circular, but it must also be near the center of the X-ray image. However, in practice, once the hole appears sufficiently circular in the X-ray image, this C-arm position is typically used for distal fixation even if the hole is not near the center of the X-ray image. Due to the conical shape of the fan of the X-ray beam, the more the direction of the X-ray beam is tilted, the further the beam is from the center of the X-ray image. Therefore, the drilling with the drill through the hole must be in the direction of the focus of the X-ray source and not parallel to the center line between the X-ray source and the detector.

[0008] In the next step, the tip of the drill can be placed at the intended drill position and an X-ray image is acquired. Here, the drill may be intentionally held at an oblique angle (i.e., not in the direction of a fixed trajectory) so that the power drill and the surgeon's hand do not block the view. The goal is to position the drill in the X-ray image such that the drill tip appears at the center of the (circular) fixation hole. This is also done repeatedly, typically requiring 5 to 10 repetitions and X-ray images.

[0009] Once this is achieved with sufficient accuracy, the drill is aligned with the target trajectory leaving the drill tip in place. At this angle, the power drill and the surgeon's hand block the view, so this alignment is typically not confirmed by X-ray. Therefore, the surgeon uses the position of the C-arm as a guide and attempts to align the drill parallel to the "C". Achieving and then maintaining such alignment during drilling with the drill requires a fairly high level of manual dexterity. Furthermore, the surgeon typically does not comply with the requirement to aim at the focus of the X-ray source. The greater the error introduced by neglecting to aim at the focus, the further the fixation hole appears from the center of the X-ray image. Typical errors range from 1 to 2 mm at the target point (fixation hole). This is close to the limit of 3 mm, beyond which distal fixation fails in most nail fixation systems. All of this means that attempts at drilling are unsuccessful, especially in the case of less experienced or less skilled surgeons.

[0010] Since it is generally necessary to fix at two or more holes, this entire procedure must be repeated for each hole. Therefore, completing the entire nail fixation procedure is typically very time-consuming, requires many X-ray images, and in many cases attempts at drilling are unsuccessful. This means that distal fixation is one of the most frustrating procedures in the area of bone union. This sometimes leads to the expedient method of using shorter nails instead of longer ones, which then further worsens the patient's prognosis and results in a fairly large number of re-replacement surgeries.

[0011] For this reason, some manufacturers provide a flexible mechanical solution (hereinafter referred to as "long aiming device") for adjusting the bending of the nail in the medullary canal. The long aiming device simplifies the procedure, but its application is still not straightforward because the X-ray image showing the long aiming device must be interpreted accurately and the C-arm position must be adjusted accordingly. The long aiming device can be properly adjusted only after the correct adjustment of the C-arm.

[0012] EP 2801320 A1 (European Patent Application Publication No. 2801320) proposes the concept of detecting a reference body having a metal marker at a known position fixed to the long aiming device and determining the imaging direction of the reference body. Based on this, the system can give an instruction on how to adjust the C-arm imaging device. The drawback of such a system is that the X-ray image must contain the reference body. For the adjustment of the long aiming device in the case of a retrograde femoral nail having a fixation hole in the outer direction, US 2013 / 0211386 A1 (US Patent Application Publication No. 2013 / 0211386) uses a reference body to determine the bending of the nail in each of the ML direction and the AP direction.

[0013] In addition to distal fixation, in order to generally enhance the safety and accuracy of minimally invasive procedures, a surgeon needs access to the necessary intraoperative 3D information, i.e., information regarding the relative 3D position and 3D orientation between the device and the target object / structure or between multiple anatomical objects. This information may be displayed by a tracking-based navigation system, which requires preoperative 3D imaging and then records intraoperative 2D imaging data together with the preoperative 3D data during the surgery. Another alternative is the use of a database (e.g., an implant database) that includes 3D information regarding the target object and additional instruments (e.g., long aiming devices), which are often in the form of additional hardware. These systems are often difficult to handle, time-consuming, and monotonous in setup and intraoperative use, and are typically expensive. Thus, due to all of these drawbacks, navigation-based systems are not always available for use in orthopedics and trauma, and furthermore, are not always feasible. The same explanation applies to systems for intraoperative 3D imaging (e.g., O-arm, 3D C-arm) that also add a high X-ray dose.

[0014] This emphasizes the need for a non-invasive and easy-to-use system that can provide intraoperative 3D information without the need for any additional hardware components without a tracking system. The present invention proposes a system and method that require only a computer and a display and / or loudspeaker for processing intraoperative 2D images. Both techniques that utilize deterministic 3D data of the target object (e.g., in the form of 3D preoperative imaging data of an implant or 3D model data) and techniques that do not require such data are provided. SUMMARY OF THE INVENTION

[0015] It is preferred to work without any reference body or other additional hardware (e.g., aiming devices) because it simplifies product development, enables a more cost-effective and operating room workflow that closely resembles typical workflows, and eliminates the additional uncertainties introduced by a mechanical interface for the reference body (e.g., when using a new implant).

[0016] As described herein, the present invention proposes to combine knowledge of the X-ray image generation process and artificial intelligence (in the form of so-called deep morphing and / or the use of neural networks) instead of any reference body to provide the information required, for example, when treating a fractured bone. Therefore, it may be considered an object of the present invention to provide an apparatus and / or method that enables 3D representation and to determine the relative 3D positions and relative 3D orientations between a plurality of objects that are at least partially visible in an X-ray projection image. Here, the object may be any object visible in the X-ray image, such as an anatomical structure, an implant, a surgical instrument, and / or a part of an implant system.

[0017] The term "3D representation" may refer to a complete or partial description of a 3D volume or 3D surface, which may also refer to selected geometric aspects such as radius, axis, and plane. It may be possible to determine complete 3D information regarding the 3D surface or volume of an object, but in many applications, it may be sufficient to determine only the selected geometric aspects.

[0018] Throughout this application, the terms "localize" and "localizing" mean determining the 3D orientation of an object and the 2D spatial position of the projection of that object onto an image plane. The imaging depth, which is the distance of the object from the image plane, on the other hand, is estimated (with some uncertainty) based on a priori information regarding the typical arrangement of objects in the operating room, such as the relative positions of the implant, the patient, and the imaging device. For most purposes of the present invention, such an estimated imaging depth is sufficient. Some applications may require more accurately determining the imaging depth, which is possible in certain cases as will be further considered below.

[0019] According to one embodiment, 3D reconstruction (i.e., determination of a 3D representation) and localization of an object having some variability in its shape and appearance are provided. This can be done based on a single X-ray image or, for high accuracy, on a plurality of X-ray images. Even in cases where an anatomical structure, implant, surgical instrument, and / or part of an implant system is not visible or only partially visible in the X-ray image, a 3D representation and localization of the relevant object can also be provided.

[0020] According to one embodiment, even when individual localization of at least one object is not possible with sufficient accuracy, determination of the relative 3D position and 3D orientation between multiple objects is provided. This can also be achieved by utilizing a priori geometric information regarding the relative position and / or orientation between the sides of at least two objects and, optionally, by restricting the range of acceptable X-ray imaging directions. Possible clinical applications include freehand distal fixation, sacroiliac joint (SI) or pedicle screw placement, and assessment of anatomical reduction of fractures.

[0021] Incidentally, the image data of the processed X-ray image may be received from an imaging device, for example, a C-arm based 2D X-ray device, or may be directly received from a database. Further, using the aspects of the present invention, medical images obtained using other imaging diagnostic methods such as ultrasonic or magnetic resonance imaging may be processed.

[0022] A system proposed according to an embodiment generally causes a processing device to (i) receive an X-ray projection image whose characteristics are determined by imaging parameters, (ii) classify at least one object in the X-ray projection image, (iii) receive a model of the classified object, and (iv) determine a 3D representation of the classified object and match a virtual projection of the model to the actual projection image of the classified object, thereby comprising at least one processing device configured to execute a computer program product including a set of instructions for localizing the classified object with respect to a coordinate system. This process may take into account the characteristics of the X-ray imaging method. In particular, as considered in a later example, the fact that the theorem of Thales is applicable may be considered.

[0023] The X-ray projection image may represent a target anatomical structure, particularly bone. The bone may be, for example, a bone of the hand or foot, i.e., a long bone of the lower limb such as the femur and tibia, and a long bone of the upper limb such as the humerus or vertebra or pelvis. The image may also include artificial objects such as surgical instruments (e.g., drills) or bone implants that are already inserted or fixed in the imaged anatomical structure of interest.

[0024] In the context of the present invention, an "object" and a "model" are distinguished. The term "object" is used for real objects, such as bones or parts of bones or other anatomical structures, or implants such as intramedullary nails, bone plates or bone screws, or surgical instruments such as sleeves, k-wires, scalpels, drills or aiming devices that may be connected to the implant. Also, the "object" may describe only a part of a real object (e.g., a part of a bone), or it may be an assembly of real objects and thus may consist of sub-objects. Also, to emphasize that an object is a sub-object of another object, it may be referred to as a "structure". For example, the "fixing hole" of a nail may be regarded as a structure (or sub-object) of the object "nail". As another example, the "pedicle" of a vertebra may be regarded as a structure of the object "vertebra". Nevertheless, a structure (such as the structure of the pedicle) itself may simply be called an "object".

[0025] The term "model" is used for the virtual representation of an object (or part-object or structure). For example, a dataset defining the shape and dimensions of an implant may constitute a model of the implant. As another example, a 3D representation of an anatomical structure, such as may be generated during a diagnostic method, may be regarded as a model of the actual anatomical object. Note that a "model" may describe a specific object, such as a specific nail or the left femur of a specific patient, or it may describe a class of objects, such as femurs in general, with some variability. In the latter case, such objects may be described, for example, by a statistical shape or appearance model. Thus, an object of the present invention may be to find a 3D representation of a specific instance from a class of objects depicted in an acquired X-ray image. For example, it may be an object to find a 3D representation of a vertebra depicted in an X-ray image acquired based on a general statistical shape model of vertebrae. It may also be possible to use a model that includes a discrete set of deterministic possibilities, and then the system selects which of these best describes the object in the image. For example, there may be several nails in a database, and then the algorithm identifies which nail is depicted in the image (if this information has not been provided in advance by the user).

[0026] Since a model is actually a computer dataset, it is readily possible to extract specific information such as the geometric aspects and / or dimensions of the object substantially represented by that data.

[0027] A model may include two or more parts of the imaged object, and in some cases one or more of these parts may not be visible in the X-ray projection image. For example, a model of an implant may include screws that are intended to be used with the implant, but if only the implant has already been introduced into the anatomical structure, only the implant is visible in the X-ray projection image.

[0028] Note that the model does not have to be a complete 3D model of the actual object in the sense that it only describes certain geometric aspects of the object, such as the fact that it can approximate the femoral head by a 3D ball and a circle in a 2D projection image, or the fact that the pedicle of the vertebra has a cylindrical shape.

[0029] According to one embodiment, a system for processing X-ray images generally comprises a processing device and a software program product, wherein when the software program product is executed by the processing device, the system is caused to perform the following steps. First, receive an X-ray image which is a projection image of at least a first object and a second object. Next, classify at least the first object and the second object and receive a 3D model of each of these objects from, for example, a database. Based directly on the X-ray image and / or based on each 3D model, determine and identify a first geometric aspect of the first object with respect to the 3D model of the first object, and determine and identify a second geometric aspect of the second object in the 3D model of the second object. The second geometric aspect may be a point.

[0030] Furthermore, based on the information that the 3D model of the first object, the 3D model of the second object, and the second geometric aspect (e.g., the point of the second object) are located on the geometric aspect of the first object, determine the spatial relationship between the first object and the second object. The geometric aspect of the first object may be a plane, a line, or a point. Of course, the geometric aspect may include a plurality of the above aspects and combinations thereof. As a result, the geometric aspect used in this specification may be a more complex shape such as the edge of a fragment in the case of a fracture.

[0031] According to one embodiment, the geometric aspect of the first object is a plane or a line, and an X-ray image is generated with the imaging direction tilted with respect to the geometric aspect at an angle in the range of 10° to 65°. In fact, the plane or line associated with the first object may be tilted with respect to the imaging direction. The X-ray image generated with the tilted imaging direction only needs to contain information about the first object sufficient to enable the determination of the geometric aspect. In other words, the appearance of the first object as visible in the X-ray image provides the processing device of the system with information sufficient to classify the object and automatically identify the geometric aspect. Also, the range of that angle may be 15° to 45°. Alternatively, the angle may be in the range of 20° to 30°. Assuming that the system gives an instruction to the user for the adjustment of the C-arm, the system may instruct the user to orient the imaging direction with respect to the geometric aspect of the first object, or may instruct the user to orient the geometric aspect of the first object at an angle of, for example, 25°.

[0032] According to a further embodiment, the system is further caused to determine a deviation of the 3D position and 3D orientation of the second object from the intended spatial relationship of the second object with respect to the first object.

[0033] For example, the first object may be an anatomical structure or an aspect of a first implant, and the second object may be an instrument or a second implant. As will be described in more detail below, the first object may be a vertebra, and the second object may be a pedicle screw. Alternatively, the first object may be an intramedullary nail, and the second object may be a fixation screw for distally fixing the nail. Alternatively, the second object may be a drill or a k-wire used to prepare a path for a screw to pass through into the bone. Alternatively, the first object and the second object may each be a bone fragment that must be anatomically reduced.

[0034] According to one embodiment, the selected point of the second object may be an object (e.g., the tip of a drill), and the information regarding the 3D position of the tip may be the contact point between the tip and the surface of the first object (e.g., the outer surface of a long bone such as a vertebra or femur).

[0035] According to one embodiment, the system may comprise a device for providing information to the user, where the information includes at least one piece of information from the group consisting of X-ray images and instructions regarding the treatment process. Of course, such a device may be a monitor for visualizing the information, or may be a loudspeaker for providing the information audibly.

[0036] According to yet another embodiment, the characteristics of the X-ray imaging device intended to be used together with the software program product executed on the processing device of the system may be known. On the one hand, the imaging characteristics may be known and taken into account when processing the X-ray image data, and on the other hand, for imaging based on the known geometric shape of the imaging device and the possibility of changing the position and orientation of the C-arm, it may facilitate instructions for the user to adjust the C-arm. The C-arm based X-ray imaging device may be part of the system.

[0037] The appearance of the object in the projection image may be affected by the X-ray imaging procedure. For example, imaging parameters such as the imaging direction with respect to gravity (which describes the direction in which the X-ray beam, also called the "viewing direction", passes through the object), zoom, radiation intensity, and / or the presence of a magnetic field may affect the appearance of the object in the projection image. These or additional imaging parameters may cause characteristic changes in the projection image such as deformation of the projected object due to the pillow effect, mechanical bending of the C-arm imaging device according to the imaging direction, curvature, noise, and / or distortion. Here, these changes are represented as image characteristics.

[0038] Of course, it may be possible to determine those image characteristics with sufficient accuracy in the projected image. For example, the position of the structure shown in the edge region of the image may be affected by the pillow effect more than the structure at the center of the image. As a result, based on a structure of a known shape extending from the edge region to the central region, the characteristics of the pillow effect can be determined with sufficient accuracy. The image characteristics determined for a region in 2D X-rays may be extrapolated to the entire image.

[0039] The appearance of an object in an X-ray image is further determined, inter alia, by the attenuation, absorption and deflection of the X-ray radiation, which are determined by the material of the object. The more materials the X-ray beam has to pass through, the less X-ray radiation is received by the X-ray detector. This can change not only the appearance of the object within its outline, but also the shape of the outline itself in the X-ray projection image, especially in regions where the object is thin. Also, the strength of this effect is determined by the X-ray intensity and the amount of tissue surrounding the object that the X-ray beam has to pass through. The latter is determined by the patient's body mass index and the imaging direction. The amount of soft tissue surrounding the object can be obtained from a database, which takes into account, for example, ethnicity, gender, body mass index, age.

[0040] Taking into account the image and object characteristics as well as the effects of X-ray attenuation, absorption and deflection, the virtual projection of the model may be deformed and / or distorted such that the object is deformed and / or distorted in the X-ray projection image. Such a virtual projection may then be made to match the projection seen in the X-ray image. Of course, matching the object to the model in the X-ray projection image may include adapting the image characteristics of the X-ray projection image to the image characteristics of the virtual projection of the model and / or adapting the image characteristics of the virtual projection of the model to the image characteristics of the X-ray projection image. Also of course, it may be possible to achieve a match in the 3D projection volume by minimizing the distance in 3D.

[0041] An X-ray beam originates from an X-ray source (focal point) and is detected by an X-ray detector in the image plane. Therefore, the physical dimensions of an object are related to the dimensions of its projection in the X-ray image by the theorem of Thales (also known as the basic proportionality theorem). An exact imaging depth (which is the distance of the object from the image plane) is generally not required in the context of the present invention. However, if the object is sufficiently large, the imaging depth may be determined by the theorem of Thales, and the larger the object, the more accurate this determination will be. Even in the case of small objects, an approximate estimation of the imaging depth may be possible. Alternatively, if the size of the X-ray detector and the distance between the image plane and the focal point are known, the imaging depth can also be determined.

[0042] According to one embodiment, a deep neural network (DNN) may be used for classifying an object (e.g., a proximal portion of a femur, a distal portion of a femur, a proximal portion of a nail, or a distal portion of a nail, etc.) in an X-ray projection image. Note that the object may be classified without determining its position by the DNN (see, e.g., Krizhevsky, A., Sutskever, I., and Hinton, G. E. “ImageNet classification with deep convolutional neural networks” in NIPS, pp. 1106-1114, 2012). It should be further noted that the object can be classified even when it is known which object should be made recognizable in the X-ray image. Also, a neural network may be used for roughly classifying the imaging direction (see, e.g., AP vs. ML, paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jurgen Schmidt: “Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge”, 2018 (available online at https: / / arxiv.org / abs / 1808.04441)). To follow the processing steps, an appropriate model may be selected using such classification of the object and the imaging direction. Note that the classification may be performed by other means or by a priori information regarding which object is visible in the image.

[0043] According to one embodiment, the outline of the classified object may be detected in the X-ray image. In the case of an object having a variable shape such as an anatomical structure, this may be advanced using the "deep morphing" technique described in the paper cited above by Pries et al. (2018). This paper proposes a technique based on a deep neural network for detecting bone structures in fluoroscopic X-ray images. This technique specifically addresses the challenges in the automatic processing of fluoroscopic X-rays, namely their low quality and the fact that only small datasets are typically available for training neural networks. This technique incorporates high-level information about objects in the form of statistical shape models. This technique consists of a two-step method (referred to as deep morphing), where in the first step, a neural segmentation network detects the contour (outline) of the bone or other object, and then in the second step, a statistical shape model is fitted to this contour using a variant of the active shape model algorithm (although other algorithms can also be used for the second step). This combination enables this technique to label points on the object contour. For example, in the segmentation of the femur, this technique can determine which points on the contour in the 2D X-ray projection image correspond to the lesser trochanter region and which points correspond to the femoral neck region, etc. Objects described by a deterministic model (e.g., a nail) may also be detected by deep morphing or simply by a neural segmentation network as in the first step of deep morphing.

[0044] In a further step, the virtual projection of the model may be adjusted taking into account the image and / or object characteristics as well as the effects of X-ray attenuation, absorption and deflection, to match the appearance of the object in the X-ray projection image. According to one embodiment, in the case of an object described by a deterministic model, this matching may proceed, for example, along the lines described in the paper by Laugier C. (ed.) "Geometric Reasoning for Perception and Action". GRPA 1991. Lecture Notes in Computer Science, Vol. 708. Springer, Berlin, Heidelberg: Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps". In this approach, the image and object characteristics as well as the effects of X-ray attenuation, absorption and deflection may be compensated by introducing additional degrees of freedom into the parameter vector or by using a suitably adjusted model.

[0045] The neural network may be trained based on a very large amount of data comparable to the data to which it is applied. In the case of evaluating the bone structure in an image, the neural network must be trained based on a very large number of X-ray images of the target bone. Naturally, the neural network may also be trained based on simulated X-ray images. The simulated X-ray images may be generated, for example, from 3DCT data as described in the appendix of the paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jurgen Schmidt: "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge" (available online at https: / / arxiv.org / abs / 1808.04441).

[0046] According to one embodiment, two or more neural networks may be used, where each of the neural networks may be trained for the sub-steps particularly necessary to achieve the desired solution. For example, the first neural network may be trained to evaluate X-ray image data in a 2D projection image to classify anatomical structures, and the second neural network may be trained to detect the position of the structure in the 2D projection image. The third network may be trained to determine the 3D position of the structure with respect to the coordinate system. It is also possible to combine the neural network with other algorithms such as, but not limited to, active shape models. Note that the neural network can also learn to localize an object or determine the imaging direction without the need to first detect the outline of the object in a 2D X-ray image. Also note that the neural network can be utilized for other tasks, such as determining one or more image characteristics such as the pillow effect.

[0047] According to one embodiment, an object may be manually classified and / or identified in an X-ray projection image. Such classification or identification may be supported by the apparatus by automatically referring to the structures recognized by the apparatus.

[0048] According to one embodiment, the system may calculate geometric aspects of an object (e.g., axes, planes, trajectories, outlines, curvatures, center points or one- or two-dimensional manifolds), and dimensions of the object (e.g., lengths, radii or diameters, distances). This may be achieved by correspondence between a model and a virtual projection that is matched to the projection seen in the X-ray image.

[0049] When displaying the X-ray projection image, geometric aspects and / or dimensions may be shown as an overlay in the projection image. Alternatively and / or additionally, at least a part of the model may be shown in the X-ray image, for example as a transparent visualization or 3D rendering, which may facilitate identification of structural aspects of the model by the user and thus of the imaged object.

[0050] The present invention provides 3D reconstruction and localization of an object whose shape and appearance have some variability. Such an object may be described, for example, by a 3D statistical shape or appearance model. This can be done based on a single X-ray image or a plurality of X-ray images. Based on one image of an anatomical object, the model may be deformed in such a way that its virtual projection matches the actual projection of the object in the X-ray image. When acquiring a plurality of X-ray images, the information from them may be fused (registered) to increase the accuracy of 3D reconstruction and / or determination of spatial position or orientation. If the imaging directions are known or can be determined, or if the 3D angles between the imaging directions (which may be represented, for example, by Euler angles) are known or can be determined when registering a plurality of X-ray images, the matching of the virtual projection to the actual projection in the X-ray image (e.g., using deep morphing) can be done with higher accuracy.

[0051] According to one embodiment, even when it is not possible to individually determine the position of at least one object with sufficient accuracy, the determination of the relative 3D position and 3D orientation between multiple objects is provided. This may be addressed by utilizing geometric a priori information regarding the relative 3D position and / or 3D orientation between at least two objects / structures in an X-ray image. This may be, for example, information that a point of one object lies on a line whose relative 3D position and orientation with respect to another object are known. Another example is that the relative 3D position and 3D orientation between a geometric aspect of one anatomical object and a geometric aspect of another anatomical object are known. Since some ambiguity may still remain, it may be necessary to limit the X-ray imaging direction to (i) a specific anatomically relevant view (e.g., the true ML) or (ii) an angular range that enables one of the objects to be viewed from a specific direction.

[0052] Note that the processing device may be implemented by only one processor that performs all steps of the process, or by a group or multiple processors that do not necessarily need to be located in the same place. For example, cloud computing enables processors to be located anywhere. For example, the processing device may be divided into (i) a first sub-processor in which a first neural network for evaluating image data including the classification of anatomical structures such as bone surfaces is implemented, (ii) a second sub-processor in which a second neural network specialized for determining the imaging direction of the classified anatomical structures is implemented, and (iii) a further processor for controlling a monitor for visualizing the results or a loudspeaker for audibly instructing the user. Also, one of these processors or a further processor may control the movement of the C-arm of the X-ray imaging device, for example.

[0053] According to one embodiment, the apparatus further comprises storage means for providing a database for storing, for example, X-ray images. It will be understood that such storage means may also be provided in a network (to which the system may be connected), and that data related to the neural network may be received through that network.

[0054] Furthermore, the apparatus may comprise an imaging device for generating at least one 2D X-ray image, where the imaging device may be capable of generating images from different directions.

[0055] The apparatus may further comprise input means for manually determining or selecting the position or part of an object in an X-ray image, such as a bone outline, for example, to measure distances in the image. Such input means may be, for example, a computer keyboard, a computer mouse or a touch screen for controlling a pointing device such as a cursor on a monitor screen, and these may also be included in the apparatus.

[0056] It should be noted that all references to movement or rotation of the C-arm in this application always refer to relative repositioning between the C-arm and the patient. Thus, any movement or rotation of the C-arm may generally be replaced by a corresponding movement or rotation of the patient / operating table, or a combination of movement / rotation of the C-arm and movement / rotation of the patient / operating table. This may be particularly relevant when dealing with the limbs, as it may be easier to move the patient's limbs than to move the C-arm. It should be noted that the required movement / rotation of the patient generally differs from that of the C-arm, and in particular typically no translation of the patient is required when the target structure is already in the desired position in the X-ray image. The system may calculate the adjustment of the C-arm and / or the adjustment of the patient.

[0057] The methods and techniques disclosed in this application may be used in systems that support human users or surgeons, or alternatively in systems where some or all of the steps are performed by robots. Thus, all references to "user" or "surgeon" in this patent application may refer to human users as well as robotic surgeons, mechanical support devices, or similar devices. Similarly, whenever it is mentioned that an instruction on how to adjust the C-arm is given, such an adjustment may of course be performed automatically by the robotic C-arm without human intervention, or alternatively by the surgical staff with some automated support. Note that since robotic surgeons and / or robotic C-arms can perform surgeries with higher accuracy than humans, repetitive procedures may require fewer repetitions and may be able to execute more complex instructions (e.g., combining multiple repetitive steps).

[0058] The computer program product may preferably be loaded into the random access memory of a data processor. Thus, according to one embodiment, the system may be provided with a data processor or processing device for performing at least a part of the described process. Furthermore, the present invention relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program can be stored. However, the computer program may also be provided through a network such as the World Wide Web and can be downloaded from the random access memory of a data processor from such a network. Additionally, the computer program may be executed on a cloud-based processor and the results may be provided through the network.

[0059] Note that prior information regarding the implant (e.g., the size and type of the nail) may be obtained by simply scanning whatever is described on the implant's packaging (e.g., barcode) or the implant itself before or during the surgery.

[0060] For a further understanding of the present invention, an exemplary method of inserting a bone screw through a hole of a bone nail into the long bone for distal fixation of the bone nail in the long bone is described. This hole has a hole axis, which may be regarded as a geometric aspect of the nail. The method may include the step of disposing the drill with its tip in contact with the outer surface of the long bone such that the tip of the drill is positioned on the hole axis of the bone nail hole, and the drill axis of this drill is oriented at an angle of 10 to 70° with respect to the hole axis. Here, the hole axis is a line as a geometric aspect of the first object, i.e., the bone nail, and the tip of the drill is a point as a geometric aspect of the second object, i.e., the drill.

[0061] A first X-ray image of the bone drill and the bone nail in the long bone with the tip of the drill on the outer surface of the long bone is generated with the imaging direction in the direction of the hole axis of the bone nail. One skilled in the art will understand that the imaging direction may be the true medial-lateral direction, and thus the hole should be visible as a circle in the case of distal fixation of the femoral nail.

[0062] The system may then determine the actual angle between the drill axis and the hole axis based on the knowledge of the contact points, based on the 3D model of the drill, and based on the 3D model of the bone nail. Based on the determined angle, the system may give an instruction to change the orientation of the bone drill such that the tip is still on the bore axis and the drill axis is close to the hole axis. Here, "close" means having a deviation of up to 15° from the hole axis. It may be sufficient to merely align the drilling trajectory with the imaging direction.

[0063] While maintaining the position of the drill tip, a second X-ray image of the drill and the bone screw in the long bone may be generated with the second imaging direction oriented at an angle of 10 to 65° with respect to the first imaging direction. An easy way to change the orientation, when starting from the internal-external imaging direction, may be to move only the C-arm in the anteroposterior direction. Also here, it is crucial that the drill, i.e., the second object, is sufficiently visible in the next X-ray image in order to enable, for example, the automatic determination of the position and orientation of the drill axis. Therefore, the angle may be in the range of 10 to 65°, preferably 15 to 45°, and most preferably 20 to 30°.

[0064] Based on the second X-ray image, the deviation of the 3D position and 3D orientation of the drill axis from the hole axis of the bone screw hole may be determined. In case of deviation, the position and orientation of the bone drill may be adjusted, and the bore may be drilled with the drill through the bone screw hole in the long bone. Drilling along the target trajectory with the drill at the same angle as the imaging direction with respect to the drill axis may be confirmed by one or more X-ray images.

[0065] The principle of the present invention may also be applied to a method of inserting a bone screw into the pedicle of a vertebra. The method may include the step of arranging the drill with its tip in contact with the outer surface of the vertebra such that the tip of the drill is positioned on an axis extending through the pedicle of the vertebra, and the drill axis of this drill is oriented at an angle of 10 to 65° with respect to the target axis passing through the pedicle.

[0066] Generate a first X-ray image including the drill and the vertebra in an imaging direction, for example, from a true AP, such that the opening of the pedicle is clearly visible as in the above method. The difference between these two methods can be recognized in that the imaging direction of the first X-ray image can be regarded as the anteroposterior direction in a state where the patient lies flat on their back, and in that the imaging direction does not need to coincide with the pedicle axis (target trajectory) because both objects are in contact with each other. Using such a tilted view, the relative 3D position and 3D orientation between the drill and the pedicle may be determined based on the knowledge of the contact point.

[0067] As a next step, based on the 3D model of the drill and also based on the 3D model of the vertebra, the actual angle between the drill axis and the axis passing through the pedicle may be determined. In accordance with an indication that can be provided by the present system, the orientation of the drill may be changed such that the tip of the drill is still on the axis passing through the pedicle and the drill axis is near the target axis passing through the pedicle. If the inclination of the pedicle axis with respect to the viewing direction is sufficiently large so that neither the power tool nor the hand obstructs the view, a second X-ray image may optionally be generated from the same direction. Typically, the inclination between the pedicle axis and the true AP viewing direction of the corresponding vertebra is 10 to 45°. If necessary, the position and orientation of the drill may be adjusted, and then a hole may be drilled through the pedicle into the vertebra.

[0068] The principles of the present invention may also be applied to a method of inserting a bone screw into the joint of the sacroiliac (SI) joint. The method may include positioning the drill such that its distal tip is in contact with the outer surface of the vertebra and its distal tip is positioned on an axis extending through a desired drilling path through the SI joint, and the drill axis of this drill is oriented at an angle of 10 to 65° with respect to the target axis passing through the pedicle.

[0069] As in the method described above, with the imaging direction coinciding with the direction of the drilling path with the drill, a first X-ray image is generated that includes the relevant portion of the drill, the ilium, and the sacrum. As a next step, based on the knowledge of the contact points, based on the 3D model of the drill, and also based on the 3D model of the vertebra, the actual angle between the drill axis and the axis passing through the drilling path with the drill may be determined. In accordance with an indication that can be provided by the present system, the orientation of the drill may be changed such that the tip of the drill is still on the axis of the drilling path with the drill and the drill axis is near the target axis of the drilling path with the drill.

[0070] Next, generate a second X-ray image of the drill, the relevant part of the ilium, and the relevant part of the sacrum with the drill and imaging directions swapped. At this point, the drill may be approximately on the target trajectory passing through the drilling path with the drill, and the second imaging direction may be oriented at an angle in the range of 10 to 65° with respect to the first imaging direction. The C-arm of the X-ray imaging device may be rotated with respect to the tilted viewing direction. Also, the angular range between the two imaging directions may be 15 to 40°. Alternatively, the range may be 20 to 30°. Using such a tilted view, the deviation of the 3D position and 3D orientation of the drill axis from the target axis passing through the drilling path with the drill may be determined. If necessary, adjust the position and orientation of the drill, and then drill holes in the ilium and SI joint.

[0071] As is apparent from the above description, the main aspect of the present invention is the processing of X-ray image data that enables automatic interpretation of visible objects. The method described herein should be understood as a method for assisting in the surgical treatment of patients. Thus, according to one embodiment, the method should not include any steps of treating the body of an animal or human by surgery.

[0072] It should be noted that the embodiments are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims (computer programs), and other embodiments are described with reference to apparatus-type claims (systems / devices). However, those skilled in the art will presumably infer from the above and following descriptions that any combination of features belonging to one type of subject and any combination of features related to different subjects are considered to be disclosed by this application, unless otherwise specified.

[0073] The aspects, further aspects, features, and advantages defined above of the present invention can also be obtained from the examples of the embodiments described later in this specification and are described with reference to the examples of the embodiments shown in the drawings, but the present invention is not limited thereto.

Brief Description of the Drawings

[0074]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

[0075] Throughout the drawings, the same reference numerals and letters are used to denote similar features, elements, components, or parts of the illustrated embodiments, unless otherwise specified. Further, the present disclosure will be described in detail with reference to the drawings, which relate to exemplary embodiments and are not limited by the specific embodiments shown in the drawings.

Mode for Carrying Out the Invention

[0076] 3D Reconstruction and Localization of Anatomical Objects Based on a Single X-ray Image The paper (2018) cited in relation to deep morphing by Pries et al. proposes a method that enables the system to detect the outline / contour of the bone (in 2D projection images) and label the points on the contour. For example, in the segmentation of the femur, this technique can determine which points on the contour in the 2D X-ray projection image correspond to the lesser trochanter and which points correspond to the femoral neck. Then, considering the 3D statistical shape or appearance model of the same anatomical structure, the model can be deformed in such a way that its virtual projection matches the actual projection in the X-ray image, thus obtaining a 3D reconstruction of the anatomical structure and enabling the determination of the position of the object and the imaging direction. On the other hand, if the imaging direction is already known, the 3D reconstruction of the anatomical object can be performed with higher accuracy. For example, the imaging direction may be known if the surgeon is instructed to acquire X-ray images in a specific direction (e.g., true AP or true ML), or if the specific imaging direction has been detected by an algorithm according to the invention by Blau filed as a patent application on August 23, 2018.

[0077] The accuracy of the 3D reconstruction of the anatomical object may be improved by using a priori information. This a priori information may be the patient's size or gender, or it may be more anatomical structure-specific information such as geometric information about the anatomical object to be reconstructed. In the case of an example of 3D reconstruction of the proximal femur based on ML images, this information may include the length of the femoral neck or the CCD angle. However, since such information may not be determined with sufficient accuracy in typical ML images, it may be extracted from the AP images acquired daily earlier in the process of surgery for the proximal femur. The more information is used from earlier images, the more accurate the 3D reconstruction can be. Another way of describing this procedure is that based on the AP image, the 3D reconstruction of the proximal femur can be performed due to typical remaining uncertainties (such as the width of the femoral neck in the AP direction), and this 3D reconstruction can function as a starting point for 3D reconstruction based on a priori information or later ML images.

[0078] The geometric a priori information may also consist of known correspondences between points in a 2D projection image and points in a 3D model of an anatomical object. For example, - the point in the 2D projection image corresponds to a point on a line where its position and orientation relative to the 3D model of the anatomical object are known, or - the point in the 2D projection image corresponds to a point on a plane where its position and orientation relative to the 3D model of the anatomical object are known When this is known, less specific geometric information may still be useful.

[0079] Such geometric a priori information may be provided, for example, by a user input via a user interface or by a surgeon who positions an object (e.g., an instrument such as a drill or a k-wire) at a specific anatomical point visible in the 2D projection image. This may be achieved optionally due to prominent anatomical features in a specific imaging direction (e.g., true AP or true ML), or by palpation or visual identification of the actual object. All of this a priori information significantly reduces ambiguity in 3D reconstruction.

[0080] Determination of the relative 3D position and 3D orientation between objects by reducing or eliminating ambiguity In the invention by Blau filed as a patent application on November 26, 2018, it is considered how to determine the relative 3D position and 3D orientation between both objects when two objects can be located based on a 2D X-ray image and it is known that the two objects are in contact with each other in a physical 3D space.

[0081] In the present invention, it is proposed how to determine the relative 3D position and 3D orientation between two (or more) objects (or structures) based on 2D X-ray images when at least one object cannot be located with sufficient accuracy. Such determination of the relative 3D position and 3D orientation may be possible based on a priori information regarding the 3D position of a specific point of one object relative to another object, which may be obtained, for example, from an X-ray image acquired previously from a specific imaging direction that enables the location of at least one structure of the other object. To make this work, it may be necessary to limit the acceptable range of the imaging direction in the current image.

[0082] For the sake of illustration, assuming that one of these objects is a drill and the other object is a nail or some anatomical object (e.g., vertebra), this will be described next. The anatomical object may be described using either a deterministic 3D model (generated, for example, using 3D imaging methods for a specific patient) or a statistical 3D model that describes general bone variability. In the former case, higher accuracy can be obtained. Even if a complete and accurate 3D model of the drill is available, it is not always possible to locate the drill with sufficient accuracy. As explained above, there is remaining uncertainty in the imaging depth (distance of the object from the image plane) when locating the object. This uncertainty in the imaging depth in the determination of the 3D position of the drill tip also causes ambiguity regarding the inclination of the drill in the direction of the imaging depth. As shown in FIG. 8, the inclination of the drill is defined as the viewing angle with respect to the tip of the drill, represented by 8.DT. Since the drill is a very thin and straight structure with a clearly defined axis, the inclination of the drill may also be defined as the angle between the dashed line represented by 8.L1 connecting the tip 8.DT of the drill and the X-ray focus 8.FP and the solid line represented by 8.L2, which is the drill axis.

[0083] Consider, for example, a 3D arrangement having two different 3D drill positions (represented as 9.D1 and 9.D2) shown in FIG. 9. The two drill positions differ in imaging depth and also (by 6.5°) in drill inclination. Further, the two drill positions in the X-ray projection image result in essentially the same X-ray projection, as shown by substantially the same drill outline 10.O in FIG. 10, and may not be distinguishable.

[0084] There are two main reasons for this, as follows.

[0085] 1. The drill is a relatively thin device of approximately constant diameter, which is simply a few pixels wide in a typical X-ray image. In an X-ray projection, an object tilted in the imaging direction is depicted wider at one end and smaller at the other. However, this can only be detected if this change in width is sufficiently drastic (e.g., at least one pixel wide). For example, in the case of an angle detection of less than 3° for a thin drill with a diameter of less than 4 mm, this generally may not apply.

[0086] Since the sine function of a small angle has a slope close to zero, it is generally not possible with sufficient accuracy to locate devices such as a drill (having a diameter of a few millimeters) where only the front part is visible in the X-ray image from a viewing angle close to 90° (which is the drill inclination). For example, at an angle of 70°, the projection of the drill is only 6 percent shorter, resulting in a slight change in the projected shape of the drill tip. Such a small change may not be sufficient to determine the drill inclination with an accuracy of about 3°. The limit of detectability is an inclination of about 65°, where the projection of the drill is 9.4 percent shorter. Depending on the instrument, this may or may not be sufficient for the required accuracy.

[0087] The smaller the viewing angle (tilt), the easier it is to distinguish, for example, a difference in tilt of 2°. This is shown in the X-ray image of FIG. 11 showing the projections and outlines of two drills, where the solid white line labeled 11.D1 corresponds to a drill with a tilt of 45°, and the dashed white line labeled 11.D2 corresponds to a drill with a tilt of 43°. Since these outlines differ at several locations, they can be distinguished by the present system. The smaller the viewing angle, the more clearly distinguishable outlines are obtained. This can be observed in the X-ray image of FIG. 12 showing the projections and outlines of two drills. The solid white line labeled 12.D1 corresponds to a drill with a tilt of 25°, and the dashed white line labeled 12.D2 corresponds to a drill with a tilt of 23°. These outlines are clearly different at several locations at this point and can thus be easily distinguished by the present system.

[0088] 2. In a typical X-ray image, only the tip and the upper part of the drill are visible, while the other end of the drill is not visible. If both ends were visible, the tilt of the drill could be determined with high accuracy in the projection image based on the shortened length of the drill. This is also possible if the drill has markings that are clearly visible in the X-ray image, for example, partway along the shaft. Marking such a drill is easy, but it also means a change in existing instruments. Another option is to use the start of the thread of the drill as such a marking. However, when using a general drill, the start of the thread may not be clearly visible enough in the X-ray projection image, so it may generally not be possible to use it for this purpose.

[0089] This problem may be addressed by a more accurate determination of the imaging depth of the drill tip, thereby sufficiently reducing or eliminating the ambiguity. This may be possible when determining the relative 3D position and 3D orientation between two objects in an X-ray image. Another object may be referred to as the target object, which may be, for example, a nail. If the position of the drill tip is on this trajectory and the imaging direction of the trajectory at the position of the drill tip is sufficiently large (in other words, this trajectory must be sufficiently different from a line parallel to the line connecting the drill tip and the X-ray focus), for example, by defining a trajectory for which its 3D position and 3D orientation relative to the target object are known, the ambiguity of the position of the drill tip relative to the target object may be reduced.

[0090] It may be even more useful when the 3D position of the drill tip relative to a point on the target object is known. This is the case, for example, when the drill tip is in contact with the target object, such as when the drill tip is in contact with the ilium in sacroiliac screw fixation. However, it may still be sufficient if it is known that the drill tip is on a plane for which its 3D position and 3D orientation relative to the target object are known. This is the case, for example, for the distal fixation of a further hole after the completion of the fixation of the first hole. Here, two degrees of freedom (DoF) are not determined. For a more detailed description of the distal fixation procedure for the nail, further reference may be made to the corresponding section below.

[0091] Furthermore, there may be ambiguity regarding the target object, especially when the target object is an anatomical object. However, when the device is in contact with the target object, there are imaging directions in which the ambiguity in localizing each of the objects is involved in different directions. Therefore, even in such cases, a sufficiently accurate determination of the relative 3D position and 3D orientation may be possible.

[0092] Next, the following shows these ideas for a drill in contact with the trochanter of the proximal femur. In this imaging direction, for example, it may be possible to clearly determine at which point the drill tip is in contact with the femur by palpation. Due to the ambiguity as described above, there are several possibilities corresponding to different relative 3D positions and 3D orientations of the anatomical structures depicted, each for the 3D position and 3D orientation of the drill with respect to the femur where the same projection of the drill occurs in the X-ray image. By considering the depicted anatomical structures together and using a priori information regarding the contact point, it may be possible to select which of these possibilities is the correct one.

[0093] Figure 13 shows an X-ray image of such a scenario. The solid white line (represented by 13.CO) is the outline of the femur corresponding to the correct 3D position and 3D orientation of the femur, and the dashed white line (represented by 13.IO) is the outline of the femur corresponding to one of the inaccurate possibilities for the 3D position and 3D orientation of the femur. By comparing these possible outlines with the segmented and labeled femur in the X-ray image (which may be achieved, for example, by deep morphing), the one that best matches the segmented femur is selected. In the depicted scenario, the inaccurate outline 13.IO is clearly different from the correct outline 13.CO, and thus may be discarded even if the inaccurate outline corresponds to only a 2.5° angular error (with respect to the inclination of the drill). Larger angular errors may result in even more clearly inaccurate outlines as depicted in Figure 14, where the inaccurate outline 14.IO corresponds to a 6° angular error (with respect to the inclination of the drill) and is clearly distinguishable from the correct outline 14.CO.

[0094] A further example may be the determination of the relative 3D position and 3D orientation of an instrument with respect to the pedicle of a vertebra. FIG. 15 shows an AP X-ray image of the lumbar spine, where a surgeon has placed a Jamshidi needle (labeled 15.JN) in the right pedicle of the lumbar spine. The opening of the pedicle (labeled 15.OP) is clearly visible as a brighter region in this particular imaging direction. Thus, the center of the pedicle can be clearly identified, and the instrument can be placed at the center of this opening (the pedicle axis). Based on the a priori information that the instrument is placed on the pedicle axis, it is brought into contact with the bone surface, and the relative 3D position and 3D orientation between the instrument and the pedicle can be determined within a suitable angular range for many other imaging directions according to the method outlined above.

[0095] The process of aligning two or more X-ray images from different directions Depending on the shape of the bone, there may still be remaining ambiguities or registration errors in 3D reconstruction based on only one image. This may be reduced by rotating and / or translating the C-arm between images to potentially acquire multiple images from different viewing directions. In general, additional images from different imaging directions are more useful, and the more different the imaging directions (e.g., AP and ML images), the more useful the additional images are in terms of determining 3D information. However, it can even be beneficial to add images from only slightly different viewing angles that can be more easily acquired during surgery instead of completely changing to different views (from AP to ML or vice versa).

[0096] The present invention also enables recording of multiple X-ray images of at least one common object taken from different directions. This is important because 3D alignment enables determination of the relative 3D position between multiple objects without an explicit determination of the imaging depth.

[0097] For the 3D reconstruction of a deformable object (typically, an anatomical structure described, for example, by a statistical shape or appearance model and referred to as "object F" in this section) based on two or more X-ray images, the procedure outlined above for one image may be extended to two or more images. That is, deep morphing may be used to detect the contour of object F and label the points on that contour in each 2D X-ray image. Considering the 3D statistical shape model of object F, the model can then be deformed so that its virtual projection coincides as closely as possible with the actual projection of object F in two or more X-ray images simultaneously. This procedure implicitly determines the imaging direction of each X-ray image and thus does not require a priori information regarding the imaging direction.

[0098] As an alternative method for the alignment of a pair of X-ray images taken from two different imaging directions, it may be possible to increase the accuracy of the alignment process by taking into account the 3D angle between the imaging directions, which can be determined using two different procedures. The more accurately this angle can be determined, the more accurate the 3D alignment can be.

[0099] One way to determine this angle is, for example, to determine the imaging direction using the invention by Blau filed as a patent application on August 23, 2018 for each X-ray image, and to compare their differences. Another way may be to utilize another object (referred to as "object G") in the X-ray image, and its model (for example, a nail optionally connected to a sighting device or instrument) is deterministic. Object G may be located by matching the virtual projection of object G to its actual projection in each X-ray image. Note that without further conditions or a priori information, some objects, particularly instruments such as drills or k-wires, may not generally have a geometric structure or size sufficient to be located. However, even in such cases, it may be possible to locate object G with sufficient accuracy provided that (i) it is visible within a specific angular range in all images in which object G is aligned, and (ii) some prior information regarding the relative 3D position between objects F and G is available. The prior information in (ii) is particularly (a) the relative 3D positions of the points of object G and the points of object F are known, or (b) the points of object G are on a line within a physical 3D space where its relative 3D position and 3D orientation with respect to object F are known, or (c) the points of object G are in a plane within a physical 3D space where its relative 3D position and 3D orientation with respect to object F are known may be such.

[0100] However, the relative 3D position and 3D orientation between objects F and G must be the same in both X-ray images, that is, they must have as little movement as possible between the objects.

[0101] Generally, two or more images may be aligned if they include objects that can be located with sufficient accuracy. If two or more objects are included such that they do not move relative to each other during image acquisition, image alignment can be performed with high accuracy. One example where this procedure can be used is a situation where an implant (e.g., a nail) has already been inserted into bone and a 3D model of the bone (e.g., a statistical shape model) is available. In such a scenario, if a drill is visible in all images in different orientations and its tip (which is also visible in all images) remains at the same point (e.g., a point on the bone surface), the 3D position of the tip of the drill can be determined relative to either of the two objects. This means that in the first image, the device may be positioned at any point and in the second image (obtained from a different viewing direction), the orientation of the device may be changed (e.g., by aiming approximately at the target trajectory), but the tip of the device remains at the same position. Based on the location of the target object / structure (e.g., nail / nail hole), both images may be aligned, which may enable determination of the 3D position (tip of the device) of the point relative to the target object. This point may then be used to determine the relative 3D position and 3D orientation between the instrument and the target object / structure with sufficient accuracy.

[0102] In other words, a software program product executed on a processing device of the system may cause the system to receive a first X-ray image that is a projection image of at least one object, classify the at least one object, and determine at least one point in the first X-ray image. The system may then be caused to receive a second X-ray image that is a projection image generated in an imaging direction different from the imaging direction used to generate the first X-ray image. In the second image, the at least one object is classified again and at least one point is determined. Based on the classification of the at least one object in the first and second X-ray images and based on the determination of the at least one point in both X-ray images, the two images can be aligned and the 3D position of the point with respect to the at least one object can be determined.

[0103] If the at least one object includes two objects and the at least one point is a point of one of the two objects, the system may determine the spatial relationship between the two objects, i.e., the 3D orientation and 3D positioning, based on the aligned images.

[0104] Furthermore, the system may determine a deviation of the 3D position and 3D orientation of one of the objects from the intended spatial relationship of one object with respect to another object. For example, one object may be a drill and it is intended to be placed parallel to and on a trajectory passing through another object that may be a bone or an implant.

[0105] The above method of aligning two X-ray images may also be useful, for example, for 3D reconstruction and / or localization of an anatomical object in which a known implant is inserted. Localizing the implant enables alignment of the images, which in turn enables determination of the 3D position (with respect to the implant) of the tip of the drill located on the bone surface. The points thus determined may function as anchor points for 3D reconstruction and / or localization of the anatomical object. According to this approach, it may be possible to determine a plurality of surface points, which means sampling the 3D bone surface at separate points with respect to the implant, thereby obtaining a point cloud. Each sample point added to this point cloud may reduce the ambiguity in determining the 3D reconstruction and the 3D position and 3D orientation of the anatomical structure with respect to the implant. When the inclination of the drill is within the range of 10 to 55°, this may also enable determination of the 3D position and 3D orientation of the anatomical structure (or implant) with respect to the drill. Thus, even if a deterministic 3D model (e.g., a CT scan) of the anatomical structure is available, the 3D position and 3D orientation can be determined using this procedure. Also, even when there is no known implant at a fixed position on the bone, a method of sampling points can be used. In such a case, reconstruction and / or localization and / or alignment proceed directly based on the anatomical structure.

[0106] In the following, the influence of factors such as the C-arm width, the size of the image detector, and zoom on 3D alignment is shown by way of example. It is shown that determination of the imaging depth is not necessary in all of these examples.

[0107] Effect of C-arm width: Figure 1 shows the left femur (represented as LF) and the nail implant together with the attached aiming device (represented as NAD). Further, it shows the AP X-ray image (represented as 1.AP) and the ML X-ray image (represented as 1.ML) as well as their corresponding foci (represented as 1.FP.AP and 1.FP.ML). The 3D ball approximates the femoral head (represented as FH), and the dashed white circles are its 2D approximated projections in the images (represented as 1.FH.AP and 1.FH.ML). The C-arm has a width of 1000 mm (here defined as the distance between the focus and the image plane). The cone shows a part of the X-ray beam passing through the femoral head. Throughout this application, the inventors follow the convention of calling the image taken in the anteroposterior direction "AP" and the image taken in the posteroanterior direction "PA" image. Similarly, the inventors call the image taken in the mediolateral direction "ML" image and the image taken in the lateral-medial direction "LM" image.

[0108] In Figure 2, instead of the true 1000 mm, the C-arm width was inaccurately estimated as 900 mm. Therefore, all objects in the image including the femoral head (FH) appear smaller in the X-ray image than they should be. Thus, it looks as if the objects are moving towards the AP image plane (represented as 2.AP) and towards the ML image plane (represented as 2.ML). The corresponding foci are represented as 2.FP.AP and 2.FP.ML. The 3D reconstruction of the femoral head (FH) based on the 2D projections of the approximated femoral head (white circles 2.FH.AP and 2.FH.ML) remains unchanged compared to Figure 1. The only parameter that has changed is the apparent imaging depth. However, since the relative 3D positions of the femoral head and the nail have not changed, the imaging depth is not relevant in this scenario.

[0109] To show that the only difference between Figure 1 and Figure 2 is the apparent imaging depth, Figure 3 shows both scenarios simultaneously.

[0110] Effect of zoom: When one of the images is captured with a zoom ratio, the object appears larger than without zoom. In the case of Figure 4, the AP image (represented by 4.AP) was captured with a zoom ratio of 1.5. Therefore, all objects in the image including the femoral head (FH) appear as if they have moved towards the focus in the AP (represented by 4.FP.AP). As before, the 3D reconstruction of the femoral head (FH) based on the 2D projection of the approximated femoral head (dashed white circles 4.FH.AP and 4.FH.ML) remains unchanged compared to Figure 1. The only parameter that changes is the apparent imaging depth. However, since the relative 3D positions of the femoral head and the nail do not change, the imaging depth is not relevant in this scenario. The same applies when both images have zoom. Figure 5 compares a scenario with zoom (similar to Figure 4) and a scenario without zoom (similar to Figure 1).

[0111] Effect of the size of the X-ray detector: If the assumed size of the X-ray detector is 12’’ instead of the true 9’’, the object appears larger in the image and seems as if the object has moved towards the focus in both images. This is shown in Figure 6, where: · 6.AP.9’’ refers to the AP image by a 9’’ X-ray detector with a focus represented by 6.FP.AP.9’’ · 6.AP.12’’ refers to the AP image by a 12’’ X-ray detector with a focus represented by 6.FP.AP.12’’ · 6.ML.9’’ refers to the ML image by a 9’’ X-ray detector with a focus represented by 6.FP.ML.9’’ · 6.ML.12’’ refers to the ML image by a 12’’ X-ray detector with a focus represented by 6.FP.ML.12’’.

[0112] This effect is equivalent to the zoom ratio applied to both images. Therefore, the same conclusion as in the case of zoom is drawn.

[0113] Measurement of the characteristics of the classified objects The present invention does not require a priori calibration. If a known object is present in the image, located near (at a similar depth to) the structure being measured, the measurement may be made in millimeters. Since the known object has known dimensions, it can be used to calibrate the measurement values. This is similar to the procedure proposed by Baumgaertner et al. for determining the TAD value (see Baumgaertner MR, Curtin SL, Lindskog DM, Keggi JM: The value of the tip-apex distance in predicting failure of fixation of peritrochanteric fractures of the hip. J Bone Joint Surg Am. 1995, 77:1058-1064).

[0114] Example 1: A nail is inserted and an AP image is available. The nail is identified and located. Since the nail is located in the middle of the diaphysis and thus at a similar imaging depth to the depicted outer cortex of the diaphysis, the known geometric shape of the nail can be used for calibration. This makes it possible to provide scaling for determining the distance between the nail axis and the outer cortex of the diaphysis.

[0115] Example 2: If the imaging depth of object A is known (e.g., because object A is large enough or the size of the X-ray detector and the distance between the image plane and the focus are known), and information exists regarding the difference in imaging depth between object A and object B (e.g., based on anatomical knowledge), it may even be possible to calculate the size of a different object (referred to as "object B") at different imaging depths based on the theorem of Thales.

[0116] Handling of image distortion for an intramedullary nail example Generally, there are two ways to handle image distortion, and these may be combined. 1. Emphasize the regions in the X-ray image that are known to have strong distortion (e.g., the image boundaries) less, and place more emphasis on the regions that are less affected by the distortion. 2. Determine the distortion and compensate for it.

[0117] Next, these will be illustrated in an example of an AP image of the femur containing the inserted pins.

[0118] Reference 1. The following characters are used for the labeling in Figure 7. The solid lines are the contours of the pins and the aiming device as seen in the distorted X-ray image. The white dashed lines indicate the hypothetical outlines of the pins and the aiming device as shown in an undistorted image. 7.D: Distal part of the intramedullary nail 7.C: Central part of the nail containing the hole for the neck screw 7.P: Proximal part of the intramedullary nail 7.A: Aiming device

[0119] Typically, 7.D is located in a more distorted region of the X-ray image. Further, the exact position of 7.D is not important when predicting the trajectory of the screw inserted through the hole at 7.C. Therefore, in predicting the screw trajectory, a higher weight may be given to the positions of 7.C and 7.P than to 7.D, and the exact weighting may be determined based on their visibility and the reliability of detection. Also, a higher weighting for 7.C and 7.P may be reasonable since these regions are closer to the regions of interest (the screw hole and the femoral head). Further, the appearance of 7.C has information regarding the rotation of the nail around its axis.

[0120] Reference 2. The distortion in the image may be determined as follows. a) A surgical procedure performed earlier (which can be learned for a specific C-arm) b) Pre-surgical calibration: Known objects (e.g., nails, k-wires, etc.) can be placed directly on the image intensifier / X-ray detector at a known distance relative to the image plane. This can also be used to determine the size of the X-ray detector and the distance between the focus and the image plane. c) Images acquired earlier (which can be learned by the algorithm during surgery) d) A database with typical distortion effects (e.g., typical pillow effects for typical C-arm positions, the earth's magnetic field). The device may use the knowledge that the digital radiography device does not cause distortion.

[0121] If such information is available, it may be used when matching the virtual projection of the model to the projection in the X-ray image. The distortion may be applied to the entire image or particularly to the shape being matched.

[0122] Instead and / or in addition, during the process of matching the virtual projection of an object (e.g., a nail) having a known deterministic 3D model to the appearance of the object in the X-ray projection image, the distortion may be determined explicitly or implicitly. According to one embodiment, this matching may proceed along the lines described in the paper by Laugier C. (ed.), "Geometric Reasoning for Perception and Action". GRPA 1991. Lecture Notes in Computer Science, Vol. 708. Springer, Berlin, Heidelberg: Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps". According to one embodiment, the distortion may be described by a suitable mathematical model (e.g., Gronenschild E., "Correction for geometric image distortion in the X-ray imaging chain: local technique versus global technique", Med Phys., 1999 Dec;26(12):2602-16), such as a radial function and / or a sigmoid function. The distortion thus modeled may be compensated by introducing additional degrees of freedom into the parameter vector of the paper cited above by Lavallee et al. (1993) when matching the virtual projection to the projection in the X-ray image.

[0123] Handling of the exchange of the positions of the X-ray source and the receiving device The X-ray imaging device enables left-right flipping of the image, and since the present invention does not use a calibration reference attached to the image detector throughout the surgical procedure, even when the treatment side (left or right bone) is known, it may not be necessary to detect the exchange of positions of the X-ray source and the receiving device. The user may be required to provide information regarding whether the left-right flipping function is activated or not.

[0124] However, even in the absence of such information, it is possible to detect the exchange of positions of the X-ray source and the receiving device. This is because in the X-ray image viewed from an imaging direction much smaller than 90°, the instrument or device extends over a large range in the imaging depth. Therefore, since the part of the instrument or device closer to the imaging plane (receiving device) is depicted smaller than the more distant part, it is possible to detect the exchange between the X-ray source and the receiving device.

[0125] Method for determining rotation / left-right flipping / up-down flipping of an X-ray image The present system provides a method that enables it to determine the rotation, left-right flipping, and up-down flipping of an X-ray image. Using this, for example, the X-ray image may be displayed such that the nail appears as if it is actually positioned in front of the surgeon.

[0126] The following is known to the present system based on the surgical procedure being performed. · Patient positioning (e.g., lying on the back) · Position of the C-arm (e.g., the image intensifier is on the inside and the X-ray source is on the outside) · Which part of the patient's body the surgery is being performed on (e.g., left / right leg: this may be known, for example, based on a previous proximal procedure, user input, or scanning of the nail packaging)

[0127] Detection of left - right inversion (i.e., determining which side is the front side and which side is the back side) may be based on the determination of the implant direction, optionally supported by the determination of the imaging direction with respect to the anatomical structure (e.g., when the patient is supine, the forehead is directed downward in the image). Alternatively, the user may be instructed to orient an instrument (e.g., a drill) in a specific direction, and then this may be used to identify this direction as the front side or the back side.

[0128] Method for positioning a target object / structure in the field of view of a C - arm from a desired viewing direction For the definition of the rotation axes of the C - arm, refer to FIG. 16. In this figure, the X - ray source is represented by XR, the rotation axis represented by the letter B is called the vertical axis, the rotation axis represented by the letter D is called the propeller axis, and the rotation axis represented by the letter E is called the C - axis. Note that in the case of some C - arm models, the axis E may be closer to the axis B. The intersection of the axis D and the central X - ray beam (labeled by XB) is called the center of the "C" of the C - arm. The C - arm may move up and down along the direction indicated by the letter A. The C - arm may move along the direction indicated by the letter C. This terminology is used throughout this application. The distance of the vertical axis from the center of the "C" of the C - arm may vary between C - arms.

[0129] The following proposes a method for instructing a user on how to adjust the C-arm so that the target structure appears at a desired position in the X-ray projection image and the structure is visible from the desired imaging direction (e.g., the fixation holes must appear circular and the projection of the nail shaft must pass through the center of the X-ray image). Even when the necessary rotations and translations based on the positioning of an object are accurately determined, it may be important to determine user instructions suitable for repositioning the C-arm. As shown in FIG. 16, the C-arm has multiple rotation and translation axes. Further, it is also possible to move the C-arm to different positions by its wheels in the operating room. This allows translation parallel to the floor and rotation around an axis parallel to the vertical axis, and it typically has a large distance (more than 1 m) from the vertical axis.

[0130] Many available options for moving the C-arm make it difficult to determine which option is best for the user to reach the desired position (for the desired imaging direction) most quickly or with the least effort. Further, there are also constraints arising from the operating room setup that prevent the user from moving the C-arm to a specific position. Thus, the user may, in some cases, especially in surgeries involving the upper limbs, choose to move the patient (or the operating table) instead of the C-arm.

[0131] A method may be proposed to (i) determine the necessary information regarding how to reposition the C-arm and / or the patient, (ii) translate this information into guidance for the user to select from the available means of moving the C-arm (by translation along the axis of the C-arm or rotation around it, or by moving the C-arm by its wheels) or moving the patient, and (iii) determine the amount of movement required for each case.

[0132] In other words, a method for assisting in adjusting the imaging direction of a C-arm-based imaging device may include a step of receiving information on the current imaging direction, a step of receiving information on the target imaging direction, a step of determining a first one from among a plurality of means for rotation and translation of the X-ray source and the detector and the amount of movement for that first means, and a step of achieving the imaging direction closest to the target imaging direction. Naturally, such a method may be implemented as a software program product that causes the system to perform this method.

[0133] These methods may take into account possible constraints imposed by the construction of the C-arm and the operating room setup, select those movements that are easiest for the user to perform, and require the fewest number of movements.

[0134] These methods may determine the current imaging direction based on an X-ray image. For example, determine the viewing direction or imaging direction based on anatomical structures (e.g., using the patent application filed by Blau on August 23, 2018), and / or optionally take into account a priori geometric information regarding implants, devices, or anatomical structures, and further optionally take into account the 3D position and 3D orientation of the target structure in a coordinate system spanning the image plane, and further optionally take into account the typical operating room setup for the current surgical procedure (the position of the patient on the operating table, the position of the C-arm relative to the operating table, the position of any object that can prevent the C-arm from moving to a specific position relative to the operating table / patient), and further optionally take into account the typical rotation axis of the C-arm to localize the target structure (or object).

[0135] For example, for a femoral nailing procedure, it may be assumed that the typical position of the patient relative to the C-arm, i.e., the supine patient, the known treatment side, and the image receiver between the patient's legs, is known. Additionally or alternatively, the user may select an operating room setup from a provided set of options. These may be based on, for example, a scan of the implant package, portions of a prior surgery learned by the system, such as patient positioning on the operating table, where the implant was used, and / or where the C-arm machine was positioned relative to the operating table (e.g., between the patient's legs).

[0136] Translating the calculated deviation from the desired position / orientation into instructions for adjusting the C-arm can be important as these instructions are likely to be as easy to execute as possible. Moving the C-arm by its wheels is less accurate and may be more difficult to do in an operating room setting, so instructions that do not require moving the C-arm by its wheels may be preferred. In general, it may be preferred to keep the number of instructions low. A neural net may assist with this entire procedure.

[0137] For example, if a large rotation around the vertical axis is required, this must be done by moving the entire C-arm by the wheel, because this may enable isocentric rotation around the target structure (keeping the target structure near the center of the C-arm beam). If the required rotation is about an axis parallel to the vertical axis of the C-arm and the rotation is relatively small, the vertical axis of the C-arm must be used. When the desired rotation axis is away from the vertical rotation axis of the C-arm, such rotation includes a relatively large translation component, and it must be borne in mind that any such translation can be compensated for when determining any potentially required translation. As described above, a rough determination of the imaging depth may be possible (up to a few centimeters). Since the distance of the C-axis from the target structure is roughly known, this may be sufficient, for example, to calculate the translation in the AP direction resulting from rotation around the C-axis. Also, the offset between the C-axis and the central x-ray beam may be calculated from the rotation around the C-axis. Furthermore, due to the fact that fluoroscopic projection is applied, any translation of the C-arm automatically includes a rotational component.

[0138] Making a rough assumption about the 3D position and 3D orientation of the axis of the C-arm with respect to the target object / structure may be sufficient for the first iteration step of the positioning procedure, whereby a position close enough to the desired position can sometimes be obtained. For higher accuracy, or to reach a sufficiently accurate position with the fewest number of steps, the system may use the information collected from the previous iteration step in subsequent steps, thereby making it possible to more accurately determine the 3D position and 3D orientation of the axis with respect to the target object / structure. As an example, if translation along two or more C-arm axes or rotation around them is required to reach the desired position of the C-arm, the system may instruct the user to move or rotate the C-arm around one axis (e.g., rotate around the C-axis). This allows approaching the desired position and at the same time determining the 3D position and 3D orientation of the C-axis with respect to the target object, and also determining the offset between the C-axis and the center of "C" of the C-arm.

[0139] Alternatively, depending on the available a priori information, the 3D position and 3D orientation of the axis of the C-arm with respect to the target object / structure may be determined by (i) moving the C-arm along two axes perpendicular to each other (if these axes do not intersect, one of the parallel axes of these axes must be perpendicular to the other axis), or (ii) rotating the C-arm along two axes perpendicular to each other (if these axes do not intersect, one of the parallel axes of these axes must be perpendicular to the other axis), or (iii) moving the C-arm along one axis and then rotating the C-arm around another axis parallel to the moving axis (e.g., forward translation combined with rotation around the vertical axis, or rotation around the C-axis combined with rotation around the propeller axis, or proximal translation combined with backward translation).

[0140] As an example, for the distal lateral fixation procedure of a femoral antegrade nail (with the C-arm positioned between the patient's legs and the patient positioned supine), a translation of the "C" in the anterior direction, on the one hand, causes a rotation of the viewing direction around the nail axis, which may enable a more accurate calculation of the 3D position of the target object relative to the C-axis in order to more accurately determine the imaging depth. On the other hand, the system may determine the rotation of the nail around its axis relative to the operating room floor. At a later time, when calculating guidance instructions regarding translation in the anterior / posterior direction, the system may calculate the required translation by using the relationship of simple trigonometry, taking into account the effect of the rotation of the nail around its axis relative to the operating room floor determined above for the required translation along the vertical axis. This information may also be utilized when calculating other movement instructions provided by the system. For example, when calculating the required rotation around the vertical axis, the distance between the object and the vertical axis may be more accurately determined (since the distance of the axis perpendicular to the C-axis can be more accurately determined). Therefore, since the distance between the central X-ray beam and the C-axis has already been determined, the effect of the translation in the proximal / distal direction caused by the rotation around the vertical axis can be more accurately taken into account.

[0141] In practice, in the first iteration step, it may be sufficient to simply determine one or two means of movement or rotation to approach the final desired position. By observing these one or two movements / rotations, the system can obtain sufficient information regarding the 3D position and 3D orientation of the axis with respect to the target object / structure in the second iteration step so that it can determine with sufficient accuracy all the remaining necessary steps to reach the final desired position and provide them to the user. As described below in the section "Example of a potential processing workflow for distal fixation procedures", the system does not require very precise positioning of the C-arm. For example, a perfectly circular projection of a circular hole may not be required.

[0142] If the user moves the patient rather than the C-arm in response to a system instruction, it may help to detect such movement by checking the image background and prevent an inaccurate determination of the movement / rotation axis. The "image background" mentioned above may be any object that is not completely radiolucent and moves with the patient (e.g., part of the operating table). By using simple image difference analysis, it may be possible to clarify whether the C-arm has been moved. In this context, it may be required that there is no rotation of the digital image in the C-arm between C-arm movement and image acquisition.

[0143] Based on the image of the target structure and the initial positioning of the target structure within the viewing direction, the system may determine which movement to initiate and / or proceed with.

[0144] Instructions may be given on how to rotate the C-arm around the C-axis and how to move the C-arm up and down. If the target structure is not positioned at the center "C" of the C-arm, translational instructions may be given on how to restore the previous position in the X-ray image or how to reach the desired position. Instructions may be given on how to rotate the C-arm around its vertical axis. This may take into account the typical geometry of the C-arm (the distance between the vertical axis and the center "C" of the C-arm), and the system may learn the geometry of a particular C-arm from previous rotations. If the target structure does not appear at the desired position in the X-ray image or no longer appears at the desired position after rotation, instructions may be given on how to translate the C-arm. If the position of the target structure in the X-ray image is already correct and remains correct after rotation (e.g., when moving the C-arm by its wheels such that the target structure remains at the center "C" of the C-arm), no translational instruction is given.

[0145] This procedure may similarly apply to other axes of the C-arm, such as the propeller axis.

[0146] Regarding the optimized adjustment of the C-arm, the following general aspects can be summarized.

[0147] Initially, it is intended to use only one means for translation or rotation of the C-arm device in order to adjust the position and orientation in the imaging direction as close as possible, or at least with sufficient accuracy in the target imaging direction, i.e., the optimal direction.

[0148] If necessary, it may be proposed to use a second means to further adjust the imaging direction. Further accuracy may be enhanced by utilizing additional means.

[0149] The current imaging direction, i.e., the starting point for adjustment, may be determined based on the localization of an object or structure in an X-ray image generated using the current imaging direction. Naturally, the object or structure may be a sub-structure and may be an anatomical structure, an implant or a device / instrument, or a combination of such objects.

[0150] The target imaging direction may be identified based on those known geometric aspects, 3D position and 3D orientation. Such geometric aspects may be known from a preoperative plan and such geometric aspects are available from a database.

[0151] Furthermore, the position and orientation of the object may generally be known with respect to the C-arc of the C-arm based imaging device. In a typical operating room setting, anatomical structures may be positioned with respect to the C-arm device in a known manner that enables prediction of the imaging direction. The system may in some cases provide information for confirmation to the user.

[0152] Alternatively or additionally, the system may learn how users tend to utilize translations and rotations of the C-arm and may take that into account when providing further instructions to move the C-arm. For example, the system may calculate the axis of rotation or translation of the C-arm device relative to an object from two images, where the imaging direction is rotated or translated around the axis of rotation during image generation. In particular, the system may learn whether the user tends to move the C-arm device by approximately the amount that is being instructed at that time and may take that into account when providing further instructions to move the C-arm device.

[0153] Example of a potential treatment workflow for distal fixation The following distal fixation procedures are described for long antegrade nails. Nevertheless, it may also be applicable to retrograde nails that are fixed proximally. The following fixation procedures are provided for holes whose axis is approximately in the ML direction, but may also be applicable to holes whose axis is in a different direction, such as the AP direction.

[0154] For such a procedure, it may be assumed that a complete 3D model for the target object (i.e., the nail) is available. Nevertheless, the procedure may still work well if only an incomplete or partial 3D model is available, for example, if only approximate information about the shape of the nail (e.g., a cylindrical object whose diameter decreases slightly towards the tip, including a cylindrical fixation hole) is known.

[0155] Hereinafter, the fixation hole is referred to as the "target structure". In principle, it is sufficient to know only the relative 3D position and 3D orientation between the instrument (e.g., a drill, a reamer, a sleeve, or even an implant such as a screw) and the target structure. Therefore, both the target object (the nail) and the target structure (the fixation hole) are considered hereinafter. In this description, it is assumed that a circular hole is fixed first.

[0156] 1. The user obtains an X-ray image of the nail in the approximate ML direction.

[0157] 2. The system may determine the imaging direction of the target structure (e.g., by detecting or localizing the target object (in 2D)). The system may search for or determine a target trajectory (or optionally a target plane) with respect to the target object or structure. The system may then determine how to adjust the C-arm to reach the desired imaging direction of the target trajectory and inform the user. In many cases, the desired imaging direction is aligned with the target trajectory, in which case the distal fixation holes are depicted as circles. FIG. 20 is an X-ray image in which a nail labeled 20.N is visible with a non-circular fixation hole labeled 20.H. Thus, the imaging direction is not the intended ML imaging direction. FIG. 21 is an X-ray image in which a nail labeled 21.N is visible from the true ML imaging direction, as is apparent from the circular fixation hole labeled 21.H.

[0158] It is desirable for the nail axis to pass through the image center and for the fixation holes to be near the image center. This is desirable when there are additional holes to be fixed without readjusting the C-arm. Thus, ideally, these holes should be on the central X-ray beam. The C-arm adjustment may be repeated and completed with a new X-ray image that meets the requirements.

[0159] 3. At this point, the system may highlight the center of the fixation hole at which the instrument must be aimed in the X-ray image. This highlighted (target) point is on the target trajectory and is the center of the circle in the described scenario. The system then detects the tip of the instrument in 2D and calculates the required movement of the tip to reach the target point. The system may support the user in an iterative process (each iteration consisting of acquiring a new X-ray image and repositioning the instrument) to reach the target point. FIG. 22 shows an X-ray image of a nail (22.N) and a drill (22.D) with an inaccurately positioned drill tip. FIG. 23 shows an X-ray image of a nail (23.N) and a drill (23.D) with an accurately positioned drill tip.

[0160] 4. If the tip of the instrument (here the female) is located at the point highlighted in the X-ray image, the surgeon may make an incision and insert the drill (optionally together with a soft tissue protection sleeve), and repeat step 3. Then the user may decide (as in the conventional procedure) to align the drill with the target trajectory without moving the tip of the drill.

[0161] 5. Rotate the C-arm, for example, about the C-axis by, for example, 25° and acquire a new X-ray image. The system may re-localize the target object (or optionally only the target structure). Based on the a priori knowledge that the tip of the instrument is on the target trajectory (remaining at a known 3D position and 3D orientation with respect to the target object / structure), the relative 3D position and 3D orientation between the drill and the target object / structure may be determined. Even if the tip of the drill in the distal-proximal direction is no longer exactly on the target trajectory, the system can calculate the corresponding deviation. This may be sufficient because it only has the a priori information that when the C-arm is rotated about the C-axis, the tip of the drill is located in the plane straddling the target trajectory and the nail axis.

[0162] At this point, the system may calculate the deviation from the target trajectory and thus communicate it to the user, for example, by displaying the required angular corrections in the proximal-distal and anterior-posterior directions. If necessary, the system may also instruct the surgeon on how to adjust the tip position of the instrument in the proximal-distal direction. Furthermore, the system may calculate the penetration depth, in this case, for example, the distance between the drill tip and the nail, and thus communicate it to the user. This step 5 may be repeated by acquiring a new X-ray image, providing information / instructions to the user, and readjusting the instrument.

[0163] Communicating to the user may be done on a display and / or audibly. The advantage of auditory information may be that the surgeon does not need to look away from the drill and can thus achieve the correct direction for drilling with fewer repetitions.

[0164] 6. To adjust the drilling direction in the drill and / or to obtain information regarding how much further to drill, step 5 may also be performed during drilling (this is independent of the fact that in a typical distal fixation situation, drilling is continued to the next cortex after hitting the nail hole).

[0165] Fixation into additional holes (e.g., oval holes):

[0166] To save time and reduce X-ray exposure, the procedure for fixation into additional holes (hereinafter assumed to be oval holes) after fixation into the first hole as described above is shown below.

[0167] Assuming that the target trajectory for the oval hole is in the same plane as the first hole, rotate the C-arm back to its original position (where it was before performing step 5) showing the first hole (with or without the screw) as a circle. Thus, the nail axis again passes through the center of the X-ray image and the oval hole is close to the image center. No realignment of the C-arm is necessary unless correction of the rotation around the C-axis is required because the original angle has not been reached accurately enough. Thus, the oval hole appears with its maximum diameter in the AP direction but is compressed in the perpendicular direction. On the other hand, if the target trajectory for the oval hole is not in the same plane as the first hole, the required realignment of the C-arm may be supported by the system as described above.

[0168] Since the system knows the approximate distance between the bone surface and the nail at the inner-outer position from fixation into the first hole, this value (and optionally the statistical model of the bone) may be used to correct the target position of the drill tip (see step 3 above). Thus, the target point in the 2D X-ray image does not appear at the center but is shifted in the distal-proximal direction to hit the oval hole centered (in both the AP direction and the distal-proximal direction).

[0169] In FIG. 17, the circular nail hole represented by 17.RH is perfectly circular in 2D, i.e., in the X-ray. The tip of the opening instrument (represented by 17.OT) has a specific distance from the center of the oval nail hole due to its position on the bone surface, so it is not at the 2D center of the oval nail hole, but its tip is placed exactly on the central orbit of the oval nail hole in 3D. The two black arrows represented by 17.C1 and 17.C2 indicate chamfers, which appear in different sizes on both sides of the oval nail hole in the perspective view.

[0170] After the instrument is (approximately) aligned with the target trajectory, the C-arm is rotated around the C-axis, and another X-ray image is acquired, any potential inaccuracies in the distal-proximal position of the tip of the instrument are detected and calculated, so any potential inaccuracies in the distal-proximal direction cannot be a problem. Then, if necessary, an instruction may be given to correct the tip position of the instrument in the distal-proximal direction. As discussed above in step 5, it may be sufficient for the tip of the instrument to be simply in the plane straddling the target trajectory and the nail axis. The remainder of the procedure follows the steps for the first hole.

[0171] If the oval hole is tilted with respect to the nail axis, the tilt may be compensated for when rotating the C-axis by potential fine-tuning supported by the system.

[0172] The whole consideration also applies to circular holes. Furthermore, it is also possible to fix additional holes following the same procedure.

[0173] As discussed above, in order to position the tip of the drill on the target trajectory, the viewing direction into the hole need not be perfect (i.e., in the case of a circular hole, having, for example, a maximum projected hole width and height that is perfectly circular). Depending on how strict the available a priori information and requirements for angle determination are (the less strict these are, the smaller the distance drilled between the bone surface and the nail will be), the viewing direction into the hole may deviate more or less from the target trajectory. For example, when an AP image is acquired, the distance between the outer bone surface and the nail along the target trajectory may be approximately determined (alternatively, simply estimated). Also based on this information and the outer X-ray image, the point in the 2D X-ray image at which the tip of the drill must be positioned to be on the target trajectory may be calculated (and then displayed) based on an oblique viewing angle, as discussed above. This point need not be exactly on the drill trajectory. Rather, a new target trajectory may be calculated using the deviation (in 2D spatial coordinates) from this point as determined in the 2D X-ray image based on the estimated or previously determined distance between the bone surface and the nail along the fixed trajectory. The new target trajectory may then be used in the next image to orient the drill. Since the tip of the drill need not be perfectly centered, this may further ease the surgeon's task of positioning the tip of the drill at the correct point. Furthermore, it may allow for greater accuracy when orienting the drill to hit the target hole.

[0174] Assuming that the position of a point of a second object relative to the geometric aspects of a first object is known with sufficient accuracy, 3D reconstruction and / or determination of relative 3D position and 3D orientation may be possible by image registration.

[0175] Deviation from the original target trajectory may also be resolved by the following method. In the first image, the device may be placed on an arbitrary point (which may or may not be on the target object). In the second image (obtained from a different viewing direction), the inclination of the device may change (e.g., by aiming at the target trajectory), but the tip of the device remains in the same position. Based on the location identification of the target object / structure, the two images may be aligned, which may enable the determination of the 3D position of the point relative to the target object. This point may then be used to determine the relative 3D position and 3D orientation between the instrument and the target object / structure with sufficient accuracy.

[0176] In step 5 or 6 above, if the tip of the instrument is blocked by the target object (e.g., a steel nail), and thus the location identification is not sufficiently accurate, the system may give an instruction on how to make the tip of the instrument visible to the system. This may be advanced by a system that calculates and indicates how the C-arm needs to be repositioned (e.g., rotation around the C-axis instead of rotation around an axis perpendicular to the nail axis). Also, if the material of the instrument absorbs significantly more X-rays than the target object (e.g., a steel instrument, a titanium nail), this may be achieved, for example, by increasing the voltage and / or current, or by selecting different C-arm program settings.

[0177] Additionally or alternatively, the system may match the statistical 3D model of the bone, and thus determine the 3D position of the nail relative to the 3D position of the bone, and in this way enable the determination of the required fixation screw length in 3D.

[0178] In the case of a typical steel nail in which all the fixing holes face the same direction, the fixing holes may not be visible in the X-ray image for rotations exceeding 30 to 35° (similar to step 5), which means that the steel nail may not be positionable for rotations greater than 30 to 35°. On the other hand, for example, titanium instruments or implants absorb considerably less radiation than, for example, steel instruments or implants. Thus, in the case of titanium nails, an inclined hole creates a density gradient at the hole boundary. This is shown in FIG. 18 for a 25° rotation away from the fixation plane and in FIG. 19 for a 45° rotation away from the fixation plane. This effect means that it may be possible to position titanium nails tilted over a much larger angular range compared to steel nails. Another beneficial effect of titanium nails is that it may be possible to make visible a drill, which is typically made of steel, relative to the nail. This may increase the accuracy in positioning the drill, for example, when the drill tip is close to the nail during drilling. It may also be possible to rotate the C-arm around a different axis, for example, the propeller axis, where typically the X-ray shows the tip superimposed on the nail.

[0179] When the C-arm is rotated around a vertical axis instead of the C-axis in step 5, the system requires that the tip of the drill be in a plane whose normal is the nail axis and which contains the target trajectory. In this case, the deviation of the drill tip from the target trajectory in the AP direction may be calculated.

[0180] Therefore, an alternative method of the above workflow may be to acquire X-ray images from both the viewing directions obtained by rotating around the C-axis so as to move away from the fixation plane and the viewing directions obtained by rotating around the vertical axis. In this case, a priori information regarding the position of the drill tip relative to the target trajectory is not required. Therefore, it is not necessary to position the C-arm in the true ML direction.

[0181] Example of a potential processing workflow for the placement of sacroiliac (SI) or pedicle screws The target object and the target structure may also be anatomical. An example of an anatomical target structure is the pedicle. Considering the 3D reconstruction of the instrument and the relative 3D position and 3D orientation, it may be sufficient to achieve the accuracy required for the target structure and thus the target trajectory.

[0182] The procedure is similar to a distal fixation procedure except for locating the anatomical target structure, which may be carried out with or without using a deterministic 3D model. The deterministic 3D model may be obtained either preoperatively (e.g., preoperative CT scan) or intraoperatively (e.g., intraoperative CT scan or O-arm). If the deterministic 3D model is not available, a statistical 3D model (e.g., statistical shape or appearance model) may be used for 3D reconstruction as previously discussed in the section "3D Reconstruction and Localization of Anatomical Objects Based on a Single X-ray Image".

[0183] In this procedure, the instrument is not fixed to the target point of the anatomical structure but is held by hand and then approximately aligned in the target trajectory. Without acquiring a new X-ray image, the C-arm may be rotated, for example, 25° around the C-axis. After such a repeated process as described above, a new X-ray image is acquired, and the system may calculate the relative 3D position and 3D orientation between the instrument and the target structure / object, taking into account that the tip of the instrument is on the target trajectory. It is assumed that the viewing angle to the instrument is in a range that enables a sufficiently accurate determination of the 3D position and 3D orientation between both objects. If only a statistical model of the anatomical structure is available, this step includes the 3D reconstruction of the target structure / object.

[0184] In the next step, the system may calculate the deviation between the instrument axis and the target trajectory with respect to angular deviation, and optionally the deviation at the tip position in a direction parallel to the rotation axis of the C-arm used for rotation between both images. The system may also calculate the penetration depth of the drill. The system may provide this data (e.g., in the form of two angular values, the translations required for the tip position of the instrument and the remaining insertion depth) to the user. The user may then adjust the instrument accordingly, acquire new X-rays, and / or drill a hole in the instrument or insert it. Since the a priori information and side constraints remain unchanged, the procedure will also work well if drilling has already been started and the instrument has already penetrated the anatomical structure.

[0185] Example of a potential processing workflow for determining the 3D position and 3D orientation between the device and the anatomical structure based on image alignment to improve accuracy By adjusting the bearing of the C-arm (translation and rotation) to align with a specific part of the anatomical structure (e.g., a narrow path such as the pedicle), the tip of the instrument (e.g., a drill, k-wire or jamshidi needle or even an implant such as a screw) may be placed on a specific anatomical reference point. This step may be supported by the system by displaying the reference point in the acquired 2D X-ray image, or alternatively, the identification of the reference point by the surgeon may be used by the system to improve its accuracy. The C-arm is then rotated, for example, 20 - 30° around the C-axis (or a comparable rotation around the propeller axis) while leaving the instrument in place, and then another X-ray image is acquired. The fact that the instrument contacts the surface of the anatomical object at the reference point may be used to reduce or even eliminate the ambiguity caused by the ambiguous positioning of the instrument. The movement of the C-arm relative to the previous image may then be determined, and thus the viewing direction to the anatomical object may be determined with high accuracy. Since this requires that the instrument does not move between X-ray images, it may be easier to fix the instrument to the anatomical structure, which can be done not only with a drill but also with a jamshidi needle or k-wire.

[0186] Procedure: 1. Based on the preoperative CT scan, reference points and reference trajectories (i.e., the intended drilling or insertion trajectories with the drill) as well as the target endpoints may be planned before the surgery. This may include planning along the intended imaging direction of the C-arm, such as true lateral or true AP, or along the pedicle or other easily recognizable landmarks. Also, this step 1 may be performed automatically by this system and / or interactively with the user during the surgery (using the intraoperative 3D imaging device).

[0187] 2. During the surgery, improvement in the accuracy of positioning may be achieved by using either the previously defined or on-line calculated C-arm imaging direction. This system may assist the user in achieving the required C-arm bearing by detecting the relative positions of specific anatomical features such as edges or points (see, for example, the patent application by Blau filed on August 23, 2018). This system may display the reference points in the X-ray image based on matching the structure or the whole object from the CT scan to the X-ray image. Then the surgeon positions the tip of the instrument (e.g., a drill, a Jamshidi needle or a k-wire or even an implant such as a screw) on this reference point, which may also be supported by this system by detecting the tip of the instrument in the 2D X-ray image. Hold the instrument intentionally at an angle such that the instrument (or the power tool) and the surgeon's hand do not block the field of view if necessary.

[0188] 3. Then align the instrument in the approximately desired direction.

[0189] 4. If possible, fix the instrument to the anatomical structure using the defined markings on the instrument so that the exact depth of penetration can be determined.

[0190] 5.Obtain another X-ray image from the same imaging direction. At this point, the position of the tip of the instrument is restricted because it is approximately located on the drilling or insertion trajectory. If the penetration depth of the instrument is accurately known, there will be less ambiguity than when the penetration depth of the instrument is not known. The system may check whether the anatomical structures shown in the X-ray image (e.g., by image difference analysis) remain unchanged. Alternatively, in this step, it may be sufficient for the user to simply indicate in which plane the tip of the instrument is (with respect to the anatomical structure). If step 5 cannot be performed because the instrument or the surgeon's hand blocks the field of view, steps 4 and 5 are performed without aligning the instrument in step 3.

[0191] 6.Move or rotate the C-arm to different positions. Based on the localized and fixed instrument, the movement of the C-arm relative to the previous image may be determined. Together with the localized anatomical structure, this enables the determination of the relative 3D position and 3D orientation between the instrument and the anatomical structure. The determined quantities may be optimized together.

[0192] 7.If the instrument has such a small diameter that it can be localized with sufficient accuracy only for a specific angle, the instrument must be visualized at a suitable angle (e.g., within the range of 10 - 55°) in all the acquired X-ray images in which it is aligned. Furthermore, the anatomical structure may be more accurately localized for a specific imaging direction. Therefore, the accuracy for determining the relative 3D position and 3D orientation in step 7 may be increased by selecting a specific imaging direction for the anatomical structure. However, since such special imaging directions are typically true AP and true ML, this means that the angle between two X-ray images is close to 90°. Additionally, when the instrument is fixed in step 4, it must already be ensured that the tip of the instrument is visualized at a suitable angle (e.g., within the range of 10 - 55°) in all the acquired and aligned X-ray images. Therefore, in such a case, the suitable angle for instrument fixation is the intermediate between the true AP visualization direction and the true ML visualization direction, i.e., approximately 45°.

[0193] 8. If an X-ray image is acquired and the above conditions are satisfied, the system may calculate the deviation between the axis of the instrument and the reference trajectory, and provide this to the user (for example, by displaying two angular values). The user may then retract the instrument to the original reference point and realign it with the reference trajectory. The system may assist the user in finding the original reference point.

[0194] Reaching the correct reference trajectory may require a repeated process of acquiring additional X-ray images. After acquiring a new X-ray image, the system may then check (for example, by image difference analysis) whether the anatomical structure is still shown in the X-ray image in the same orientation and position. In that case, the relative 3D position and 3D orientation between the instrument and the anatomical structure may be recalculated based on the a priori information that the tip of the instrument is on the reference trajectory. Furthermore, the system may calculate the insertion depth, in this case for example the distance between the tip of the instrument and the target endpoint, and thus communicate it to the user. During the insertion of the instrument, additional X-ray images may be acquired and the above steps may be repeated.

[0195] Also, if it is not feasible or not desired to ensure that the instrument is visible within a suitable angular range (for example, 10 to 55°) in all acquired X-rays, a power tool that holds the instrument (for example, a power drill that holds a drill) may be taken out (which is obvious when using a k-wire or a Jamshidi needle). If the entire instrument (tip and bottom end) is visible in the X-ray image, the length of the projection of the instrument in the X-ray image may be determined, thus enabling the instrument to be located with sufficient accuracy. In such a case, it may even be possible to view the instrument at an angle close to 90°. This allows the instrument to be initially fixed at an approximately correct angle (eliminating the need to initially fix the instrument at an intentionally inaccurate angle, see step 7), thus reducing the number of repetitions required in step 8.

[0196] For the insertion of pedicle screws, it may be possible to identify the entry point of the pedicle in an AP view in which the pedicle axis is tilted, for example, 10 to 45° with respect to the viewing direction. In such an imaging direction, the drilling machine does not block the field of view, positions the tip of the device on the entry point of the anatomical structure, and after aligning the angle of the instrument axis with the target trajectory, there is no need to acquire a second image from another direction. Since this procedure often requires a k-wire or Jamshidi needle, the instrument may be fixed to the bone with its axis already aligned with the desired target trajectory.

[0197] If necessary, the user may then acquire one or more X-ray images from other viewing angles, and the system may use this to perform the image alignment described above to improve accuracy. Further, if necessary, the drilling angle may be further optimized based on additional information (optionally without retracting the drill), and drilling may proceed.

[0198] Due to the fact that in this first AP view the other pedicle has an inclination that is reversed left-right with respect to the first pedicle (see FIG. 15 showing two Jamshidi needles 15.JN and 15.JN2), it is possible to repeat the above procedure for the other pedicle of the same vertebra to enable more robust alignment of the images, and (by using the projection in its X-ray image) to utilize the Jamshidi needle already inserted in the first pedicle.

[0199] Flowcharts of FIGS. 24 to 26 FIG. 24 shows a general flowchart covering all of the procedures shown in the sections of the above “Example of a potential processing workflow for distal fixation procedures,” “Example of a potential processing workflow for sacroiliac joint (SI) or pedicle screw placement,” and “Example of a potential processing workflow for determining the 3D position and 3D orientation between a device and an anatomical structure based on image registration to enhance accuracy.” There are two possible executions, namely the rapid execution shown in FIG. 25 applicable to the procedures shown in the sections of the above “Example of a potential processing workflow for distal fixation procedures” and “Example of a potential processing workflow for sacroiliac joint (SI) or pedicle screw placement,” and an execution for enhancing accuracy shown in FIG. 26 applicable to the procedures shown in the section of the above “Example of a potential processing workflow for determining the 3D position and 3D orientation between a device and an anatomical structure based on image registration to enhance accuracy.”

[0200] Those skilled in the art will understand that not all steps have to be performed and that additional sub-steps not mentioned may be performed depending on the specific use situation of the teachings provided herein.

[0201] The steps of FIGS. 24, 25, and 26 are as follows.

[0202] S10: Generate and load a 3D model. S11: Preoperative planning (optional). S12: Load the entire 3D model. S13: Intraoperative automatic determination of one or more target trajectories / one or more planes and, where applicable, target points (e.g., in the case of anatomical structures).

[0203] S20: Support C-arm adjustment. S21: Acquire X-ray images. S22: Provide, for example, the rotation angle (including direction) around the C-axis, rotation around the propeller axis, etc., to support reaching a special viewing direction to a target object, such as a circular hole (potentially supported by nail positioning), or a true AP / ML view to an anatomical structure (potentially supported by DNN). S23: If ambiguity occurs, the system provides only the value of the rotation angle without including the direction. S24: If the viewing direction is not close enough to the desired viewing direction, the user follows the system instructions and continues with S21. If a corresponding 3D model of the anatomical structure (e.g., CT-scan, i.e., deterministic) is available, in this case, it is possible to obtain the target trajectory from the current viewing direction and the 3D model, and since it is known that the tip of the access device is placed on the anatomical structure, the desired viewing direction may be different from the target trajectory. Example: Distal fixation where the access device is placed on the femur and a 3D model of the femur is available. After locating the target object (S33), the system calculates the intersection of the target trajectory of the nail model and the surface of the femur model. Using the 3D model of the anatomical structure, the system gives adjustment instructions for the tip of the access device (S37).

[0204] S30: Support access device positioning. S31: Positioning of the access device. S32: Acquire an X-ray image. S33: Locate the target object / structure. S34: If the target trajectory is sufficiently aligned with the viewing direction, the target point is directly visible in the 2D X-ray image (one DoF for the undefined tip position of the device). Proceed to S36. S35: The system displays the intersection of the 3D surface of the anatomical structure (including the distal fixation part) and the target trajectory superimposed on the 2D X-ray image (all DoF for the defined tip position of the device). S36: 2D alignment of the access device. S37: The system gives instructions to the user to support tip adjustment of the access device. S38: If the position is not reached with sufficient accuracy, the user follows the system instructions and continues with S32.

[0205] S40: Determine the 3D position and orientation between the access device and the target object in order to align the access device with the target trajectory. S41e: Fixing of the required access device. S411e: Fixing of the access device. If it is intended to apply S44, the system provides support for fixing the access device to the target object at an angle that ensures that the angle between the access device and all special viewing directions is less than 65°. Two angular values are provided. If the user acquires another image from the same viewing direction, the system checks the fixing angle of the access device. S412e: Acquire an X-ray image that does not include changes in the relative position between the C-arm and the anatomical structure. S413e: Image difference analysis to determine the penetration depth of the access device. S414e: Determine the 3D position and 3D orientation of the access device relative to the anatomical structure. S41q: Do not fix the access device. Do not support the user in aiming at the trajectory. S42: The system calculates and displays the adjustment value for C-arm rotation to reach an angle of 25° between the viewing direction and the target trajectory. If ambiguity occurs, the system provides only the value of the rotation angle without the direction. S43: The user positions the C-arm according to the displayed adjustment value and acquires an X-ray image. If the viewing direction is not close enough to the desired viewing direction from S42, proceed to S42. S44: Calculate the 3D position and 3D orientation between the target object and the access device for the final access device adjustment instruction. S441e: In the case of iterative optimization, calculation of the 3D position identification of the access device and the transformation matrix between the 3D position and 3D orientation of the access device between the current image and the previous special viewing direction. S442e: (i) the above transformation matrix (S441e), (ii) all previous 3D orientations and 3D positions of the anatomical structure together with the current position of the oral appliance, and (iii) a potentially improved 3D orientation and 3D position of the anatomical structure based on the 3D orientation and 3D position of the current anatomical structure (either iterative or co-optimization). S441q: For iterative optimization, target object localization. Based on the target object localization and the a priori information that the tip of the oral appliance is positioned on the target trajectory / plane, determine the 3D position and 3D orientation between the oral appliance and the target object. Proceed to 443. S442q: Co-optimization of the 3D orientation and 3D position of the target object with respect to the oral appliance. S443: If, for example, the distal-proximal deviation of the nail can be confirmed and corrected, confirm and correct the a priori information (the tip position of the oral appliance with respect to the target object). S444e: If the accuracy of the 3D position and 3D orientation between the anatomical structure and the oral appliance requires further improvement, the system calculates and displays adjustment values for C-arm rotation to reach a further special viewing direction. S445e: The user positions the C-arm according to the displayed adjustment values and acquires an X-ray image. If the viewing direction is not close enough to the desired viewing direction, proceed to S44. S45: The user moves the oral appliance by the adjustment values provided to align the oral appliance with the target trajectory. S451: Since the 3D model of the target object provides the target trajectory, the system provides the angles (two angles including the direction) for adjusting the direction of the oral appliance to align the oral appliance with the target trajectory, obtained from the 3D position and 3D orientation of the appliance determined above with respect to the target object. S452e: If the oral appliance is still fixed in the first position (S41e), the user retracts the oral appliance until its tip is on the target trajectory, then aligns the oral appliance based on the system output and acquires an X-ray image (alternatively, aim at the target trajectory using a second oral appliance. In this case, proceed to S44). S453e: The system compares images (e.g., by image difference analysis). If the images are locally close enough (with respect to the target object), proceed to S44. S454: If the alignment between the access device and the target trajectory is not close enough, the user positions the access device based on the system output, acquires an X-ray image, and continues with S44. S455: If the remaining alignment indication provides a sufficiently small value, the system displays information on how far away from the drill the user is. The user may align the access device based on the alignment indication and decide whether to acquire another X-ray image. S46: Drilling with the drill. S461: The user drills a hole with the drill. S462: Whenever the user desires to check the drill direction or drill depth, the user acquires a new X-ray image and continues with S44.

[0206] In addition, if the system can provide instructions regarding how it adjusts the orientation (and possibly the position) of the device already present in the first X-ray image, these can be used to reach a temporary alignment between the device and the target trajectory. After this alignment, if the inclination of the device is already within the required angular range and neither the power tool nor the surgeon's hand blocks the view, another image from a different imaging direction may not be required, and if necessary, the confirmation of the instructions (regarding the orientation and possibly the position of the device) applied by the X-ray image from the same imaging direction may be performed. If (i) no correction is needed or only a very small correction is needed, or (ii) a device that guarantees a sufficiently accurate application of the given instructions is used, another X-ray image may not be required. Such a device may be a manual device or a robot.

[0207] As described in the section "Method for Positioning a Target Object / Structure in the Field of View of the C-Arm from a Desired Visualization Direction", particularly when the instrument is placed on the target object and the target point is identifiable in the current X-ray, perfect alignment of the C-arm in the direction of the target trajectory may not be required. It may be possible that all the information necessary to calculate the relative 3D position and 3D orientation in the first X-ray image is already available. Thus, depending on the setup, the first X-ray image may be sufficient to perform the entire distal fixation procedure. An example of such a setup is the adoption of a robot that holds the instrument at a given point already taking into account the required inclination. If it is possible to identify both the target trajectory and the required starting point in the first acquired X-ray image, and to determine the 3D position and 3D orientation between the instrument and the target object, the robot may translate and rotate the instrument as required and drill a hole. In a general case, if the target trajectory and the resulting starting point for drilling a hole cannot be identified based on the first X-ray, another X-ray may be acquired from a different suitable visualization direction. The two X-ray images may be aligned based on the localization of the target object, which may enable the calculation of the 3D position and 3D orientation of the instrument with respect to the target object and thus also with respect to the target trajectory. Thus, the entire repositioning (translation and rotation) including drilling a hole may be performed by the robot. This procedure is not limited to drilling.

[0208] Reduction support When anatomically reducing a proximal femoral fracture, it can happen that, even though the reduction appears correct in both the AP X-ray image (e.g., the Adam's arch appears intact) and the lateral image, there is still a remaining dorsal gap. For this reason, it may be recommended to acquire a true ML X-ray image because the true ML image has the highest probability of showing such a gap. However, it can still happen that even a true ML image does not reveal such an inaccurate reduction.

[0209] The remaining dorsal gap, which is also not visible in X-rays, has limited degrees of freedom, which must mean that inaccurate reduction can be corrected by rotating the medial fragment around the axis defined by the main fracture line.

[0210] Such correction may be achieved by the following procedure, which is shown here for the case of two fragments.

[0211] 1. The system loads a 3D model (typically obtained using a preoperative CT scan) showing the segmented 3D bone fragments. 2. The surgeon acquires an AP X-ray image. 3. The system may optionally detect the fracture line as a reference. 4. The system may optionally determine a line approximating the main fracture line. 5. The surgeon reduces the fracture until it looks correct in the AP X-ray image. 6. The surgeon rotates the C-arm to the ML position and acquires an X-ray image. 7. The system may determine the relative 3D position and 3D orientation between the two bone fragments, thus supporting the surgeon in evaluating the reduction and potentially determining the correct reduction.

[0212] For step 7, the system may use the a priori information that the fragments are in contact with each other along the anterior fracture line. This line is actually defined in 3D, so the detection in step 3 above is merely optional.

[0213] In this scenario, the two objects (fragments) are in contact not only at a single point but also along a one-dimensional structure approximating a line in 3D space. Therefore, only one degree of freedom that is not defined (i.e., rotation around the anterior fracture line) exists. Therefore, it is possible to determine the relative 3D position and 3D rotation between the objects using the concepts shown in the previous section.

[0214] By combining this method with other techniques described elsewhere (i.e., alignment of a very large number of X-ray images, with or without additional devices / implants (e.g., pins) in the X-ray images), higher accuracy can be obtained. This may not even require any additional effort if done after the pins are inserted, and it may help to use this to record both images.

[0215] Note that this type of a priori information exists in several reconstruction scenarios in orthopaedic trauma, e.g., determination of the varus / valgus position of fragments. Further examples are scenarios where it is known (e.g., based on X-rays), i.e., · the fragments are in contact with each other (the least restrictive type of a priori information) · If the fragments are in contact with each other, it may be sufficient to know that their position is within one of several possibilities. For example, if the reconstruction looks correct in the AP X-ray image, it may be assumed that the fragments are in contact along the fracture line in either the dorsal or ventral direction. In a more extreme scenario, the dorsal fracture line of one fragment may be in contact with the ventral fracture line of another fragment, and then the algorithm may evaluate all these possibilities and select the one that provides the best 3D match. · How the fragments are in contact with each other (in a 1D structure such as a point or a line, or a 2D structure such as a plane, etc.) may be relevant.

[0216] Note that the procedure shown above may also be applied to three or more fragments. If there is a known relationship between bone fragments A and B and a known relationship between bone fragments B and C, this may be used to relate bone fragments A and C.

[0217] The system may also automatically determine whether any detected anatomical gaps between bone fragments are within the normal range and whether any protruding bone fragments are statistically significant because they deviate significantly from the applied statistical shape model.

[0218] A further example where a surgeon may incorrectly believe that a reduction is correct based on what is seen in the X-ray image is the scenario of multiple fragments of the proximal tibia. FIG. 27 shows an axial view of the proximal end of the tibia including fragments A - E. Here, the fragments labeled A - D are already anatomically reduced, but fragment E is sunken, i.e., it has moved distally compared to the correct anatomical reduction. Such a situation can be difficult for the surgeon to decide because the X-ray (in the AP or ML direction) shows many other fracture lines and also lines corresponding to regular anatomical structures (e.g., the fibula). The present invention may be able to detect such a scenario by accurately determining the relative 3D positions and 3D orientations between all the fragments. This may be possible because the system can use the a priori information that all fragments, except possibly the medial fragment (such as fragment E in FIG. 27, which may have moved distally), are anatomically reduced. Here, the free parameter of the system is the proximal / distal position of the fragment located inside the bone.

Claims

1. A software program for processing X-ray images in computer-assisted surgery, the software program including instructions that, when executed by a processing device, cause the processing device to: Receive an X-ray image, the X-ray image being a projection image of a first object and a second object; Classify the first object based on the X-ray image and receive a 3D model of the first object, determine a geometric aspect of the first object based on the X-ray image, and identify the geometric aspect with respect to the 3D model of the first object; Classify the second object based on the X-ray image and receive a 3D model of the second object, select a point of the second object, and identify the point in the 3D model of the second object; Determine a relative 3D position and 3D orientation between the first object and the second object based on knowledge of (i) the 3D model of the first object, (ii) the 3D model of the second object, and (iii) the spatial relationship between the point of the second object and the geometric aspect of the first object; A software program that causes the above to be performed.

2. The geometric aspect of the first object is a plane or a line, and the X-ray image is generated in a state where the imaging direction is inclined with respect to the geometric aspect at an angle within a range of 20° to 30°. The software program according to claim 1.

3. When executed by the processing device, the processing device is further caused to include an instruction to determine a deviation of the 3D position and 3D orientation of the second object from the intended spatial relationship of the second object with respect to the first object. The software program according to any one of claims 1 and 2.

4. The first object is an anatomical structure or a side of a first implant, and the second object is an anatomical structure, an instrument, or a side of a second implant. The software program according to any one of claims 1 to 3.

5. The point of the second object is the distal tip of the second object, and information on the 3D position of the distal tip is the contact point of the distal tip with the surface of the first object. The software program according to any one of claims 1 to 4.

6. The software program is configured to assist in the insertion of a bone screw into the bone and through a hole in the bone screw for the fixation of the bone screw, the bone screw being in a known position within the bone, The first object is the bone, and the geometric aspect of the first object is the outer surface of the bone, The second object is a drill, and the point of the second object is the distal tip of the drill, The X-ray image is a projection image of the drill and the bone screw within the bone. The drill is arranged such that its distal tip is in contact with the outer surface of the bone and is positioned on a hole axis extending through the hole in the bone screw with the tip of the drill extending through the hole in the bone screw, Determining the relative 3D position and 3D orientation between the first object and the second object includes determining the actual angle between the drill axis of the drill and the hole axis based on (i) a 3D model of the bone, (ii) a 3D model of the drill, and (iii) the knowledge that the distal tip of the drill is in contact with the outer surface of the bone, The software program according to claim 1, further comprising an instruction that, when executed by the processing device, causes the processing device to determine a deviation of the 3D position and 3D orientation of the drill axis from the hole axis.

7. The software program is configured to assist in the insertion of a bone screw into the bone and through a hole in the bone screw for the fixation of the bone screw, the bone screw being within the bone, The first object is the bone screw, and the geometric aspect of the first object is the hole axis extending through the hole in the bone screw, The second object is a drill, and the point of the second object is the distal tip of the drill, The X-ray image is a first X-ray image of the drill and the bone screw within the bone. The first X-ray image has a first imaging direction in the direction of the hole axis. The drill is arranged such that its distal tip is in contact with the outer surface of the bone and is positioned on the hole axis, and the drill axis of the drill is oriented at an angle within the range of 10 to 70° with respect to the hole axis. Determining the relative 3D position and 3D orientation between the first object and the second object includes determining the actual angle between the drill axis and the hole axis based on (i) a 3D model of the bone screw, (ii) a 3D model of the drill, and (iii) knowledge of the contact point on the outer surface of the bone at the distal tip of the drill. The software program further includes an instruction that, when executed by the processing device, causes the processing device to receive a second X-ray image of the drill and the bone screw within the bone, the second X-ray image having a second imaging direction oriented at an angle within the range of 10 to 65° with respect to the first imaging direction, and the orientation of the drill being changed such that the tip is still on the hole axis and the drill axis is near the hole axis. determine the deviation of the 3D position and 3D orientation of the drill axis from the hole axis. The software program according to claim 1, further including an instruction to cause the above to be performed.

8. The software program is configured to assist in the insertion of a bone screw into the pedicle of a vertebra. The first object is the vertebra, and the geometric aspect of the first object is the pedicle axis extending through the pedicle of the vertebra. The second object is a drill, and the point of the second object is the distal tip of the drill. The X-ray image is a projection image of the drill and the vertebra, and the drill is arranged such that its distal tip is in contact with the outer surface of the vertebra with the tip positioned on the pedicle axis, and the drill axis of the drill is oriented at an angle within the range of 10 to 65° with respect to the imaging direction of the X-ray image. Determining the relative 3D position and 3D orientation between the vertebra and the drill includes determining the actual angle between the drill axis and the pedicle axis based on (i) a 3D model of the vertebra, (ii) a 3D model of the drill, and (iii) knowledge of the contact point on the outer surface of the vertebra at the distal tip of the drill, based on the 3D model of the drill and based on the 3D model of the vertebra. The software program according to claim 1, further including an instruction that, when executed by the processing device, causes the processing device to determine the deviation of the 3D position and 3D orientation of the drill axis from the pedicle axis.

9. The software program, when executed by the processing device, further includes an instruction to cause the processing device to generate a second X-ray image of the drill and the vertebra, and the orientation of the drill is changed such that the tip of the drill is still on the pedicle axis and the drill axis is near the pedicle axis. The software program according to claim 8.

10. The apparatus for generating the first X-ray image and the second X-ray image is a C-arm based X-ray imaging device, and the C-arc of the X-ray imaging device is rotated around the C-axis so as to change from the first imaging direction to the second imaging direction. The software program according to claim 7.

11. The software program according to claim 7, further including an instruction to cause the processing device to align the first X-ray image and the second X-ray image when executed by the processing device.

Images