Determining the relative 3D position and orientation between objects in 2D medical images.

Abstract

This invention provides an apparatus, method, and computer program for determining 3D representations and the relative 3D positions and orientations between objects based on X-ray projection images. [Solution] The device receives X-ray images, which are projection images of a first object and a second object, classifies the first object and the second object, and receives 3D models of each of those objects. For the first object, geometric aspects such as axes or lines are determined, and for the second object, other geometric aspects such as points are determined. Finally, based on the information that the 3D model of the first object, the 3D model of the second object, and points of the second object are located on the geometric aspects of the first object, the spatial relationship between the first object and the second object is determined.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer-assisted surgery. In particular, the present invention relates to an apparatus and method for determining a 3D representation as well as relative 3D positions and relative 3D orientations between objects based on X-ray projection images. The method may be implemented as a computer program executable on a processing device of the apparatus.

Background Art

[0002] In orthopedic or orthopedic trauma surgery or spinal surgery, it is a common procedure to aim relatively thin instruments at a target object or a target structure (as part of the target object). The target structure may be an anatomical structure (e.g., a pedicle) or part of another instrument or an implant (e.g., the distal fixation hole of a long antegrade intramedullary nail). Generally, the goal may be to determine the relative 3D position and relative 3D orientation between the instrument and the target object. This can be difficult based on available intraoperative 2D imaging techniques. It is particularly difficult when the exact geometric shape of the target object is unknown and / or the instrument is known but not uniquely locatable in 3D space based on 2D X-ray images.

[0003] A preoperative CT scan may be performed for a surgical procedure, which allows for a more accurate planning of the procedure. This is for example when performing surgery within a complex 3D structure, or when drilling holes or placing screws within a narrow anatomical structure or near important structures (e.g., the spinal cord, nerves, aorta). Typical examples of such procedures are the placement of sacroiliac joints or pedicle screws. When the target structure is an instrument or an implant, its 3D geometry is typically known. An example is a distal fixation procedure, where 3D models or 3D information regarding the target object (the nail) and in particular the target structure "distal fixation hole" (cylinder) are available.

[0004] However, for surgeons to utilize this 3D information and apply it to intraoperative 2D X-ray images requires a high level of spatial awareness and imagination.

[0005] In some cases, for certain procedures, it may be possible to determine the drilling direction by aligning the drill with a specific viewing direction on the imaging equipment (e.g., a true medial-lateral lateral view for distal fixation procedures). Furthermore, it is generally not possible to guarantee that the drilling will actually proceed precisely along this direction. This will be illustrated next with the example of distal fixation of a long antegrade intramedullary nail.

[0006] In conventional distal fixation procedures for antegrade nails, the surgeon moves the C-arm to a true lateral position (meaning the fixation hole appears perfectly circular in the X-ray image). This positioning is repetitive, monotonous, and time-consuming, sometimes taking several minutes, and typically requires the acquisition of 5 to 20 X-ray images due to the corresponding readjustment of the C-arm. A faster way to achieve this positioning is to use the C-arm's fluoroscopy mode (which generates a continuous X-ray video stream), but this results in a higher X-ray dose.

[0007] Furthermore, to ensure high accuracy for distal fixation, the hole must not only appear circular but also be close to the center of the X-ray image. However, in practice, if the hole appears sufficiently circular in the X-ray image, this C-arm position is typically used for distal fixation even if the hole is not close to the center of the X-ray image. Due to the conical shape of the X-ray beam fan, the more the direction of the X-ray beam is tilted, the further the beam moves from the center of the X-ray image. Therefore, drilling with a drill through the hole must be in the direction of the focal point of the X-ray source and not parallel to the center line between the X-ray source and the detector.

[0008] In the next step, the drill tip can be positioned at the intended drill location, and an X-ray image is taken. Here, the drill may be intentionally held at an angle (i.e., not in the direction of the fixed trajectory) so that the power drill and the surgeon's hand do not obstruct the view. The goal is to position the drill so that in the X-ray image, the drill tip appears in the center of the (circular) fixed hole. This is also repeated, typically requiring 5 to 10 repetitions and X-ray images.

[0009] Once this is achieved with sufficient precision, the drill is aligned with the target trajectory while leaving the drill tip in place. At this angle, the power drill and the surgeon's hand obstruct the view, so this alignment is typically not visible on X-ray. Therefore, the surgeon attempts to use the position of the C-arm as a guide to align the drill parallel to the "C". Achieving and then maintaining such alignment during drilling requires a fairly high level of manual dexterity. Furthermore, surgeons typically do not adhere to the requirement to aim at the focal point of the X-ray source. The greater the error caused by failing to aim at the focal point, the further the fixation hole appears from the center of the X-ray image. A typical error is in the range of 1-2 mm at the target point (fixation hole). This is close to the 3 mm limit, and beyond this, distal fixation will fail with most nail fixation systems. All of this means that attempts to drill are likely to be unsuccessful, especially for inexperienced or less skilled surgeons.

[0010] Since fixation generally requires two or more holes, the entire procedure must be repeated for each hole. Consequently, completing the entire nail fixation procedure is typically very time-consuming, requires many X-ray images, and often results in unsuccessful attempts at drilling holes. This means that distal fixation is one of the most frustrating procedures in the field of osteosynthesis. This sometimes leads to the quick fixation method of using shorter nails instead of longer ones, which then further worsens the patient's prognosis and results in a considerable number of revision surgeries.

[0011] For this reason, some manufacturers offer flexible mechanical solutions (hereinafter referred to as "long sighting devices") that adjust the bending of the nail in the pulp canal. While long sighting devices simplify the procedure, their application is still not straightforward, as the X-ray image showing the long sighting device must be accurately interpreted and the C-arm position adjusted accordingly. The long sighting device can only be properly adjusted after the C-arm has been correctly adjusted.

[0012] EP 2801320 A1 (European Patent Application Publication No. 2801320) proposes a concept for detecting a reference body having a metal marker at a known position fixed to a long aiming device and determining the imaging direction of the reference body. Based on this, the system can provide instructions on how to adjust the C-arm imaging device. A drawback of such a system is that the X-ray image must include the reference body. For adjusting a long aiming device in the case of an antegrade femoral nail with a fixing hole facing laterally, US 2013 / 0211386 A1 (United States Patent Application Publication No. 2013 / 0211386) uses a reference body to determine the bending of the nail in the ML direction and the AP direction, respectively.

[0013] Beyond distal fixation, to enhance the safety and accuracy of minimally invasive procedures, surgeons generally need access to necessary intraoperative 3D information, namely information regarding the relative 3D position and orientation between instruments and target objects / structures or between multiple anatomical objects. This information may be displayed by a tracking-based navigation system, which requires preoperative 3D imaging, followed by the recording of intraoperative 2D imaging data along with the preoperative 3D data during surgery. Another alternative is the use of a database (e.g., an implant database) containing 3D information about the target object and further instruments (e.g., long aiming devices), which often take the form of additional hardware. These systems are often cumbersome, time-consuming, and monotonous to set up and use intraoperatively, and are typically expensive. Therefore, for all these shortcomings, navigation-based systems are not always available, let alone viable, for use in orthopedics and trauma. The same explanation applies to systems for intraoperative 3D imaging (e.g., O-arm, 3D C-arm), which also involve high X-ray doses.

[0014] This highlights the need for a non-invasive and easy-to-use system that can provide intraoperative 3D information without a tracking system and without requiring any further hardware components. The present invention proposes a system and method that requires only a computer and a display and / or loudspeaker to process intraoperative 2D images. Both techniques that utilize deterministic 3D data of the target object (e.g., in the form of 3D preoperative imaging data or 3D model data of an implant) and techniques that do not require such data are provided. [Overview of the project]

[0015] To simplify product development, enable more cost-effective and more closely resemble typical operating room workflows, and eliminate the further uncertainty posed by mechanical interfaces for reference devices (e.g., when using new implants), it is preferable to work without using any reference devices or other additional hardware (e.g., aiming devices).

[0016] As described herein, the present invention proposes combining knowledge of the X-ray image generation process with artificial intelligence (in the form of so-called deep morphing and / or the use of neural networks) in place of any reference body to provide information required, for example, when treating a fractured bone. Accordingly, the object of the present invention may be considered to be providing an apparatus and / or method that enables 3D representation and determining the relative 3D position and relative 3D orientation between a plurality of objects that are at least partially visible in an X-ray projection image. Here, the objects may be any objects visible in the X-ray image, such as anatomical structures, implants, surgical instruments and / or parts of implant systems.

[0017] The term "3D representation" may refer to a complete or partial description of a 3D volume or 3D surface, which may also refer to selected geometric aspects such as radii, axes, and planes. While it may be possible to determine complete 3D information about the 3D surface or volume of an object, in many applications, it may suffice to determine only selected geometric aspects.

[0018] Throughout this application, the terms “locating” and “positioning” refer to the determination of the 3D orientation of an object and the determination of the 2D spatial position of its projection onto the image plane. The imaging depth (which is the distance of the object from the image plane), on the other hand, is estimated (with some uncertainty) based on a priori information regarding the typical arrangement of objects in an operating room, e.g., the relative positions of implants, patients, and imaging devices. For most purposes of the present invention, such estimated imaging depth is sufficient. Some applications may require a more accurate determination of the imaging depth, which is possible in certain cases, as will be further discussed below.

[0019] According to one embodiment, 3D reconstruction (i.e., determination of 3D representation) and localization of an object whose shape and appearance have some variability are provided. This can be done based on a single X-ray image, or, for higher accuracy, based on multiple X-ray images. Even if the object is not visible or only partially visible in the X-ray image, the 3D representation and localization of the relevant object, such as anatomical structures, implants, surgical instruments, and / or parts of implant systems, can also be provided.

[0020] According to one embodiment, relative 3D position and 3D orientation determination between multiple objects is provided even when the localization of at least one object is not individually possible with sufficient accuracy. This may also be achieved by utilizing a priori geometric information regarding the relative position and / or orientation between the sides of at least two objects, and optionally by limiting the range of acceptable X-ray imaging directions. Possible clinical applications include freehand distal fixation, placement of sacroiliac joint (SI) or pedicle screws, and evaluation of anatomical reduction of fractures.

[0021] The processed X-ray image data may be received from an imaging device, for example, a C-arm-based 2DX-ray device, or it may be received directly from a database. Furthermore, aspects of the present invention may be used to process medical images acquired using other imaging diagnostic methods such as ultrasound or magnetic resonance imaging.

[0022] A system proposed according to one embodiment generally comprises at least one processing unit configured to execute a computer program product that includes a set of instructions for (i) causing the processing unit to receive an X-ray projection image whose characteristics are determined by imaging parameters, (ii) causing the processing unit to classify at least one object in the X-ray projection image, (iii) causing the processing unit to receive a model of the classified object, and (iv) causing the processing unit to determine a 3D representation of the classified object and to locate the classified object with respect to a coordinate system by matching a virtual projection of the model with the actual projection image of the classified object. This process may take into account the characteristics of the X-ray imaging method. In particular, the fact that Thales' theorem applies may be taken into account, as discussed in later examples.

[0023] The X-ray projection image may represent the anatomical structure of interest, particularly bone. The bones may be, for example, the bones of the hand or foot, i.e., the long bones of the lower limb such as the femur and tibia, as well as the long bones of the upper limb such as the humerus, vertebrae, or pelvis. The image may also include artificial objects such as surgical instruments (e.g., drills) or bone implants already inserted into or fixed to the imaged anatomical structure of interest.

[0024] In the context of this invention, a distinction is made between “object” and “model.” The term “object” is used for a physical object, such as a bone or a part of a bone or another anatomical structure, or for an implant such as an intramedullary nail, bone plate or bone screw, or for a surgical instrument such as a sleeve, k-wire, scalpel, drill or aiming device that may be connected to an implant. “Object” may also describe only a part of a physical object (e.g., a part of a bone), or it may be an assembly of physical objects and therefore consist of sub-objects. Furthermore, to emphasize that an object is a sub-object of another object, it may be called a “structure.” For example, the “fixation hole” of a nail may be considered a structure (or sub-object) of the “nail” which is an object. As another example, the “pedicle” of a vertebra may be considered a structure of the “vertebra” which is an object. Nevertheless, a structure (such as a pedicle) itself may simply be called an “object.”

[0025] The term "model" is used for a virtual representation of an object (or a sub-object or structure). For example, a dataset defining the shape and dimensions of an implant may constitute a model of the implant. As another example, a 3D representation of an anatomical structure, such as one generated during a diagnostic procedure, may be considered a model of a real anatomical object. The "model" may describe a specific object, such as a specific nail or the left femur of a particular patient, or it may describe a class of objects, such as a femur, with some variability. In the latter case, such objects may be described, for example, by a statistical shape or appearance model. Therefore, an object of the present invention may be to find a 3D representation of a specific instance of an object from a class of objects depicted in an acquired X-ray image. For example, the object may be to find a 3D representation of a vertebra depicted in an acquired X-ray image based on a general statistical shape model of a vertebra. It may also be possible to use a model that includes a discrete set of deterministic possibilities, and the system then selects which of these best describes the object in the image. For example, there may be several nails in the database, and the algorithm then identifies which nails are depicted in the image (if this information has not been provided by the user beforehand).

[0026] Since the model is essentially a computer dataset, it is easy to extract specific information from that data, such as the geometric aspects and / or dimensions of the objects that are essentially represented.

[0027] A model may include two or more parts of the imaged object, and in some cases one or more of these parts may be invisible in the X-ray projection image. For example, a model of an implant may include a screw intended to be used with the implant, but if only the implant has already been introduced into the anatomical structure, only the implant will be visible in the X-ray projection image.

[0028] Note that the model may not be a complete 3D model of the actual object in the sense that it describes only specific geometric aspects of the object, such as the fact that it can approximate the femoral head by a 3D ball and a circle in a 2D projection image, or the fact that the pedicle of the vertebra has a cylindrical shape.

[0029] According to one embodiment, a system for processing X-ray images generally includes a processing device and a software program product. Here, when the software program product is executed by the processing device, the system is caused to perform the following steps. First, receive an X-ray image that is a projection image of at least a first object and a second object. Then, classify at least the first object and the second object, and receive a 3D model of each of these objects, for example, from a database. Based directly on the X-ray image and / or based on each 3D model, determine and identify a first geometric aspect of the first object with respect to the 3D model of the first object, and determine and identify a second geometric aspect of the second object in the 3D model of the second object. The second geometric aspect may be a point.

[0030] Furthermore, based on the information that the 3D model of the first object, the 3D model of the second object, and the second geometric aspect (e.g., a point of the second object) are located on the geometric aspect of the first object, determine the spatial relationship between the first object and the second object. The geometric aspect of the first object may be a plane, a line, or a point. Naturally, the geometric aspect may include a plurality of the above aspects and combinations thereof. As a result, the geometric aspect used herein may be a more complex shape, such as the edge of a fragment, in the case of a fracture.

[0031] According to one embodiment, the geometric side of the first object is a plane or a line, and an X-ray image is generated with the imaging direction tilted relative to the geometric side at an angle in the range of 10° to 65°. In fact, the plane or line associated with the first object may be tilted relative to the imaging direction. The X-ray image generated at the tilted imaging direction only needs to contain enough information about the first object to enable the determination of the geometric side. In other words, the appearance of the first object as visible in the X-ray image provides the processing unit of the system with enough information to classify the object and automatically identify its geometric side. The angle range may also be 15° to 45°, or it may be in the range of 20° to 30°. Assuming the system gives instructions to the user for adjusting the C-arm, the system may instruct the user to orient the imaging direction relative to the geometric side of the first object, or to orient the geometric side of the first object at an angle of, for example, 25°.

[0032] In a further embodiment, the system is made to further determine the deviation of the 3D position and 3D orientation of the second object from the intended spatial relationship of the second object to the first object.

[0033] For example, the first object may be an anatomical structure or the side of the first implant, and the second object may be an instrument or the second implant. As described in more detail below, the first object may be a vertebra and the second object may be a pedicle screw. Alternatively, the first object may be an intramedullary nail and the second object may be a fixing screw for distally fixing the nail. Alternatively, the second object may be a drill or k-wire used to prepare a pathway for the screw that passes through into the bone. Alternatively, the first and second objects may each be bone fragments that must be anatomically reduced.

[0034] According to one embodiment, the selected point of the second object may be the tip of the object (for example, the tip of a drill), and the information regarding the 3D position of the tip may be the point of contact between the tip and the surface of the first object (for example, the outer surface of a long bone such as a vertebra or femur).

[0035] According to one embodiment, the system may include a device for providing information to the user, where the information includes at least one piece of information from a group consisting of X-ray images and instructions relating to a procedure. Naturally, such a device may be a monitor for visualizing the information, or a loudspeaker for providing the information audibly.

[0036] In yet another embodiment, the characteristics of the X-ray imaging apparatus intended for use with the software program product executed on the processing unit of the System may be known. On the one hand, the imaging characteristics may be known and taken into consideration when processing the X-ray image data, and on the other hand, instructions for adjusting the C-arm may be made to the user for imaging based on the known geometric shape of the imaging apparatus and the possibility of changing the position and orientation of the C-arm. A C-arm based X-ray imaging apparatus may be part of the System.

[0037] The appearance of an object in a projected image may be influenced by the X-ray imaging procedure. For example, imaging parameters such as the imaging direction relative to gravity (this describes the direction in which the X-ray beam passes through the object, also called the "viewing direction"), zoom, radiation intensity, and / or the presence of a magnetic field may influence the appearance of an object in a projected image. These or further imaging parameters may cause characteristic changes in the projected image, such as deformation of the projected object due to the pillow effect, mechanical bending of the C-arm imaging device depending on the imaging direction, curvature, noise, and / or distortion. Here, these changes are expressed as image characteristics.

[0038] Naturally, it may be possible to determine these image properties with sufficient accuracy in projected images. For example, the location of structures shown in the edge regions of an image may be more affected by the pillow effect than structures in the center of the image. As a result, the properties of the pillow effect can be determined with sufficient accuracy based on known shapes of structures extending from the edge regions to the central regions. The image properties determined for a region in 2DX lines may be extrapolated to the entire image.

[0039] The appearance of an object in an X-ray image is further determined, in particular, by the attenuation, absorption, and deflection of X-ray radiation, which are determined by the material of the object. The more material the X-ray beam must pass through, the less X-ray radiation is received by the X-ray detector. This can alter not only the appearance of the object within its outline, but also the shape of the outline itself in the X-ray projection image, especially in areas where the object is narrow. The strength of this effect is determined by the X-ray intensity and the amount of tissue surrounding the object through which the X-ray beam must pass. The latter is determined by the patient's body mass index and imaging direction. The amount of soft tissue surrounding the object can be obtained from a database, which takes into account, for example, ethnicity, sex, body mass index, and age.

[0040] Taking into account image and object properties, as well as the effects of X-ray attenuation, absorption, and deflection, the virtual projection of the model may be deformed and / or distorted so that the object is deformed and / or distorted in the X-ray projection image. Such a virtual projection may then be matched to the projection seen in the X-ray image. Naturally, matching the object to the model in the X-ray projection image may include fitting the image properties of the X-ray projection image to the image properties of the virtual projection of the model and / or fitting the image properties of the virtual projection of the model to the image properties of the X-ray projection image. Naturally, matching in 3D projection volume by minimizing distance in 3D may also be possible.

[0041] Since the X-ray beam originates from an X-ray source (focal point) and is detected by an X-ray detector in the image plane, the physical dimensions of an object are related to the dimensions of its projection in the X-ray image according to Thales' theorem (also known as the basic proportionality theorem). An accurate imaging depth (which is the distance of the object from the image plane) is generally not required in the context of this invention. However, if the object is sufficiently large, the imaging depth may be determined by Thales' theorem, and this determination becomes more accurate as the object increases in size. Even for small objects, an approximate estimation of the imaging depth may be possible. Alternatively, the imaging depth can be determined if the size of the X-ray detector and the distance between the image plane and the focal point are known.

[0042] In one embodiment, a deep neural network (DNN) may be used to classify objects in an X-ray projection image (e.g., the proximal portion of the femur, the distal portion of the femur, the proximal portion of a nail, or the distal portion of a nail). Furthermore, objects may be classified without determining their position using the DNN (see, for example, Krizhevsky, A., Sutskever, I., and Hinton, GE, "ImageNet classification with deep convolutional neural networks" in NIPS, pp. 1106-1114, 2012). It should also be noted that objects can be classified even when it is known which objects should be recognizable in the X-ray image. Furthermore, neural networks may be used for a rough classification of imaging directions (see, for example, AP vs. ML, paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jurgen Schmidt: "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge", 2018 (available online at https: / / arxiv.org / abs / 1808.04441)). An appropriate model may be selected using such classification of objects and imaging directions to follow the processing steps. The classification may also be performed by other means or by a priori information regarding which objects are visible in the image.

[0043] In one embodiment, the outline of a classified object may be detected in an X-ray image. For objects with variable shapes, such as anatomical structures, this may be done using the “deep morphing” technique described in the paper cited above by Pries et al. (2018). This paper proposes a deep neural network-based method for detecting bone structures in fluoroscopic X-ray images. Specifically, this technique addresses the challenges in automated processing of fluoroscopic X-rays, namely their low quality and the fact that typically only small datasets are available for training neural networks. This technique incorporates a high level of information about the morphological object of a statistical shape model. This technique consists of a two-stage method (called deep morphing), where in the first stage a neural segmentation network detects the contour (outline) of bone or other objects, and then in the second stage a statistical shape model is fitted to this contour using a variation of the active shape model algorithm (although other algorithms can also be used for the second stage). This combination allows the technique to label points on the object contour. For example, in femoral segmentation, this technique can determine which points on the contour in a 2DX projection image correspond to the lesser trochanter region and which points correspond to the femoral neck region. Objects described by deterministic models (e.g., nails) may also be detected by deep morphing, or simply by a neural segmentation network, similar to the first stage of deep morphing.

[0044] In a subsequent step, the virtual projection of the model may be adjusted to match the appearance of the object in the X-ray projection image, taking into account image and / or object properties as well as the effects of X-ray attenuation, absorption, and deflection. According to one embodiment, for an object described by a deterministic model, this matching may be carried out in accordance with the content described, for example, in the article Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps" in Laugier C. (ed.), "Geometric Reasoning for Perception and Action," GRPA 1991, Lecture Notes in Computer Science, Vol. 708, Springer, Berlin, Heidelberg. In this method, image and object properties as well as the effects of X-ray attenuation, absorption, and deflection may be compensated for by introducing further degrees of freedom into the parameter vector or by using a suitably tuned model.

[0045] A neural network may be trained on a very large amount of data comparable to the data to which it is applied. In the case of evaluating bone structures in images, the neural network must be trained on a very large amount of X-ray images of the bone of interest. Of course, the neural network may also be trained on simulated X-ray images. Simulated X-ray images may be generated, for example, from 3DCT data, as described in the appendix of the paper: Aaron Pries, Peter J. Schreier, Artur Lamm, Stefan Pede, Jurgen Schmidt: "Deep morphing: Detecting bone structures in fluoroscopic X-ray images with prior knowledge" (available online at https: / / arxiv.org / abs / 1808.04441).

[0046] According to one embodiment, two or more neural networks may be used, each of which may be trained for a sub-step specifically required to achieve a desired solution. For example, a first neural network may be trained to evaluate X-ray image data in a 2D projection image to classify anatomical structures, a second neural network may be trained to detect the location of the structure in the 2D projection image, and a third network may be trained to determine the 3D location of the structure with respect to a coordinate system. Neural networks may also be combined with other algorithms, such as active shape models, but are not limited to these. It should be noted that neural networks can also be trained to locate objects or determine imaging directions without first needing to detect the outline of the object in a 2D X-ray image. It should also be noted that neural networks can be used for other tasks, such as determining one or more image properties, such as the pillow effect.

[0047] According to one embodiment, objects may be manually classified and / or identified in an X-ray projection image. Such classification or identification may be supported by the device by automatically referencing structures recognized by the device.

[0048] According to one embodiment, the system may calculate the geometric aspects of an object (e.g., axes, planes, orbits, outlines, curvature, center points, or one- or two-dimensional manifolds) and the dimensions of the object (e.g., length, radius, or diameter, distance). This may be achieved by corresponding a model with a virtual projection that matches the projection seen in an X-ray image.

[0049] When displaying X-ray projection images, geometric aspects and / or dimensions may be shown as overlays in the projection image. Alternatively and / or additionally, at least a portion of the model may be shown in the X-ray image, for example, as a transparent visualization or 3D rendering, which may facilitate the user's identification of the structural aspects of the model and thus the imaged objects.

[0050] The present invention provides 3D reconstruction and localization of an object whose shape and appearance have some variability. Such an object may be described, for example, by a 3D statistical shape or appearance model. This can be done based on a single or multiple X-ray images. Based on a single image of an anatomical object, the model may be deformed in such a way that its virtual projection matches the actual projection of the object in the X-ray image. When multiple X-ray images are acquired, the information from them may be fused (aligned) to improve the accuracy of the 3D reconstruction and / or determination of the spatial position or orientation. If the imaging direction is known or can be determined, or if the 3D angle between the imaging directions (which may be expressed, for example, by Euler angles) is known or can be determined when aligning multiple X-ray images, the matching of the virtual projection to the actual projection in the X-ray image (e.g., using deep morphing) can be done with greater accuracy.

[0051] According to one embodiment, even when the localization of at least one object is not individually possible with sufficient accuracy, the determination of relative 3D position and 3D orientation between multiple objects is provided. This may be addressed by utilizing geometric a priori information regarding the relative 3D position and / or 3D orientation between at least two objects / structures in an X-ray image. This may be, for example, information that a point on one object lies on a line in which its relative 3D position and orientation to another object are known. Another example is that the relative 3D position and 3D orientation between a geometric side of one anatomical object and a geometric side of another anatomical object are known. Since remaining ambiguity may still exist, it may be necessary to restrict the X-ray imaging direction to (i) a specific anatomically relevant figure (e.g., true ML) or (ii) an angular range that allows one of the objects to be viewed from a particular direction.

[0052] The processing unit may be implemented by a single processor that performs all steps of the process, or by a group or multiple processors that do not need to be located in the same place. Cloud computing, for example, makes it possible to place processors anywhere. For example, the processing unit may be divided into (i) a first subprocessor that implements a first neural network that evaluates image data including the classification of anatomical structures such as bone surfaces, (ii) a second subprocessor that implements a second neural network that is specialized in determining the imaging direction of the classified anatomical structures, and (iii) a further processor that controls a monitor for visualizing the results or a loudspeaker for providing audible instructions to the user. Alternatively, one of these processors or the further processors may control, for example, the movement of the C-arm of an X-ray imaging device.

[0053] According to one embodiment, the apparatus further comprises storage means for providing a database for storing, for example, X-ray images. It will be understood that such storage means may also be provided on a network (to which the system may be connected), and that data related to neural networks may be received through that network.

[0054] Furthermore, the apparatus may include an imaging device for generating at least one 2DX line image, and the imaging device may be capable of generating images from different directions.

[0055] The device may further include input means for manually determining or selecting the position or part of an object in an X-ray image, such as a bone outline, in order to measure distance in the image. Such input means may also be a computer keyboard, computer mouse, or touchscreen for controlling a pointing device such as a cursor on a monitor screen, and these may also be included in the device.

[0056] It should be noted that all references to C-arm movement or rotation in this application always refer to relative repositioning between the C-arm and the patient. Therefore, any C-arm movement or rotation may generally be replaced by the corresponding patient / operating table movement or rotation, or a combination of C-arm movement / rotation and patient / operating table movement / rotation. This may be particularly relevant when dealing with limbs, as actually moving the patient's limbs may be easier than moving the C-arm. It should be noted that required patient movement / rotation generally differs from C-arm movement / rotation, and in particular typically, patient translation is not required if the target structure is already in the desired position in the X-ray image. The system may calculate C-arm adjustments and / or patient adjustments.

[0057] The methods and techniques disclosed in this application may be used in systems supporting a human user or surgeon, or in systems in which some or all of the processes are performed by a robot. Therefore, all references to “user” or “surgeon” in this patent application may refer to a human user, a robotic surgeon, a mechanical support device, or similar device. Similarly, whenever it is mentioned that instructions are given on how to adjust the C-arm, such adjustments may, of course, be performed without human intervention, i.e., automatically by the robotic C-arm, or by surgical staff with some automated support. Since the robotic surgeon and / or robotic C-arm can perform surgery with greater precision than a human, repetitive procedures may require fewer repetitions, and more complex instructions (e.g., combining multiple repetitive steps) may be performed.

[0058] The computer program product may preferably be loaded into the random access memory of a data processor. Therefore, according to one embodiment, the system may be equipped with a data processor or processing unit to execute at least a portion of the described process. Furthermore, the present invention relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program can be stored. However, the computer program may also be provided through a network such as the World Wide Web and can be downloaded from such a network into the random access memory of a data processor. Furthermore, the computer program may be executed on a cloud-based processor, and the results may be provided through a network.

[0059] Prior information regarding the implant (e.g., nail size and type) may be obtained by simply scanning anything written on the implant packaging (e.g., barcode) or the implant itself before or during the surgical procedure.

[0060] For further understanding of the present invention, an exemplary method for distal fixation of a bone pin into a long bone is described, in which a bone screw is inserted into the long bone through a hole for the bone pin. This hole has a hole axis, which may be considered as the geometric side of the pin. The method may include the step of positioning a drill with its tip in contact with the outer surface of the long bone so that the tip of the drill is positioned on the hole axis of the hole for the bone pin, and the drill axis of the drill is oriented at an angle of 10 to 70° with respect to the hole axis. Here, the hole axis is a line as the geometric side of the first object, i.e., the bone pin, and the tip of the drill is a point as the geometric side of the second object, i.e., the drill.

[0061] The first X-ray image of a bone drill and bone nail in a long bone, with the tip of the drill on the outer surface of the long bone, is generated with the imaging direction aligned with the axis of the hole in the bone nail. Those skilled in the art will understand that the imaging direction may also be true medial-lateral, and therefore the hole should be visible as a circle in the case of distal fixation of a femoral nail.

[0062] The system may then determine the actual angle between the drill axis and the bore axis based on knowledge of the contact points, a 3D model of the drill, and a 3D model of the bone nail. Based on the determined angle, the system may instruct the bone drill to change its orientation so that the tip remains on the bore axis and the drill axis is close to the bore axis. Here, "close" means having a deviation of up to 15° from the bore axis. It may be sufficient to simply roughly align the drilling trajectory with the imaging direction.

[0063] A second X-ray image of the drill and bone nail in a long bone may be generated with the second imaging direction oriented relative to the first imaging direction within an angular range of 10 to 65°, while maintaining the position of the drill tip. An easy way to change the orientation is to move only the C-arc in the forward and backward directions, starting from the internal-external imaging direction. It is also crucial that the drill, i.e., the second object, is clearly visible in the subsequent X-ray image to enable automatic determination of, for example, the position and orientation of the drill axis. Therefore, its angle may be in the range of 10 to 65°, preferably 15 to 45°, and most preferably 20 to 30°.

[0064] Based on the second X-ray image, the 3D position and orientation deviation of the drill axis from the hole axis of the bone nail may be determined. If deviation occurs, the position and orientation of the bone drill may be adjusted, and the bore may be drilled through the bone nail hole into a longer bone. Drilling along the target trajectory at the same angle as the imaging direction relative to the drill axis may be confirmed by one or more X-ray images.

[0065] The principle of the present invention may also be applied to a method for inserting a bone screw into the pedicle of a vertebra. This method may include the step of positioning a drill with its tip in contact with the outer surface of the vertebra so that the tip of the drill is positioned on an axis extending through the pedicle of the vertebra, and the drill axis of the drill is oriented at an angle of 10 to 65° with respect to the target axis passing through the pedicle.

[0066] As described above, a first X-ray image is generated from the imaging direction, for example, the true AP, including the drill and vertebra, so that the opening of the pedicle is clearly visible. The difference between these two methods can be seen in that the imaging direction of the first X-ray image can be considered as the anterior-posterior direction with the patient lying face down and flat, and that the imaging direction does not need to coincide with the pedicle axis (target trajectory) because both objects are in contact with each other. Using such an inclined figure, the relative 3D position and 3D orientation between the drill and the pedicle may be determined based on knowledge of the contact points.

[0067] As a next step, the actual angle between the drill axis and the axis passing through the pedicle may be determined based on a 3D model of the drill and a 3D model of the vertebra. The drill may be rotated according to the instructions that the system can provide, so that the tip of the drill is still on the axis passing through the pedicle and the drill axis is close to the target axis passing through the pedicle. A second X-ray image may be generated from the same direction if the inclination of the pedicle axis with respect to the viewing direction is sufficiently large so that neither the power tool nor the hand obstructs the view. Typically, the inclination between the pedicle axis and the true AP viewing direction of the corresponding vertebra is 10–45°. The position and orientation of the drill may be adjusted as needed, and then a hole may be drilled through the pedicle into the vertebra.

[0068] The principles of the present invention may also be applied to a method for inserting a bone screw into the sacroiliac joint (SI). This method may include the step of positioning a drill with its distal tip in contact with the outer surface of the vertebra so that the tip of the drill is positioned on an axis extending through a desired drilling path through the SI joint, the drill axis of which is oriented at an angle of 10 to 65° with respect to a target axis passing through the pedicle.

[0069] As described above, a first X-ray image is generated including the relevant part of the drill, the ilium, and the sacrum, with the imaging direction coinciding with the direction of the drilling path. In the next step, the actual angle between the drill axis and the axis through which the drilling path passes may be determined based on knowledge of the contact points, based on a 3D model of the drill, and based on a 3D model of the vertebrae. The drill may be rotated according to instructions that can be given by the system so that the tip of the drill is still on the axis of the drilling path and the drill axis is close to the target axis of the drilling path.

[0070] Next, a second X-ray image of the drill, the relevant portion of the ilium, and the relevant portion of the sacrum is generated with the drill and imaging direction reversed. At this point, the drill may be approximately on the target trajectory through the drilling path, and the second imaging direction may be oriented relative to the first imaging direction at an angle in the range of 10 to 65°. The C-arc of the X-ray imaging device may be rotated relative to the tilted viewing direction. The angle range between the two imaging directions may also be 15 to 40°, or it may be 20 to 30°. Using such a tilted view, the 3D position and 3D orientation deviation of the drill axis from the target axis through the drilling path may be determined. If necessary, the position and orientation of the drill may be adjusted, and then holes may be drilled into the ilium and SI joint.

[0071] As will be apparent from the above description, the main aspect of the present invention is the processing of X-ray image data that enables the automatic interpretation of visible objects. The method described herein should be understood as a method to assist in the surgical treatment of patients. Accordingly, according to one embodiment, the method shall not include any steps of surgical treatment of the body of an animal or human.

[0072] It should be noted that the embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims (computer programs), and other embodiments are described with reference to apparatus-type claims (systems / apparatus). However, those skilled in the art will infer from the above and below descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, and any combination of features relating to different subject matter, are disclosed in this application.

[0073] The embodiments and further embodiments, features and advantages set forth herein may also be derived from the examples of embodiments described later herein, and are described with reference to the examples of embodiments shown in the drawings, but the present invention is not limited thereto. [Brief explanation of the drawing]

[0074] [Figure 1] This shows an example of 3D alignment of AP and ML images. [Figure 2] This example demonstrates 3D alignment of AP and ML images and shows the effect of inaccurately estimated C-arm width. [Figure 3] Compare the situations in Figure 1 and Figure 2. [Figure 4] This example demonstrates 3D alignment of AP and ML images and shows the effect of zooming. [Figure 5] Compare the situations in Figure 1 and Figure 4. [Figure 6] This example demonstrates 3D alignment of AP and ML images and shows the effect of X-ray receiver size. [Figure 7] An example of image distortion due to intramedullary nailing is shown. [Figure 8] The definition of drill angle is shown. [Figure 9] This shows a 3D array with two different drill positions. [Figure 10] Figure 9 shows the outline of the drill in the X-ray projection image corresponding to the 3D array. [Figure 11] The image shows a zoomed-in view within an X-ray image, revealing the outlines of two drills corresponding to different inclines (43-45°). [Figure 12] The image shows a zoomed-in view within an X-ray image, revealing the outlines of two drills corresponding to different inclinations (23-25°). [Figure 13] The outlines of the proximal femur are shown accurately and inaccurately, with the latter corresponding to an angular error of 2.5°. [Figure 14] The outlines of the proximal femur are shown accurately and inaccurately, with the latter corresponding to an angular error of 6°. [Figure 15] This shows the APX (Advanced Photon X-ray) of the lumbar spine. [Figure 16] Determine the rotation axis of the C-arm. [Figure 17] The diagram shows the circular and oblong holes of nails, including the chamfered edges. [Figure 18] This is an X-ray image showing a titanium nail rotated 25° around its axis so as to be away from a fixed plane. [Figure 19] This is an X-ray image showing a titanium nail rotated 45° around its axis so as to be away from a fixed plane. [Figure 20] This is an X-ray image showing the distal portion of a nail from an inaccurate imaging direction. [Figure 21] This is an X-ray image showing the distal portion of the nail from the correct imaging direction. [Figure 22] This is an X-ray image showing a nail and a drill with an improperly positioned drill bit. [Figure 23] This is an X-ray image showing a nail and a drill with a precisely positioned drill bit. [Figure 24] This shows a general workflow for the proposed procedure. [Figure 25] Figure 24 shows details for the rapid execution of a typical workflow. [Figure 26] Figure 24 shows details for executing a typical workflow with high accuracy. [Figure 27] This shows an axial view of the proximal end of the tibia.

[0075] Throughout the drawings, the same reference numerals and letters are used to indicate similar features, elements, components, or parts of the illustrated embodiments, unless otherwise specified. Furthermore, the present disclosure will be described in detail with reference to the drawings, which describe exemplary embodiments and are not limited to the specific embodiments shown in the drawings. [Modes for carrying out the invention]

[0076] 3D reconstruction and localization of anatomical objects based on a single X-ray image. The cited paper by Pries et al. (2018) on deep morphing proposes a method that allows a system to detect the outline / contour of bone (in 2D projection images) and label points on the contour. For example, in femoral segmentation, this technique can determine which points on the contour in the 2D X-ray projection image correspond to the lesser trochanter, which points correspond to the femoral neck, and so on. Then, considering a 3D statistical shape or appearance model of the same anatomical structure, this model can be deformed in such a way that its virtual projection matches the actual projection in the X-ray image, thus obtaining a 3D reconstruction of the anatomical structure, enabling the positioning of the object and the determination of the imaging direction. On the other hand, if the imaging direction is already known, the 3D reconstruction of the anatomical object can be performed with greater accuracy. For example, the imaging direction may be known because the surgeon is instructed to acquire X-ray images in a specific direction (e.g., true AP or true ML), or because a specific imaging direction has been detected, for example, by an algorithm invented by Blau, filed as a patent application on August 23, 2018.

[0077] The accuracy of 3D reconstruction of anatomical objects may be improved by using a priori information. This a priori information may be the patient's size or sex, but it may also be more anatomically specific information, such as geometric information about the anatomical object being reconstructed. In the example of 3D reconstruction of the proximal femur based on ML images, this information may include the length of the femoral neck or the CCD angle. However, since such information may not be determined with sufficient accuracy in typical ML images, it may be extracted from AP images, which are routinely acquired earlier in the course of surgical procedures on the proximal femur. The more information used from earlier images, the more accurate the 3D reconstruction may become. Another way in which this procedure is described is that 3D reconstruction of the proximal femur can be performed based on AP images with typical remaining uncertainties (such as the width of the femoral neck in the AP direction), and this 3D reconstruction can serve as a starting point for 3D reconstruction based on a priori information or later ML images.

[0078] Furthermore, geometric a priori information may consist of known correspondences between points in a 2D projection image and points in a 3D model of an anatomical object. For example, - The point in the 2D projection image corresponds to a point on a line whose position and orientation relative to the 3D model of the anatomical object are known, or - Points in a 2D projection image correspond to points on a plane whose position and orientation relative to a 3D model of an anatomical object are known. When certain parameters are already known, less specific geometric information can still be useful.

[0079] Such geometric a priori information may be provided by a surgeon, for example, through user input via a user interface, to position an object (e.g., an instrument such as a drill or k-wire) at a specific anatomical point visible in the 2D projection image. This may be achieved, in some cases, due to prominent anatomical features in a particular imaging direction (e.g., true AP or true ML), or by palpation or visual identification of the actual object. All of this a priori information significantly reduces ambiguity in 3D reconstruction.

[0080] Determining the relative 3D position and 3D orientation between objects by reducing or eliminating ambiguity. In an invention by Blau, filed as a patent application on November 26, 2018, it is considered how the relative 3D position and 3D orientation between two objects can be determined when the positions of two objects can be located based on 2D X-ray images and it is known that the two objects are in contact with each other in physical 3D space.

[0081] This invention proposes how to determine the relative 3D position and 3D orientation between two (or more) objects (or structures) based on 2D X-ray images when at least one object cannot be located with sufficient accuracy. Such determination of relative 3D position and 3D orientation may be possible based on a priori information regarding the 3D position of a specific point of one object relative to another object, which may be obtained, for example, from an X-ray image acquired earlier from a specific imaging direction that allows for the location of at least one structure of the other object. In order for this to work, it may be necessary to limit the acceptable range of imaging directions in the current image.

[0082] For illustrative purposes, let us assume that one of these objects is a drill and the others are nails or some anatomical objects (e.g., vertebrae), and we will now explain this. The anatomical objects may be described using either a deterministic 3D model (generated, for example, using a 3D imaging method for a specific patient) or a statistical 3D model describing general bone variability. In the former case, greater accuracy can be obtained. Even if a complete and accurate 3D model of the drill is available, it is not always possible to locate the drill with sufficient accuracy. As explained above, there is remaining uncertainty in the imaging depth (the distance of the object from the image plane) when locating an object. This uncertainty in the imaging depth in determining the 3D position of the drill tip also gives rise to ambiguity regarding the inclination of the drill in the direction of imaging depth. As shown in Figure 8, the inclination of the drill is defined as the viewing angle relative to the drill tip, represented by 8.DT. Because the drill is very thin and has a straight structure due to its clearly defined axis, the inclination of the drill may be defined as the angle between the dashed line represented by 8.L1, which connects the drill tip 8.DT and the X-ray focus 8.FP, and the solid line represented by 8.L2, which is the drill axis.

[0083] Consider, for example, a 3D configuration having two different 3D drill positions (represented as 9.D1 and 9.D2) as shown in Figure 9. The two drill positions differ in imaging depth and also in drill inclination (by 6.5°). Furthermore, the two drill positions in the X-ray projection images may not be distinguishable because they result in essentially the same X-ray projection, as indicated by nearly identical drill outlines 10.O in Figure 10.

[0084] There are two main reasons for this, as follows:

[0085] 1. A drill is a relatively thin instrument with a nearly constant diameter, which in a typical X-ray image is simply a few pixels wide. In X-ray projection, an object tilted in the imaging direction is depicted wider at one end and smaller at the other. However, this can only be detected if this change in width is sufficiently large (e.g., the width of at least one pixel). This may not generally apply to the accuracy required for angle detection of less than 3°, for example, in the case of a thin drill with a diameter of less than 4 mm.

[0086] Because the sine function for small angles has a slope close to zero, locating instruments such as drills (with a diameter of a few millimeters) where only the front portion is visible in an X-ray image from a viewing angle close to 90° (which is the tilt of the drill) may not be possible with sufficient accuracy in general. For example, at an angle of 70°, the projection of the drill is shortened by only 6 percent, which results in a slight change in the projected shape of the drill tip. Such a small change may not be sufficient to determine the tilt of the drill with an accuracy of about 3°. The limit of detection capability is a tilt of about 65°, where the projection of the drill is shortened by 9.4 percent. Depending on the instrument, this may or may not be sufficient for the required accuracy.

[0087] As the viewing angle (tilt) decreases, it becomes easier to distinguish a difference in tilt, for example, 2°. This is shown in the X-ray image of Figure 11, which shows the projections and outlines of two drills. The solid white line labeled 11.D1 corresponds to a drill with a 45° tilt, and the dashed white line labeled 11.D2 corresponds to a drill with a 43° tilt. These outlines differ in several places, and therefore can be distinguished by this system. As the viewing angle decreases, even more clearly distinguishable outlines are obtained. This can be observed in the X-ray image of Figure 12, which shows the projections and outlines of two drills. The solid white line labeled 12.D1 corresponds to a drill with a 25° tilt, and the dashed white line labeled 12.D2 corresponds to a drill with a 23° tilt. These outlines differ in several places at this point, and therefore can be easily distinguished by this system.

[0088] 2. In a typical X-ray image, only the tip and top of the drill are visible, while the other end is not. If both ends are visible, the angle of the drill can be determined with high accuracy based on the shortened length of the drill in the projected image. This is also possible if the drill has, for example, a clearly visible marking in the X-ray image partway along the shaft. Marking such a drill is easy, but it also means a change in existing instruments. Another option is to use the start of the threads of the drill as such a marking. However, when using a general drill, the start of the threads may not be clearly visible in the X-ray projection image, and therefore it may not be possible to use it for this purpose in general.

[0089] This problem may be addressed by more accurately determining the imaging depth of the drill tip, thereby significantly reducing or eliminating ambiguity. This may be possible when determining the relative 3D position and 3D orientation between two objects in an X-ray image. Let the other object be called the target object, which may be, for example, a nail. The ambiguity of the drill tip's position relative to the target object may be reduced by defining a trajectory whose 3D position and 3D orientation relative to the target object are known, provided that the drill tip's position lies on this trajectory and the imaging direction of the trajectory at the drill tip's position is sufficiently large (in other words, this trajectory must be sufficiently different from a line parallel to the line connecting the drill tip and the X-ray focal point).

[0090] This may be even more useful when the 3D position of the drill tip relative to a point on the target object is known. This is, for example, when the drill tip is in contact with the target object, such as when the drill tip is in contact with the ilium in sacroiliac joint screw fixation. However, it may be sufficient if it is known that the drill tip is on a plane in which its 3D position and 3D orientation relative to the target object are known. This is, for example, in the case of distal fixation of a further hole after the fixation of the first hole is complete. Here, the two degrees of freedom (DoF) are not determined. For a more detailed explanation of distal fixation procedures for nails, please refer to the corresponding sections below.

[0091] Furthermore, ambiguity regarding the target object may exist, especially when the target object is an anatomical object. However, when the instrument is in contact with the target object, there are imaging directions in which ambiguity in localizing each object is involved in different directions. Therefore, even in such cases, it may be possible to determine the relative 3D position and 3D orientation with sufficient accuracy.

[0092] The following describes these considerations regarding a drill in contact with the proximal femoral trochanter. In this imaging orientation, the point at which the drill tip is in contact with the femur may be clearly determined, for example, by palpation. Due to the ambiguity described above, there are several possibilities for the 3D position and orientation of the drill relative to the femur, all of which result in the same projection of the drill in the X-ray image, each corresponding to a different relative 3D position and orientation of the anatomical structure being depicted. By considering the depicted anatomical structure together and using a priori information about the point of contact, it may be possible to select which of these possibilities is correct.

[0093] Figure 13 shows an X-ray image of such a scenario. The solid white line (represented as 13.CO) is the outline of the femur corresponding to the correct 3D position and orientation of the femur, and the dashed white line (represented as 13.IO) is the outline of the femur corresponding to one of the possible inaccuracies regarding the 3D position and orientation of the femur. By comparing these possible outlines with a segmented and labeled femur in the X-ray image (which may be achieved, for example, by deep morphing), the one that best matches the segmented femur is selected. In the scenario depicted, the inaccurate outline 13.IO is clearly different from the correct outline 13.CO and may therefore be discarded even if the inaccurate outline corresponds to only a 2.5° angular error (with respect to the drill tilt). Larger angular errors can result in even more obviously inaccurate outlines, as depicted in Figure 14, where the inaccurate outline 14.IO corresponds to a 6° angular error (regarding the drill's tilt) and is clearly distinguishable from the correct outline 14.CO.

[0094] A further example may involve determining the relative 3D position and orientation of an instrument to the pedicle of a vertebra. Figure 15 shows an APX image of the lumbar spine, where a surgeon has placed a Jamsidi needle (labeled 15.JN) on the right pedicle of the lumbar spine. The opening of the pedicle (labeled 15.OP) is clearly visible as a brighter region in this particular imaging direction. Thus, the center of the pedicle can be clearly identified, and the instrument can be positioned at the center of this opening (pedicle axis). Based on the a priori information that the instrument is positioned on the pedicle axis, the relative 3D position and orientation between the instrument and the pedicle may be determined within a suitable angular range for many other imaging directions, by making contact with the bone surface and following the method outlined above.

[0095] Alignment process of two or more X-ray images from different directions Depending on the bone's shape, remaining ambiguity or mismatch may still exist in 3D reconstruction based on only one image. This can be mitigated by acquiring multiple images from potentially different viewing angles by rotating and / or translating the C-arm between images. Generally, additional images from different imaging directions are more useful, and the more different the imaging directions (e.g., AP and ML images), the more useful the additional images are in determining 3D information. However, instead of completely changing to a different figure (AP to ML or vice versa), even adding images from only slightly different viewing angles that are more readily available during surgery can be beneficial.

[0096] The present invention also makes it possible to record multiple X-ray images of at least one common object taken from different directions. This is important because 3D alignment allows for the determination of relative 3D positions between multiple objects without explicit determination of imaging depth.

[0097] For the 3D reconstruction of a variable-shaped object (typically an anatomical structure described by, for example, a statistical shape or appearance model, referred to in this section as "object F") based on two or more X-ray images, the procedure outlined above for one image may be extended to two or more images. That is, deep morphing may be used to detect the contour of object F and label points on that contour in each 2X-ray image. Considering the 3D statistical shape model of object F, this model can then be deformed so that its virtual projection matches the actual projection of object F as closely as possible in two or more X-ray images simultaneously. This procedure implicitly determines the imaging direction of each X-ray image and therefore does not require a priori information about the imaging direction.

[0098] As an alternative method for aligning a pair of X-ray images acquired from two different imaging directions, the accuracy of the alignment process can be improved by taking into account the 3D angle between the imaging directions, which can be determined using two different procedures. The more accurately this angle can be determined, the more accurate the 3D alignment can become.

[0099] One method for determining this angle is to determine the imaging direction using, for example, the invention of Blau, filed as a patent application on August 23, 2018, for each X-ray image, and to compare the differences. Another method may involve using another object in the X-ray image (referred to as "object G"), whose model (e.g., a nail connected to a sighting device or instrument, depending on the case) is deterministic. Object G may be localized by matching its virtual projection to its actual projection in each X-ray image. Without further conditions or a priori information, some objects, in particular instruments such as drills or k-wires, may not generally have sufficient geometric structure or size to be localized. However, even in such cases, it may be possible to localize object G with sufficient accuracy, provided that (i) object G is visible within a specific angular range in all images being aligned, and (ii) some prior information regarding the relative 3D positions between objects F and G is available. The prior information in (ii) is particularly, (a) The relative 3D positions of the points of object G and object F are known, or (b) A point of object G lies on a line in physical 3D space whose relative 3D position and 3D orientation to object F are known, or (c) A point of object G lies on a plane in physical 3D space whose relative 3D position and 3D orientation to object F are known. That would also be acceptable.

[0100] However, the relative 3D position and 3D orientation between objects F and G must be identical in both X-ray images, meaning there must be as little movement between the objects as possible.

[0101] In general, two or more images may be aligned if they contain objects that can be located with sufficient accuracy. Image alignment can be performed with high accuracy if the two or more images contain objects that do not move relative to each other between the acquisition of the images. One example in which this procedure can be used is when an implant (e.g., a nail) has already been inserted into the bone and a 3D model of the bone (e.g., a statistical shape model) is available. In such a scenario, the 3D position of the drill tip can be determined relative to either of the two objects if the drill is visible in all images, albeit in different orientations, and its tip (also visible in all images) remains on the same point (e.g., a point on the bone surface). This means that in the first image the device may be positioned at an arbitrary point, and in the second image (obtained from a different viewing direction) the device may be oriented differently (e.g., by aiming at an approximate target trajectory), but the tip of the device remains in the same position. Based on the location of the target object / structure (e.g., a nail / nail hole), both images may be aligned, which may enable the determination of the 3D position of a point relative to the target object (the tip of the instrument). This point may then be used to determine the relative 3D position and 3D orientation between the instrument and the target object / structure with sufficient accuracy.

[0102] In other words, a software program product running on the system's processing unit may cause the system to receive a first X-ray image, which is a projection image of at least one object, to classify at least one object, and to determine at least one point in the first X-ray image. The system may then receive a second X-ray image, which is a projection image generated in an imaging direction different from the imaging direction used to generate the first X-ray image. In the second image, at least one object is again classified, and at least one point is determined. Based on the classification of at least one object in the first and second X-ray images, and based on the determination of at least one point in both X-ray images, the two images can be aligned, and the 3D position of the point relative to at least one object can be determined.

[0103] If at least one object contains two objects, and at least one point is one of the two objects, the system may determine the spatial relationship between the two objects, i.e., the 3D orientation and 3D positioning, based on the aligned image.

[0104] Furthermore, the system may determine the deviation of the 3D position and 3D orientation of one of the objects based on the intended spatial relationship of one object to another. For example, one object may be a drill, and it is intended to be positioned parallel to and on a trajectory passing through another object, which may be bone or an implant.

[0105] The above method for aligning two X-ray images may also be useful for, for example, 3D reconstruction and / or localization of an anatomical object into which a known implant has been inserted. Localizing the implant enables image alignment, which then allows for the determination of the 3D position (relative to the implant) of the drill tip located on the bone surface. The points thus determined may serve as anchor points for 3D reconstruction and / or localization of the anatomical object. It may be possible to determine multiple surface points according to this technique, meaning sampling the 3D bone surface at separate points relative to the implant, thereby obtaining a point cloud. Each sample point added to this point cloud may reduce ambiguity in 3D reconstruction and the determination of the 3D position and orientation of the anatomical structure relative to the implant. If the drill angle is in the range of 10–55°, this may also allow for the determination of the 3D position and orientation of the anatomical structure (or implant) relative to the drill. Therefore, even if a deterministic 3D model of the anatomical structure (e.g., CT scan) is available, this procedure can be used to determine the 3D position and orientation. Furthermore, even if there are no known implants in a bone-fixed position, a point sampling method can be used. In such cases, reconstruction and / or localization and / or alignment are carried out directly based on the anatomical structure.

[0106] The following examples illustrate the effects of C-arm width, image detector size, and zoom on 3D alignment. In all of these examples, it is shown that determining the imaging depth is not necessary.

[0107] Effect of C-arm width: Figure 1 shows the left femur (represented as LF) and nail implant along with the mounted aiming device (represented as NAD). It further shows the APX image (represented as 1.AP) and MLX image (represented as 1.ML) as well as their corresponding focal points (represented as 1.FP.AP and 1.FP.ML). The 3D ball approximates the femoral head (represented as FH), and the dashed white circle is its 2D approximate projection in the image (represented as 1.FH.AP and 1.FH.ML). The C-arm has a width of 1000 mm (defined here as the distance between the focal point and the image plane). The cone indicates a portion of the X-ray beam passing through the femoral head. Throughout this application, the inventors follow the convention of referring to images taken in the posterior-anterior direction as "AP" images and images taken in the anterior-posterior direction as "PA" images. Similarly, the inventors refer to images captured in the outward-inward direction as "ML" images, and images captured in the inward-outward direction as "LM" images.

[0108] In Figure 2, the C-arm width was inaccurately estimated as 900 mm instead of the true 1000 mm. Therefore, all objects in the image, including the femoral head (FH), appear smaller than they should be in the X-ray image. Consequently, it appears as if the objects are moving toward the AP image plane (represented as 2.AP) and the ML image plane (represented as 2.ML). The corresponding focal points are represented as 2.FP.AP and 2.FP.ML. The 3D reconstruction of the femoral head (FH) based on the 2D projection of the approximated femoral head (white circles 2.FH.AP and 2.FH.ML) remains unchanged compared to Figure 1. The only parameter that has changed is the apparent imaging depth. However, the imaging depth is irrelevant in this scenario because the relative 3D positions of the femoral head and nails have not changed.

[0109] To show that the only difference between Figure 1 and Figure 2 is the apparent imaging depth, Figure 3 shows both scenarios simultaneously.

[0110] Effect of Zoom: When one of the images is captured with a zoom ratio, objects appear larger than they would without zoom. In Figure 4, the AP image (represented as 4.AP) was captured with a zoom ratio of 1.5. Therefore, all objects in the image, including the femoral head (FH), appear as if they have moved toward the focal point in AP (represented as 4.FP.AP). As previously mentioned, the 3D reconstruction of the femoral head (FH) based on the 2D projection of the approximated femoral head (dashed white circles 4.FH.AP and 4.FH.ML) remains unchanged compared to Figure 1. The only parameter that has changed is the apparent imaging depth. However, the imaging depth is irrelevant in this scenario because the relative 3D positions of the femoral head and nails have not changed. A similar situation holds true when both images have zoom. Figure 5 compares the situation with zoom (similar to Figure 4) and the situation without zoom (similar to Figure 1).

[0111] Effect of X-ray detector size: If the assumed size of the X-ray detector is 12'' instead of the true 9'', the object will appear larger in the image, as if the object had moved towards the focal point in both images. This is shown in Figure 6, where, ·6.AP.9'' refers to AP images taken with a 9'' X-ray detector having the focal point represented as 6.FP.AP.9''. ·6.AP.12'' refers to an AP image taken by a 12'' X-ray detector with the focal point represented as 6.FP.AP.12''. ·6.ML.9'' refers to an ML image taken with a 9'' X-ray detector having the focal point represented as 6.FP.ML.9''. ·6.ML.12'' refers to an ML image taken with a 12'' X-ray detector having the focal point represented as 6.FP.ML.12''.

[0112] This effect is equivalent to the zoom ratio applied to both images. Therefore, the same conclusion can be drawn as with zooming.

[0113] Measurement of the characteristics of classified objects The present invention does not require a priori calibration. If a known object located near (at a similar depth to) the structure being measured is present in the image, the measurement may be made in millimeters. Since the known object has known dimensions, it can be used to calibrate the measurement. This is similar to the procedure proposed by Baumgaertner et al. for determining the TAD value (see Baumgaertner MR, Curtin SL, Lindskog DM, Keggi JM: The value of the tip-apex distance in predicting failure of fixation of peritrochanteric fractures of the hip. J Bone Joint Surg Am. 1995, 77:1058-1064).

[0114] Example 1: A nail has been inserted and AP images are available. The nail is identified and localized. Since the nail is located midway through the diaphysis and therefore at a similar imaging depth to the lateral cortex being depicted in the diaphysis, a known nail geometry can be used for calibration. This makes it possible to provide scaling for determining the distance between the nail axis and the lateral cortex of the diaphysis.

[0115] Example 2: If the imaging depth of object A is known (for example, because object A is sufficiently large or the size of the X-ray detector and the distance between the image plane and the focal point are known), and information exists regarding the difference in imaging depth between object A and object B (for example, based on anatomical knowledge), it may even be possible to calculate the size of a different object (let's call it "object B") at different imaging depths based on Thales' theorem.

[0116] Handling of image distortion in the case of intramedullary nails Generally, there are two methods for dealing with image distortion, and these can be combined. 1. Instead of emphasizing areas in X-ray images that are known to be highly distorted (e.g., image boundaries), emphasis is placed on areas that are less affected by distortion. 2. Determine the distortion and compensate for it.

[0117] Next, these are illustrated in an example of an AP image of the femur containing an inserted nail.

[0118] Reference 1. The following characters are used for labeling in Figure 7. Solid lines represent the outlines of the nails and aiming devices as seen in the distorted X-ray image. White dashed lines represent the hypothetical outlines of the nails and aiming devices as shown in the undistorted image. 7.D: Distal portion of the intramedullary nail 7.C: Center portion of the nail including the hole for the neck screw 7.P: Proximal portion of the intramedullary nail 7.A: Aiming device

[0119] Typically, 7.D is located in a more distorted region of the X-ray image. Furthermore, the precise location of 7.D is not important when predicting the trajectory for a screw inserted through the hole at 7.C. Therefore, in predicting the screw trajectory, the locations of 7.C and 7.P may be given higher weight than 7.D, and the precise weighting may be determined based on their visibility and the reliability of detection. Also, higher weighting for 7.C and 7.P may be reasonable because these regions are closer to the region of interest (screw hole and femoral head). Furthermore, the appearance of 7.C contains information about the rotation of the nail around its axis.

[0120] Reference 2. Distortion in an image may be determined by the following: a) Earlier surgical procedures (allowing training on specific C-arms) b) Pre-surgical calibration: A known object (e.g., a nail, k-wire, etc.) can be placed directly on the image intensifier / X-ray detector at a known distance from the image plane. This can also be used to determine the size of the X-ray detector and the distance between the focal point and the image plane. c) Images acquired earlier (which can be trained by an algorithm during surgery) d) A database with typical distortion effects (e.g., typical pillow effect for typical C-arm positions, Earth's magnetic field). The device may use the knowledge that digital X-ray imaging devices do not produce distortion.

[0121] If such information is available, it may be used to match the virtual projection of the model to the projection in the X-ray image. The distortion may be applied to the entire image or, in particular, to the shape being matched.

[0122] Alternatively and / or additionally, distortion may be explicitly or implicitly determined during the process of matching a virtual projection of an object with a known deterministic 3D model (e.g., a nail) to the appearance of the object in an X-ray projection image. According to one embodiment, this matching may be carried out in accordance with the content described in the article Lavallee S., Szeliski R., Brunie L. (1993) "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps" in Laugier C. (ed.), "Geometric Reasoning for Perception and Action," GRPA 1991, Lecture Notes in Computer Science, Vol. 708, Springer, Berlin, Heidelberg. In one embodiment, the distortion may be described by a suitable mathematical model (e.g., a radial function and / or sigmoid function, as described by Groenschild E., "Correction for geometric image distortion in the X-ray imaging chain: local technique versus global technique", Med Phys., 1999 Dec;26(12):2602-16). The distortion thus modeled may be compensated for by introducing further degrees of freedom into the parameter vector of the paper cited above by Lavallee et al. (1993) when the virtual projection is made to match the projection in the X-ray image.

[0123] Handling of X-ray source and receiver position changes The X-ray imaging device allows for left-right reversal of images, and because the present invention does not use a calibration reference attached to the image detector throughout the surgical procedure, it does not need to detect changes in the position of the X-ray source and receiver, even when the treatment side (left or right bone) is known. The user may be required to provide information on whether the left-right reversal function is activated or not.

[0124] However, even without such information, it is possible to detect the exchange of positions between the X-ray source and receiver. This is because, in X-ray images viewed from imaging directions much smaller than 90°, the instrument or device occupies a large area in terms of imaging depth. Therefore, the part of the instrument or device closer to the imaging plane (receiver) is depicted smaller than the part further away, making it possible to detect the exchange between the X-ray source and receiver.

[0125] Method for determining rotation, horizontal flipping, and vertical flipping of X-ray images This system provides a method for determining rotation, horizontal flipping, and vertical flipping of X-ray images. This can be used to display X-ray images so that, for example, a nail appears as if it were actually positioned in front of the surgeon.

[0126] The following is known about this system based on the surgical procedures performed. • Positioning the patient (e.g., lying on their back) • C-arm position (for example, image intensifier on the inside and X-ray source on the outside) • Which part of the patient's body will be operated on (e.g., left / right leg: this may be known based on previous proximal procedures, user input, or scanning of the nail packaging).

[0127] Detection of left-right reversal (i.e., determining which side is the anterior and which side is the posterior) may be based on determining the orientation of the implant, which may in some cases be supported by determining the imaging direction relative to the anatomical structure (for example, the condyle faces downward in the image when the patient is lying supine). Alternatively, the user may be instructed to orient an instrument (e.g., a drill) in a specific direction, which may then be used to identify whether that direction is anterior or posterior.

[0128] Method for positioning a target object / structure in the field of view of a C-arm from a desired viewing direction. Refer to Figure 16 for the definition of the rotation axis of the C-arm. In this figure, the X-ray source is represented by XR, the rotation axis represented by the letter B is called the vertical axis, the rotation axis represented by the letter D is called the propeller axis, and the rotation axis represented by the letter E is called the C axis. Note that in some C-arm models, axis E may be closer to axis B. The intersection of axis D and the central X-ray beam (labeled XB) is called the center of the "C" of the C-arm. The C-arm may move up and down along the direction indicated by the letter A. The C-arm may move along the direction indicated by the letter C. This terminology is used throughout this application. The distance of the vertical axis from the center of the "C" of the C-arm may differ between C-arms.

[0129] The following describes a method for instructing the user on how to adjust the C-arm so that the target structure appears in the desired position in the X-ray projection image and is visible from the desired imaging direction (for example, the fixing hole must appear circular, and the projection of the nail shaft must pass through the center of the X-ray image). Even if the necessary rotation and translation based on the positioning of the object are precisely determined, it may be important to determine user instructions suitable for repositioning the C-arm. As shown in Figure 16, the C-arm has multiple rotation and translation axes. Furthermore, it is possible to move the C-arm to different positions in the operating room using its wheels. This allows for translation parallel to the floor and rotation around an axis parallel to the vertical axis, which typically has a large distance (more than 1 m) from the vertical axis.

[0130] The numerous available options for moving the C-arm make it difficult for the user to determine which option is best for reaching the desired position (for the desired imaging direction) most quickly or with the least effort. Furthermore, constraints arising from the operating room setup may prevent the user from moving the C-arm to a specific position. Therefore, in some cases, the user may choose to move the patient (or operating table) rather than the C-arm, especially in upper limb surgeries.

[0131] (i) determine the necessary information regarding how to reposition the C-arm and / or the patient; (ii) translate this information into guidance for the user to select from available means of moving the C-arm (by translation along or around the axis of the C-arm, or by moving the C-arm by its wheels) or the patient; and (iii) propose a method for determining the amount of movement required in each case.

[0132] In other words, a method for assisting the adjustment of the imaging direction of a C-arm-based imaging device may include the steps of receiving information on the current imaging direction, receiving information on the target imaging direction, determining a first of several means for the rotation and translation of the X-ray source and detector, and the amount of movement for that first means, and achieving an imaging direction closest to the target imaging direction. Naturally, such a method may be implemented as a software program product that causes the system to perform this method.

[0133] These methods may take into account any constraints imposed by the construction of the C-arm and the operating room setup, and may select the movements that are easiest for the user to perform and require the fewest number of movements.

[0134] These methods may determine the current imaging direction based on the X-ray image. For example, the viewing or imaging direction may be determined based on anatomical structures (e.g., using the patent application filed by Blau on August 23, 2018), and / or optionally taking into account a priori geometric information relating to implants, devices or anatomical structures, and further optionally taking into account the 3D position and 3D orientation of the target structure in a coordinate system spanning the image plane, and further optionally taking into account a typical operating room setup for the current surgical procedure (the patient's position on the operating table, the position of the C-arm relative to the operating table, the position of any object that can prevent the C-arm from moving to a specific position relative to the operating table / patient), and further optionally taking into account a typical axis of rotation of the C-arm to locate the target structure (or object).

[0135] For example, for a femoral nailing procedure, it may be assumed that the typical patient position relative to the C-arm is known, i.e., the patient is lying supine, the treatment side is known, and the image receiver between the patient's legs is known. Additionally or alternatively, the user may select an operating room setup from a set of provided options. These may be based on information gathered by the system from, for example, scans of implant packages, previously learned surgical procedures, such as the patient's position on the operating table, where the implants are used, and / or where the C-arm machine is positioned relative to the operating table (e.g., between the patient's legs).

[0136] Translating calculated deviations from the desired position / orientation into instructions for adjusting the C-arm can be important, as these instructions should be as easy to execute as possible. Instructions that do not require moving the C-arm by its wheels may be preferable, as moving the C-arm by its wheels is not always precise and can be more difficult in an operating room setting. Generally, it may be preferable to keep the number of instructions to a minimum. A neural network may assist with this entire procedure.

[0137] For example, if a large rotation around the vertical axis is required, this must be done by moving the entire C-arm by its wheels, because this may allow isocentric rotation around the target structure (keeping the target structure close to the central C-arm beam). If the required rotation is around an axis parallel to the vertical axis of the C-arm and the rotation is relatively small, the vertical axis of the C-arm must be used. If the desired axis of rotation is far from the vertical axis of rotation of the C-arm, such a rotation will include a relatively large translation component, but it should be kept in mind that any such translation can be compensated for when determining any potentially required translation. As explained above, a rough determination of the imaging depth may be possible (to a few centimeters). Since the distance of the C-axis from the target structure is roughly known, this may be sufficient to calculate the translation in the AP direction resulting from rotation around the C-axis, for example. The offset between the C-axis and the central X-ray beam may also be calculated from the rotation around the C-axis. Furthermore, due to the fact that fluoroscopic projection is applied, any translation of the C-arm automatically includes a rotation component.

[0138] Making rough assumptions about the 3D position and orientation of the C-arm axis relative to the target object / structure may be sufficient for the first iteration of the positioning procedure, sometimes allowing a position close enough to the desired location. For greater accuracy, or to reach a sufficiently accurate position in the fewest number of iterations, the system may use information collected from previous iterations in subsequent iterations, thereby enabling a more precise determination of the 3D position and orientation of the axis relative to the target object / structure. For example, if translation along or rotation around two or more C-arm axes is required to reach the desired position of the C-arm, the system may instruct the user to move or rotate the C-arm around one axis (e.g., around the C-axis). This allows approaching the desired position while simultaneously determining the 3D position and orientation of the C-axis relative to the target object, and may also determine the offset between the C-axis and the center of the "C" of the C-arm.

[0139] Alternatively, depending on the available a priori information, the 3D position and orientation of the C-arm axis relative to the target object / structure may be determined by (i) moving the C-arm along two axes perpendicular to each other (if these axes do not intersect, one of the parallel axes must be perpendicular to the other), (ii) rotating the C-arm along two axes perpendicular to each other (if these axes do not intersect, one of the parallel axes must be perpendicular to the other), or (iii) moving the C-arm along one axis and then rotating the C-arm around another axis parallel to the axis of movement (for example, forward translation combined with rotation around a vertical axis, or rotation around the C-axis combined with rotation around a propeller axis, or proximal translation combined with rearward translation).

[0140] For example, for lateral distal fixation of an antegrade femoral nail (with the patient positioned supine and the C-arm positioned between the patient's legs), anterior translation of the "C" may, on the one hand, cause rotation of the viewing direction around the nail axis, thereby enabling a more accurate calculation of the 3D position of the target object relative to the C-axis in order to more accurately determine the imaging depth. On the other hand, the system may determine the rotation of the nail around its axis relative to the operating room floor. At a later point, when calculating guidance instructions for anterior / posterior translation, the system may calculate the required translation by using a simple trigonometric relationship, taking into account the effect of the rotation of the nail around its axis relative to the operating room floor, as determined above, on the required translation along the vertical axis. This information may also be used when calculating other translation instructions provided by the system, for example, when calculating the required rotation around the vertical axis, to more accurately determine the distance between the object and the vertical axis (because the distance of the axis perpendicular to the C-axis can be determined more accurately). Therefore, since the distance between the central X-ray beam and the C-axis is already determined, the effects of translation in the proximal / distal directions caused by rotation around the vertical axis can be taken into more accurate consideration.

[0141] In practice, in the first iterative step, it may be sufficient to determine only one or two means of movement or rotation to approach the final desired position. By observing these one or two movements / rotations, the system can obtain sufficient information about the 3D position and 3D orientation of the axis relative to the target object / structure so that in the second iterative step, it can determine and provide to the user with sufficient accuracy all the remaining necessary steps to reach the final desired position. As described below in the section "Examples for Potential Processing Workflows for Distal Immobilization Procedures," the system does not require very precise positioning of the C-arm, and, for example, a perfectly circular projection of a circular hole may not be required.

[0142] If the user moves the patient rather than the C-arm in response to system instructions, checking the image background may help detect such movement and prevent inaccurate determination of the movement / rotation axis. The “image background” in the preceding paragraph may be objects that do not move with the patient and are not entirely radiopaque (e.g., part of the operating table). Simple image difference analysis may be used to determine whether or not the C-arm has moved. In this context, it may be required that there is no rotation of the digital image in the C-arm between the C-arm movement and image acquisition.

[0143] Based on the image of the target structure and the initial positioning of the target structure within the viewing direction, the system may determine which movement to initiate and / or proceed with.

[0144] Instructions may be given on how to rotate the C-arm around the C-axis and how to move the C-arm up and down. If the target structure is not positioned at the center "C" of the C-arm, translation instructions may be given on how to recover the previous position in the X-ray image or how to reach the desired position. Instructions may be given on how to rotate the C-arm around its vertical axis. This may take into account the typical geometric shape of a C-arm (the distance between the vertical axis and the center of "C" of the C-arm), and the system may learn the geometric shape of a particular C-arm from previous rotations. If the target structure does not appear at the desired position in the X-ray image or no longer appears at the desired position after rotation, instructions may be given on how to translate the C-arm. If the position of the target structure is already correct in the X-ray image and remains correct after rotation (for example, if the C-arm is moved by its wheels so that the target structure remains at the center of "C" of the C-arm), no translation instructions are given.

[0145] This procedure may also be applied to other axes of the C-arm, such as the propeller axis.

[0146] Regarding the optimized adjustment of the C-arm, the following general aspects can be summarized.

[0147] Initially, it is intended that only one means be used for the translation or rotation of the C-arm device in order to adjust the position and orientation of the imaging direction as close as possible, or to the target imaging direction, i.e., the optimal direction, with at least sufficient accuracy.

[0148] If necessary, it may be suggested to use a second method to further adjust the imaging direction. Further means may be used to further improve accuracy.

[0149] The current imaging direction, i.e., the starting point for adjustment, may be determined based on the positioning of an object or structure in the X-ray image generated using the current imaging direction. Naturally, the object or structure may be a substructure, and may be an anatomical structure, implant or instrument / device, or a combination of such objects.

[0150] The target imaging direction may be identified based on known geometric aspects, 3D position, and 3D orientation. Such geometric aspects may be known from the preoperative plan and are available from a database.

[0151] Furthermore, the position and orientation of objects may generally be known relative to the C-arc of a C-arm-based imaging device. In a typical operating room setting, anatomical structures may be positioned relative to the C-arm device in a known manner that allows for prediction of the imaging direction. The system may, in some cases, provide information for user confirmation.

[0152] Alternatively or additionally, the system may learn how users tend to utilize C-arc translation and rotation, and may take that into consideration. For example, the system may calculate the rotation axis or translation of the C-arm device relative to an object from two images, where the imaging direction is rotated or translated around the rotation axis during image generation. In particular, the system may learn whether users tend to move the C-arm device by approximately the amount indicated at that time, and may take that into consideration when giving further instructions to move the C-arm device.

[0153] Example of a potential processing workflow for distal fixation procedures The following distal fixation procedures describe long antegrade nails. Nevertheless, they may also be applied to retrograde nails fixed proximal to the occlusion. The following fixation procedures are provided for holes whose axis is approximately in the ML direction, but they may also be applied to holes whose axis is in a different direction, for example, in the AP direction.

[0154] For such a procedure, one might assume that a complete 3D model of the target object (i.e., the nail) is available. Nevertheless, even if only an incomplete or partial 3D model is available, the procedure may still be successful if only approximate information about the nail's shape is known (e.g., a cylindrical object with a cylindrical fixing hole and a diameter that slightly decreases towards the tip).

[0155] In the following, the fixing hole will be referred to as the "target structure." In principle, it is sufficient to know only the relative 3D position and 3D orientation between the instrument (e.g., an implant such as a drill, shim, sleeve, or even a screw) and the target structure. Therefore, in the following, we will consider both the target object (nail) and the target structure (fixing hole). In this explanation, we will assume that a circular hole is fixed first.

[0156] 1. The user acquires an X-ray image of the nail in the approximate ML direction.

[0157] 2. The system may determine the imaging direction of the target structure (for example, by detecting or locating the target object (in 2D)). The system may search for or determine a target trajectory (or optionally a target plane) relative to the target object or structure. The system may then determine and inform the user how to adjust the C-arm to reach the desired imaging direction of the target trajectory. In many cases, the desired imaging direction is aligned with the target trajectory, in which case the distal fixation hole is depicted as a circle. Figure 20 is an X-ray image in which a nail labeled 20.N is visible along with a non-circular fixation hole labeled 20.H. Thus, the imaging direction is not the intended ML imaging direction. Figure 21 is an X-ray image in which a nail labeled 21.N is visible from the true ML imaging direction, as is evident from the circular fixation hole labeled 21.H.

[0158] It is desirable that the nail shaft passes through the center of the image and that the fixing holes are near the center of the image. This is desirable if there are further holes to which the C-arm can be fixed without readjusting. Therefore, ideally, these holes should be on the central X-ray beam. The C-arm adjustment may be repeated and completed with a new X-ray image that satisfies the above requirements.

[0159] 3. At this point, the system may highlight the center of the fixed hole that the instrument must aim at in the X-ray image. This highlighted (target) point is on the target trajectory and is the center of the circle in the scenario described. The system then detects the tip of the instrument in 2D and calculates the required movement of the tip to reach the target point. The system may support the user in an iterative process (each iteration consisting of acquiring a new X-ray image and repositioning the instrument) to reach the target point. Figure 22 shows X-ray images of a nail (22.N) and a drill (22.D) with an incorrectly positioned drill tip. Figure 23 shows X-ray images of a nail (23.N) and a drill (23.D) with an correctly positioned drill tip.

[0160] 4. Once the tip of the instrument (in this case, a scalpel) is positioned at the point highlighted in the X-ray image, the surgeon may make an incision and insert the drill (with a soft tissue protection sleeve if applicable), and repeat step 3. The user may then decide to align the drill with the target trajectory without moving the tip of the drill (as in the conventional procedure).

[0161] 5. The C-arm is rotated, for example, 25° around the C-axis, and a new X-ray image is acquired. The system may reposition the target object (or, in some cases, only the target structure). Based on the a priori knowledge that the tip of the instrument is on the target trajectory (remaining in a known 3D position and 3D orientation relative to the target object / structure), the relative 3D position and 3D orientation between the drill and the target object / structure may be determined. Even if the tip of the drill in the distal-proximal direction is no longer precisely on the target trajectory, the system can calculate the corresponding deviation. This is because, when the C-arm is rotated around the C-axis, it may be sufficient to have only the a priori information that the tip of the drill is located in a plane spanning the target trajectory and the nail axis.

[0162] At this point, the system may calculate the deviation from the target trajectory and communicate this to the user, for example, by displaying the required angular corrections in the proximal-distal and anterior-posterior directions. If necessary, the system may also instruct the surgeon on how to adjust the tip position of the instrument in the proximal-distal direction. Furthermore, the system may calculate the penetration depth, in this case for example, the distance between the drill tip and the nail, and communicate this to the user. Step 5 may be repeated by acquiring new X-ray images, providing information / instructions to the user, and readjusting the instrument.

[0163] Information can be conveyed to the user on a display and / or audibly. The advantage of auditory information may be that the surgeon does not need to take their eyes off the drill, and therefore can achieve the correct direction for drilling with fewer repetitions.

[0164] 6. Step 5 may also be performed during drilling to adjust the direction of drilling and / or to obtain information on how far to drill further (this is unrelated to the fact that in a typical distal fixation situation, drilling continues to the next cortex after hitting the nail hole).

[0165] Fixation to further holes (e.g., elongated oval holes):

[0166] To save time and minimize X-ray exposure, the following procedure is used to fix the device in a further hole (hereafter assumed to be an elongated elliptical hole) after fixing it in the first hole as described above.

[0167] Assuming the target trajectory for the elongated elliptical hole lies in the same plane as the first hole, the C-arm is rotated back to its original position (where it was before step 5) that shows the first hole (with or without a screw) as a circle. Thus, the nail axis again passes through the center of the X-ray image, and the elongated elliptical hole is close to the center of the image. No readjustment of the C-arm is necessary unless correction of the rotation around the C-axis is required because it does not reach the original angle accurately enough. Thus, the elongated elliptical hole appears with its maximum diameter in the AP direction but is compressed in the vertical direction. On the other hand, if the target trajectory for the elongated elliptical hole lies in the same plane as the first hole, the system may support the necessary readjustment of the C-arm as described above.

[0168] Since the system knows the approximate distance between the bone surface and the nail at the medial-lateral position from the fixation to the first hole, this value (and optionally a statistical model of the bone) may be used to correct the target position of the drill tip (see step 3 above). Therefore, to hit the centrally located elongated elliptical hole (both in the AP direction and distal-proximal direction), the target point in the 2DX image does not appear in the center but has shifted in the distal-proximal direction.

[0169] In Figure 17, the circular nail hole, represented by 17.RH, is perfectly circular in 2D, i.e., on X-ray. The tip of the mouth opener (represented by 17.OT) is at a specific distance from the center of the elliptical nail hole due to its position on the bone surface; therefore, it is not at the 2D center of the elliptical nail hole, but its tip is perfectly aligned with the central trajectory of the elliptical nail hole in 3D. The two black arrows, represented by 17.C1 and 17.C2, indicate chamfers, which appear in perspective on both sides of the elliptical nail hole at different sizes.

[0170] After aligning the instrument (approximately) to the target trajectory, rotating the C-arm around the C-axis, and acquiring another X-ray image, any inaccuracy in the distal-proximal positioning of the instrument tip is detected and calculated, so that any potential inaccuracies in the distal-proximal direction are not a problem. If necessary, instructions may then be given to correct the instrument tip position in the distal-proximal direction. As discussed above in step 5, it may be sufficient for the instrument tip to be in a plane spanning the target trajectory and the nail axis. The remainder of the procedure follows the steps for the first hole.

[0171] If the elongated elliptical hole is tilted relative to the nail shaft, this tilt may be compensated for when rotating the C-axis by the potential fine adjustments supported by this system.

[0172] The entire considerations apply to circular holes as well. Furthermore, it is possible to fix them into even more holes using the same procedure.

[0173] As discussed above, the viewing direction to the hole does not need to be perfect (i.e., for a circular hole, for example, having the maximum projected hole width and height that is perfectly circular) in order to position the drill tip on the target trajectory. Depending on how stringent the available a priori information and the requirements for angle determination are (the less stringent these are, the smaller the distance drilled between the bone surface and the nail), the viewing direction to the hole may deviate more or less from the target trajectory. For example, if an AP image is obtained, the distance between the outer bone surface and the nail along the target trajectory may be roughly determined (or simply estimated). Based on this information and the outer X-ray image, a point in the 2D X-ray image where the drill tip must be positioned to be on the target trajectory may be calculated (and then displayed) based on the oblique viewing angle, as discussed above. This point does not need to be perfectly on the drill trajectory. Rather, a new target trajectory may be calculated using a deviation from this point (in 2D spatial coordinates) that is determined in the 2D X-ray image based on the estimated or previously determined distance between the bone surface and the nail along the fixed trajectory. Next, a new target trajectory may be used in the following image to direct the drill. This may make it even easier for the surgeon to position the drill tip at the correct point, as it is not necessary to make a perfect hit with the drill tip. Furthermore, it may also allow for greater accuracy when directing the drill to hit the target hole.

[0174] Assuming that the sufficiently precise positions of points on a second object relative to the geometric aspects of a first object are known, image alignment may allow for the 3D reconstruction and / or determination of the relative 3D position and 3D orientation.

[0175] Furthermore, deviations from the original target trajectory may be resolved by the following method: In the first image, the instrument may be positioned on an arbitrary point (which may or may not be on the target object), and in the second image (obtained from a different viewing direction), the inclination of the instrument may have changed (for example, by aiming at the target trajectory), but the tip of the instrument remains in the same position. Based on the positioning of the target object / structure, both images may be aligned, which may enable the determination of the 3D position of the point relative to the target object. This point may then be used to determine the relative 3D position and 3D orientation between the instrument and the target object / structure with sufficient accuracy.

[0176] In step 5 or 6 above, if the tip of the instrument is blocked by a target object (e.g., a steel nail) and therefore the positioning is not sufficiently accurate, the system may provide instructions on how to make the tip of the instrument visible to the system. This may be done by the system calculating and instructing how the C-arm needs to be repositioned (e.g., rotation around the C-axis instead of rotation around an axis perpendicular to the nail axis). Also, if the instrument material absorbs significantly more X-rays than the target object (e.g., a steel instrument, a titanium nail), this may be achieved, for example, by increasing the voltage and / or current, or by selecting a different C-arm program setting.

[0177] Alternatively, the system may match a statistical 3D model of the bone, thereby determining the 3D position of the nail relative to the 3D position of the bone, and thus enabling the determination of the required fixing screw length in 3D.

[0178] In the case of a typical steel nail where all fixing holes face the same direction, the fixing holes may become invisible in the X-ray image when rotated more than 30-35° (similar to step 5), meaning that steel nails may not be locatable when rotated greater than 30-35°. On the other hand, instruments or implants made of titanium, for example, absorb considerably less radiation than instruments or implants made of steel, for example. Therefore, in the case of titanium nails, the inclined holes create a density gradient at the boundary of the holes. This is shown in Figure 18 for a 25° rotation away from the fixed plane and in Figure 19 for a 45° rotation away from the fixed plane. This effect means that it may be possible to locate inclined titanium nails over a much larger angular range compared to steel nails. Another beneficial effect of titanium nails is that they can make drills, which are typically made of steel, visible relative to the nail. This may improve the accuracy of locating the drill, for example, when the drill tip is close to the nail during drilling. It may also be possible to rotate the C-arm around a different axis, such as the propeller axis, where typically the X-rays show the tip superimposed on a nail.

[0179] If the C-arm is rotated around a vertical axis instead of the C-axis in step 5, the system requires that its normal is the nail axis and that the drill tip is in the plane containing the target trajectory. In this case, the deviation of the drill tip from the target trajectory in the AP direction may be calculated.

[0180] Therefore, an alternative to the above workflow may be to acquire X-ray images from both viewing directions obtained by rotating each arm around the C-axis so that they are away from the fixed plane, and from viewing directions obtained by rotating each arm around the vertical axis. In this case, a priori information regarding the position of the drill tip relative to the target trajectory is not required. Therefore, it is not necessary to position the C-arm in the true ML direction.

[0181] Example of a potential processing workflow for sacroiliac joint (SI) or pedicle screw placement Furthermore, the target object and target structure may be anatomical. An example of an anatomical target structure is the pedicle. It may be sufficient to achieve the necessary accuracy for the target structure and therefore the target trajectory by considering 3D reconstruction relative to the instrument and its relative 3D position and orientation.

[0182] The procedure is similar to distal fixation, except for the localization of the anatomical target structure, and may or may not be performed using a deterministic 3D model. The deterministic 3D model may be obtained either preoperatively (e.g., preoperative CT scan) or intraoperatively (e.g., intraoperative CT scan or O-arm). If a deterministic 3D model is not available, a statistical 3D model (e.g., statistical shape or appearance model) may be used for 3D reconstruction, as previously discussed in the section "3D Reconstruction and Localization of Anatomical Objects Based on a Single X-ray Image."

[0183] In this procedure, the instrument may not be fixed to the target point of the anatomical structure, but rather held by hand and then roughly aligned to the target trajectory. The C-arm may be rotated, for example, 25° around the C-axis without acquiring a new X-ray image. After the iterative process described above, a new X-ray image may be acquired, and the system may calculate the relative 3D position and 3D orientation between the instrument and the target structure / object, taking into account that the tip of the instrument is on the target trajectory. This assumes that the viewing angle to the instrument is within a range that allows for a sufficiently accurate determination of the 3D position and 3D orientation between both objects. If only a statistical model of the anatomical structure is available, this step includes a 3D reconstruction of the target structure / object.

[0184] In the next step, the system may calculate the deviation between the instrument axis and the target trajectory with respect to angular deviation, and optionally the deviation at the tip position in a direction parallel to the rotation axis of the C-arm used for rotation between both images. The system may also calculate the drill penetration depth. The system may provide this data (e.g., in the form of two angular values, the translation required for the instrument tip position, and the remaining insertion depth) to the user. The user may then readjust the instrument accordingly, obtain a new X-ray, and / or drill a hole in the instrument / insert it. Since the a priori information and lateral constraints remain unchanged, the procedure is also successful if drilling has already begun and the instrument has already penetrated the anatomical structure.

[0185] Example of a potential processing workflow for determining the 3D position and orientation between equipment and anatomical structures based on image registration for improved accuracy. The tip of an instrument (e.g., a drill, k-wire, or Jamsidi needle, or even an implant such as a screw) may be positioned on a specific anatomical reference point by adjusting (translation and rotation) the bearing of the C-arm to align with a specific part of an anatomical structure (e.g., a narrow pathway such as a pedicle). This process may be supported by the system by displaying the reference point in the acquired 2D X-ray image, or alternatively, the system may be used to improve the accuracy of the surgeon's identification of the reference point. The C-arm is then rotated, for example, 20-30° around the C-axis (or a comparable rotation around the propeller axis) while leaving the instrument in place, and another X-ray image is acquired. The fact that the instrument is in contact with the surface of the anatomical object at the reference point may be used to reduce or even eliminate ambiguity caused by the ambiguous positioning of the instrument. The movement of the C-arm relative to the previous image may then be determined, thus determining the viewing direction to the anatomical object with high accuracy. This requires that the instrument does not move between X-ray images, which can make it easier to fix the instrument to the anatomical structure, and this can be done not only with a drill but also with a Jamsidi needle or k-wire.

[0186] procedure: 1. Based on a preoperative CT scan, reference points and reference trajectories (i.e., the intended drilling or insertion trajectory) as well as the target endpoint may be planned before the surgical procedure. This may include planning along the intended imaging direction of the C-arm, e.g., true lateral or true AP, or along the pedicle or other easily recognizable landmarks. This step 1 may also be performed automatically by the system and / or in interaction with the user during surgery (using an intraoperative 3D imaging device).

[0187] 2. During surgical procedures, improved positioning accuracy may be achieved by using either a predefined or online-calculated C-arm imaging direction. The system may provide instructions to the user to help achieve the required C-arm bearing by detecting the relative position of specific anatomical features, such as edges or points (see, for example, the patent application filed by Blau on August 23, 2018). The system may display a reference point in the X-ray image based on matching the entire structure or object from the CT scan to the X-ray image. The surgeon then positions the tip of an instrument (e.g., a drill, Jamsidi needle or k-wire or even an implant such as a screw) on this reference point, which may also be supported by the system by detecting the tip of the instrument in the 2D X-ray image. If necessary, the instrument is held intentionally at an angle that does not obstruct the view of the instrument (or power tool) and the surgeon's hand.

[0188] 3. Next, align the instruments in approximately the desired direction.

[0189] 4. If possible, secure the instrument to the anatomical structure using the designated markings on the instrument so that the exact penetration depth can be determined.

[0190] 5. Acquire another X-ray image from the same imaging direction. At this point, the position of the instrument tip is limited because it is approximately on the drilling or insertion trajectory. When the instrument penetration depth is precisely known, there is less ambiguity than when the instrument penetration depth is unknown. The system may verify (for example, by image difference analysis) whether the anatomical structures shown in the X-ray images remain unchanged. Alternatively, in this step, it may be sufficient for the user to simply indicate which plane (relative to the anatomical structure) the instrument tip is in. If step 5 cannot be performed because the instrument or the surgeon's hand obstructs the view, perform steps 4 and 5 without aligning the instrument in step 3.

[0191] 6. Move or rotate the C-arm to a different position. Based on the positioning and fixed instrument, the movement of the C-arm relative to the previous image may be determined. Together with the positioning anatomical structure, this allows for the determination of the relative 3D position and 3D orientation between the instrument and the anatomical structure. The amounts determined together may be optimized.

[0192] 7. If the instrument has such a small diameter that it can be positioned with sufficient precision for only a specific angle, the instrument must be visible at a suitable angle (e.g., within the range of 10-55°) in all acquired X-ray images to be aligned. Furthermore, the anatomical structure may be positioned more precisely for a specific imaging direction. Therefore, the accuracy for determining the relative 3D position and 3D orientation in step 7 may be increased by selecting a specific imaging direction to the anatomical structure. However, since such special imaging directions are typically true AP and true ML, this means that the angle between the two X-ray images is close to 90°. Furthermore, when fixing the instrument in step 4, it must already be observed that the tip of the instrument is visible at a suitable angle (e.g., within the range of 10-55°) in all acquired and aligned X-ray images. Therefore, in such cases, the appropriate angle for fixing the instrument is midway between the true AP viewing direction and the true ML viewing direction, i.e., approximately 45°.

[0193] 8. Once an X-ray image has been acquired and the above conditions are met, the system may calculate the deviation between the instrument axis and the reference trajectory and provide this to the user (for example, by displaying two angle values). The user may then move the instrument back to its original reference point and realign it with the reference trajectory. The system may assist the user in finding the original reference point.

[0194] Reaching the correct reference trajectory may require a repeated process of acquiring further X-ray images. After acquiring new X-ray images, the system may then verify (e.g., by image difference analysis) whether the anatomical structure is still shown in the X-ray images in the same orientation and position. In that case, the relative 3D position and 3D orientation between the instrument and the anatomical structure may be recalculated based on the a priori information that the tip of the instrument is on the reference trajectory. Furthermore, the system may also calculate and communicate to the user the penetration depth, in this case, for example, the distance between the tip of the instrument and the target endpoint. During instrument insertion, further X-ray images may be acquired, and the above steps may be repeated.

[0195] Furthermore, if it is not feasible or desirable to ensure that the instrument is visible within a suitable angular range (e.g., 10–55°) in all acquired X-rays, the power tool holding the instrument (e.g., a power drill holding the drill) may be removed (as is evident when using k-wire or Jamsidi needles). If the entire instrument (tip and base) is visible in the X-ray image, the length of the instrument's projection in the X-ray image may be determined to allow for sufficiently accurate positioning of the instrument. In such cases, the instrument may even be visible at an angle close to 90°. This allows the instrument to be initially fixed at an approximately correct angle (eliminating the need to initially fix the instrument at an inaccurate angle; see step 7), thus reducing the number of repetitions required in step 8.

[0196] For the insertion of pedicle screws, it may be possible to identify the pedicle entry point in an AP (Augmented Position) image where the pedicle axis is tilted, for example, at 10–45° with respect to the viewing direction. In such imaging directions, the drill does not obstruct the view, the tip of the instrument is positioned on the entry point of the anatomical structure, and it is not necessary to acquire a second image from another direction after positioning the angle of the instrument axis with the target trajectory. Since this procedure often requires a k-wire or Jamsidi needle, the instrument may be fixed to the bone with its axis already aligned with the desired target trajectory.

[0197] If necessary, the user may then acquire one or more X-ray images from other viewing angles, which the system may use to perform the image alignment described above to improve accuracy. If further necessary, the drilling angle may be further optimized based on additional information (sometimes without retracting the drill), and drilling may be continued.

[0198] The fact that in this first AP diagram the other pedicles have a left-right inversion of the inclination relative to the first pedicle (see Figure 15 showing two Jamsidi needles 15.JN and 15.JN2) allows for more robust image alignment by repeating the above procedure for the other pedicles of the same vertebra and utilizing the Jamsidi needle already inserted in the first pedicle (by using the projection in its X-ray image).

[0199] Flowcharts in Figures 24-26 Figure 24 shows a general flowchart that covers all the procedures outlined in the sections “Example of a potential processing workflow for distal fixation procedures,” “Example of a potential processing workflow for sacroiliac joint (SI) or pedicle screw placement,” and “Example of a potential processing workflow for determining 3D position and orientation between the device and anatomical structure based on image registration for improved accuracy.” There are two possible executions: a rapid execution shown in Figure 25, applicable to the procedures outlined in the sections “Example of a potential processing workflow for distal fixation procedures” and “Example of a potential processing workflow for sacroiliac joint (SI) or pedicle screw placement,” and a more accurate execution shown in Figure 26, applicable to the procedure outlined in the section “Example of a potential processing workflow for determining 3D position and orientation between the device and anatomical structure based on image registration for improved accuracy.”

[0200] Those skilled in the art will understand that it is not necessary to perform all steps, and that in practice, further steps not mentioned herein may be performed depending on the specific circumstances of the application of the teachings provided herein.

[0201] The steps shown in Figures 24, 25, and 26 are as follows.

[0202] S10: Generate and load a 3D model. S11: Preoperative plan (optional). S12: Load the entire 3D model. S13: Intraoperative automatic determination of one or more target trajectories / planes and, where applicable, target points (e.g., in the case of anatomical structures).

[0203] S20: Supports C-arm adjustment. S21: Acquire an X-ray image. S22: For example, by providing the rotation angle (including direction) around the C-axis, rotation around the propeller axis, etc., it supports reaching a target object, such as a specific viewing direction to a circular hole (potentially supported by nail positioning), or a true AP / ML diagram to an anatomical structure (potentially supported by DNN). S23: If ambiguity arises, the system will provide only the rotation angle value, without including the direction. S24: If the viewing direction is not close enough to the desired viewing direction, the user follows the system instructions and continues from S21. If a corresponding 3D model of the anatomical structure (e.g., CT scan, i.e., deterministic) is available, in which case the target trajectory can be obtained from the current viewing direction and the 3D model, and since it is known that the tip of the mouth opener will be positioned on the anatomical structure, the desired viewing direction may differ from the target trajectory. Example: Distal fixation where the mouth opener is positioned on the femur and a 3D model of the femur is available. After locating the target object (S33), the system calculates the intersection of the target trajectory of the nail model and the surface of the femur model. Using the 3D model of the anatomical structure, the system gives adjustment instructions for the tip of the mouth opener (S37).

[0204] S30: Supports the positioning of the opening device. S31: Positioning of the opening device. S32: Acquire X-ray images. S33: Identify the location of the target object / structure. S34: If the target trajectory is well aligned with the viewing direction, the target point is directly visible in the 2DX image (one DoF for an undefined instrument tip position). Proceed to S36. S35: The system displays the intersection of the 3D surface of the anatomical structure (including the distal fixation site) and the target trajectory (all DoF for the defined instrument tip position) superimposed on the 2D X-ray image. S36: 2D matching of opening devices. S37: The system provides instructions to the user to assist with adjusting the tip of the mouth opener. S38: If the position is not reached with sufficient accuracy, the user follows the system instructions and continues to S32.

[0205] S40: Determine the 3D position and orientation between the aperture device and the target object in order to align the aperture device with the target trajectory. S41e: Fixing the required opening device. S411e: Fixing the aperture device. When S44 is intended to be applied, the system provides support for fixing the aperture device to the target object at an angle that ensures the angle between the aperture device and all special viewing directions is less than 65°. Two angle values ​​are provided. If the user is taking a different image from the same viewing direction, the system will check the fixing angle of the aperture device. S412e: Acquire an X-ray image that does not include changes in the relative position between the C-arm and the anatomical structure. S413e: Image difference analysis to determine the penetration depth of an opening instrument. S414e: Determine the 3D position and orientation of the mouth opening device relative to the anatomical structure. S41q: The opening device is not fixed in place. The user does not support aiming at the trajectory. S42: The system calculates and displays the adjustment value for the C-arm rotation to achieve a 25° angle between the viewing direction and the target trajectory. In case of ambiguity, the system provides only the rotation angle value without including the direction. S43: The user positions the C-arm according to the displayed adjustment values ​​and acquires an X-ray image. If the viewing direction is not close enough to the desired viewing direction from S42, proceed to S42. S44: Calculate the 3D position and 3D orientation between the target object and the mouthpiece for final mouthpiece adjustment instructions. S441e: For iterative optimization, 3D positioning of the aperture device and calculation of a transformation matrix between the 3D position and 3D orientation of the aperture device between the current image and the previous special viewing direction. S442e: (i) the above transformation matrix (S441e), (ii) all previous 3D orientations and 3D positions of the anatomical structure along with the current position of the mouth opener, and (iii) potentially improved 3D orientations and 3D positions of the anatomical structure based on the current 3D orientations and 3D positions of the anatomical structure (either iteratively or co-optimized). S441q: For iterative optimization, locate the target object. Based on the locate of the target object and the a priori information that the tip of the aperture is positioned on the target trajectory / plane, determine the 3D position and 3D orientation between the aperture and the target object. Proceed to 443. S442q: Joint optimization of the 3D orientation and 3D position of a target object relative to an opening device. S443: For example, if distal-proximal deviation of the nail can be confirmed and corrected, the a priori information (the position of the tip of the mouth opener relative to the target object) is confirmed and corrected. S444e: If further improvements are needed in the accuracy of the 3D position and orientation between the anatomical structure and the mouth opening instrument, the system calculates and displays adjustment values ​​for C-arm rotation to reach further specific viewing directions. S445e: The user positions the C-arm according to the displayed adjustment values ​​and acquires an X-ray image. If the viewing direction is not close enough to the desired viewing direction, proceed to S44. S45: The user moves the mouthpiece according to the adjustment values ​​provided to align the mouthpiece with the target trajectory. S451: Since the 3D model of the target object provides the target trajectory, the system provides angles (two angles including direction) for adjusting the orientation of the aperture instrument to align the instrument with the target trajectory, obtained from the 3D position and 3D orientation of the instrument determined above relative to the target object. S452e: If the aperture is still fixed in the first position (S41e), the user retracts the aperture until its tip is on the target trajectory, then aligns the aperture based on the system output and acquires an X-ray image (or uses a second aperture to aim at the target trajectory; in this case, proceed to S44). S453e: The system compares the images (e.g., by image difference analysis). If the images are locally close enough (with respect to the target object), proceed to S44. S454: If the alignment between the aperture and the target trajectory is not close enough, the user positions the aperture based on the system output, acquires an X-ray image, and continues with S44. S455: If the remaining alignment instructions provide sufficiently small values, the system displays information on how far away the drill is. The user may align the retractor based on the alignment instructions and decide whether or not to take another X-ray image. S46: Drilling holes. S461: The user drills a hole. S462: Whenever the user wishes to confirm the drill direction or drill depth, the user obtains a new X-ray image and continues with S44.

[0206] Furthermore, if the system can provide instructions on how to adjust the orientation (and, if applicable, position) of an instrument already present in the first X-ray image, these can be used to achieve a temporary alignment of the instrument with the target trajectory. After this alignment, if the instrument's tilt is already within the required angular range and neither the power tool nor the surgeon's hand obstructs the view, another image from a different imaging direction may not be required, and if necessary, the applied instructions (regarding the instrument's orientation and, if applicable, position) may be confirmed by an X-ray image from the same imaging direction. Another X-ray image may not be required if (i) no correction is needed or only a very small correction is needed, or (ii) a device is used that ensures sufficiently accurate application of the given instructions. Such a device may be manual or robotic.

[0207] As described in the section "Method for positioning a target object / structure in the C-arm's field of view from a desired viewing direction," perfect alignment of the C-arm in the direction of the target trajectory may not be required, especially when the instrument is positioned on the target object and the target point is identifiable in the current X-ray. In fact, all the information necessary to calculate the relative 3D position and 3D orientation may already be available in the first X-ray image. Therefore, depending on the setup, the first X-ray image may be sufficient to perform the entire distal fixation procedure. An example of such a setup is the use of a robot that holds the instrument at a given point, taking into account the required inclination. If it is already possible to identify both target trajectories and the required starting point in the first acquired X-ray image, and to determine the 3D position and 3D orientation between the instrument and the target object, the robot may translate and rotate the instrument as needed and drill a hole. In common cases, if the target trajectory and the resulting starting point for drilling cannot be identified based on the first X-ray, another X-ray may be acquired from a different preferred viewing direction. Both X-ray images may be aligned based on the positioning of the target object, thereby enabling the calculation of the instrument's 3D position and orientation relative to the target object and thus to the target trajectory. Therefore, the entire repositioning (translation and rotation), including drilling, may be performed by a robot. This procedure is not limited to drilling.

[0208] Reduction support When anatomically reducing a proximal femoral fracture, it is possible that a remaining dorsal gap may still exist even if the reduction appears correct on both APX images (e.g., Adams' arch appears intact) and lateral images. For this reason, obtaining true ML images is sometimes recommended, as they have the highest probability of showing such a gap. However, even true ML images may still fail to reveal such inaccurate reductions.

[0209] Furthermore, the remaining dorsal space, which is not visible on X-ray, has limited degrees of freedom, meaning that inaccurate reduction can surely be corrected by rotating the medial fragment around the axis defined by the main fracture line.

[0210] Such correction may also be achieved by the following procedure, which is shown here for the case of two fragments.

[0211] 1. This system loads a 3D model (typically obtained using a preoperative CT scan) showing segmented 3D bone fragments. 2. The surgeon obtains APXI images. 3. This system may optionally detect fracture lines as a reference. 4. This system may arbitrarily determine a line that approximates the major fracture line. 5. The surgeon reduces the fracture until it is clearly visible on the APXI image. 6. The surgeon rotates the C-arm to the ML position and acquires the X-ray image. 7. The system may also determine the relative 3D position and 3D orientation between two bone fragments, thereby supporting the surgeon in evaluating the reduction and potentially determining the correct reduction.

[0212] For step 7, the system may use a priori information that the bone fragments are in contact with each other along the anterior fracture line. This line is actually defined in 3D, and therefore detection in step 3 is simply optional.

[0213] In this scenario, the two objects (fragments) are in contact not only at a single point, but also along a one-dimensional structure that approximates a line in 3D space. Therefore, there is only one undefined degree of freedom (i.e., rotation around the anterior fracture line). Thus, the relative 3D position and 3D rotation between the objects can be determined using the ideas presented in the previous section.

[0214] Combining this method with other previously described techniques (i.e., the alignment of many X-ray images, with or without additional equipment / implants (e.g., nails)) can yield greater accuracy. This may require no further effort at all if done after nail insertion and may be used to help record both images.

[0215] Furthermore, this type of a priori information exists in several reduction scenarios in orthopedic trauma, such as determining the varus / valgus position of a fragment. A further example is a scenario where it is known (e.g., based on X-rays), namely • The fragments are in contact with each other (the least restrictive kind of a priori information). If fragments are in contact with each other, it may be sufficient to know that their position is among several possibilities. For example, if the reduction looks correct on the APX image, it may be assumed that the fragments are in contact along the fracture line in either the dorsal or ventral direction. In more extreme scenarios, the dorsal fracture line of one fragment may be in contact with the ventral fracture line of another fragment. The algorithm can then evaluate all these possibilities and select the one that provides the best 3D match. • How the fragments come into contact with each other (in a 1D structure such as a point or line, or a 2D structure such as a plane, etc.) It may also be related to something else.

[0216] The procedure described above may also be applied to three or more fragments. If a known relationship exists between bone fragments A and B, and a known relationship exists between bone fragments B and C, this can be used to associate bone fragments A and C.

[0217] This system may also automatically determine whether any detected anatomical gaps between bone fragments are within the normal range, and whether any protruding bone fragments are statistically significant for deviating significantly from the fitted statistical shape model.

[0218] A further example of a situation where a surgeon might inaccurately believe a reduction is correct based on what is seen in an X-ray image is the scenario of multiple fragments of the proximal tibia. Figure 27 shows an axial view of the proximal end of the tibia containing fragments A–E. Here, fragments labeled A–D are already anatomically reduced, but fragment E is depressed, i.e., displaced distally compared to a correct anatomical reduction. Such situations can be difficult for a surgeon to determine because the X-ray (in the AP or ML direction) shows many other fracture lines, as well as lines corresponding to regular anatomical structures (e.g., the fibula). The present invention may be able to detect such scenarios by accurately determining the relative 3D position and 3D orientation between all fragments. This may be possible because the system can use a priori information that all fragments are anatomically reduced, except for the medial fragment (e.g., fragment E in Figure 27) (which may be displaced distally). Here, the free parameter of the system is the proximal / distal position of the fragments located within the bone.

Claims

1. A system for assisting medical imaging, wherein the system comprises a processing device and a software program product, and when the software program product is executed by the processing device, the system The characteristics of this are determined by the imaging parameters, and the image is an image of at least one object. Classifying the objects within the aforementioned image, To determine the position of the object with respect to the coordinate system, Based on the location of the object, the current imaging direction relative to the object is determined, A system that forces you to do something.

2. The system according to claim 1, wherein the system is configured to determine a desired imaging direction for the object.

3. The system according to claim 2, wherein the system is configured to calculate the translation and / or rotation of the imaging device and / or the object required to change the imaging direction of the imaging device from the current imaging direction to the desired imaging direction.

4. The system according to claim 3, wherein the system is configured to give commands to at least one of a group consisting of a human, a user, a robotic C-arm, a robotic surgeon, a robotic operating table, an ultrasound imaging device, and operating room staff to change the imaging direction of the imaging device from the current imaging direction to the desired imaging direction.

5. The system according to claim 1, wherein the system is configured to calculate the translation and / or rotation of the imaging device and / or the object, such that the target structure of the object appears at a desired position in a new image.

6. The system according to claim 5, wherein the system is configured to give commands to at least one of a group consisting of a human, a user, a robotic C-arm, a robotic surgeon, a robotic operating table, an ultrasound imaging device, and operating room staff to change the position of the imaging device relative to the target structure of the object so that the target structure of the object appears at a desired position in the new image.

7. The system according to any one of claims 1 to 6, comprising a robotic imaging device, configured to position the robotic imaging device and / or generate an image using the robotic imaging device.

8. The system according to any one of claims 1 to 7, wherein the system is configured to determine the relative 3D position and 3D orientation between the object and the second object.

9. The system according to claim 8, wherein the system is configured to restrict a desired imaging direction to an angular range that allows (i) a particular anatomically relevant figure or (ii) one of the objects to be viewed from a particular direction, in order to determine the relative 3D position and 3D orientation between the object and the second object.

10. The system according to any one of claims 8 and 9, wherein a robotic device or robotic surgeon holds the second object, the second object being an instrument, and the robotic device or robotic surgeon performs a surgical procedure.

11. The system according to any one of claims 1 to 10, wherein a 3D reconstruction of the object is determined based on at least one of the group consisting of (i) locating an anatomical object, (ii) a further image having a desired imaging direction relative to the object, (iii) a second object shown in the image, (iv) an anchor point shown in the image, (v) a known imaging direction of the imaging device, and (vi) the relative 3D position and 3D orientation between the object and the second object.

12. The system according to any one of claims 1 to 11, wherein a neural network is used for at least one calculation and / or decision from the group consisting of determining the current imaging direction, determining a desired imaging direction, calculating the translation of the imaging device, calculating the rotational translation of the imaging device, and providing a 3D reconstruction of an anatomical object.

13. The system according to any one of claims 1 to 12, wherein the image is an X-ray projection image or an ultrasound image.

14. A software program product, the software program product includes a set of instructions that can be executed by a system processing unit, and the set of instructions, when executed on the processing unit, The characteristics of this are determined by the imaging parameters, and the image is an image of at least one object. Classifying the objects within the aforementioned image, To determine the position of the object with respect to the coordinate system, Based on the location of the object, the current imaging direction relative to the object is determined, A software program product that configures the system to perform the aforementioned tasks.

15. The software program product according to claim 14, wherein the set of instructions, when executed on the processing device, configures the system to determine a desired imaging direction for the object.

Images